Methods and systems for determining the interdependencies between the subsystems of a vehicle

Abstract

The embodiments relate to a method and system for determining interdependencies between subsystems of a vehicle. One method includes: selecting a scenario for simulating a vehicle and an agent in a virtual world environment; selecting a set of subsystem component models for the vehicle; using the selected subsystem component models to simulate the scenario in the virtual world environment, wherein the simulation includes: estimating the attitude of the vehicle and the agent; determining the detection probability of an agent utilizing the vehicle's sensors based on the perception subsystem component models, the estimated attitude of the vehicle / agent, and a latency model modeling the latency of a message queue network used for communicating data between the subsystems; generating a set of candidate trajectories for the vehicle; evaluating each trajectory based on the number of rule violations associated with it; selecting one trajectory from the set of trajectories based on the number of rule violations; and analyzing the outputs of the subsystem component models to determine their interdependencies.

Description

Technical Field

[0001] This invention relates to vehicles. Background Technology

[0002] During the development of autonomous vehicle (AV) systems, it is often difficult to understand the impact of specific AV subsystems on the overall AV performance, and it is difficult to identify performance improvements at the subsystem level that will improve AV performance. Summary of the Invention

[0003] A method includes: using at least one processor, selecting a scenario for simulating a vehicle and at least one intelligent agent in a virtual world environment; using the at least one processor, selecting a set of subsystem component models for the vehicle, wherein the subsystem component models include a positioning subsystem component model, a perception subsystem component model, a trajectory proposer subsystem component model, and a trajectory selector subsystem component model; using the at least one processor, simulating the scenario in the virtual world environment using the selected subsystem component models, wherein the simulation includes: estimating the attitude of the vehicle and the at least one intelligent agent based on the positioning subsystem component models, and estimating the attitude of the vehicle and the at least one intelligent agent based on the perception subsystem component models, the estimated attitude of the vehicle, and the estimated attitude of the at least one intelligent agent. The system utilizes at least one processor to determine the detection probability of at least one agent using at least one sensor of the vehicle, and a latency model that models the latency of a message queuing network for communicating data between subsystems; uses at least one processor to generate a set of candidate trajectories for the vehicle based on the trajectory proposer subsystem component model to avoid collisions with the at least one agent; uses at least one processor to evaluate a trajectory based on the number of rule violations associated with each trajectory; uses at least one processor to select a trajectory from the set of trajectories based on the number of rule violations for each candidate trajectory; and uses at least one processor to analyze the output of the subsystem component model to determine the interdependencies between two or more subsystems of the vehicle. Attached Figure Description

[0004] Figure 1 It is an example environment that can realize a vehicle that includes one or more components of an autonomous system;

[0005] Figure 2 It is a diagram of one or more example systems that include autonomous vehicles;

[0006] Figure 3 yes Figure 1 and Figure 2 A diagram of example components of one or more devices and / or one or more systems;

[0007] Figure 4 This is a diagram of an example component of an autonomous system;

[0008] Figure 5 Provides an example framework for modeling subsystems of an AV system, as well as an overview of the impact of these subsystems on AV performance;

[0009] Figure 6 An example model structure of an AV system and its subsystems is shown;

[0010] Figure 7 This shows an example model structure for timing;

[0011] Figure 8 Example rulebook is shown;

[0012] Figure 9 and Figure 10 Showing the use Figure 5 and Figure 6 Example scenarios for the framework;

[0013] Figure 11A This demonstrates the use of a Markov model for determining recall.

[0014] Figure 11B It is a plot of the probability of pedestrian identification at the current and past time steps relative to the distance to the pedestrian, thus modeling perceptual recall based on the Markov model;

[0015] Figure 12 An example of perceptual recall modeling using Markov models is shown;

[0016] Figure 13 Example trajectory proposals and selection scenarios are shown;

[0017] Figure 14A It shows that for Figure 13 The example scenario shown is illustrated with a bar chart of example rule R2 violations as a function of the detection probability.

[0018] Figure 14B It shows that for Figure 13 The example scenario shown is a scatter plot of example rule R2 violations as a function of perception period and recall error slope;

[0019] Figure 14C It shows that for Figure 13 The example scenario shown is a scatter plot of example rule R2 violations as a function of positioning error and selection period;

[0020] Figure 14D It shows that for Figure 13The example scenario shown is a scatter plot of example rule R2 violations as a function of the slopes of localization error and recall error;

[0021] Figure 15 This example demonstrates a correlation analysis of design variables;

[0022] Figures 16 to 18 Example rule trade-offs are shown;

[0023] Figure 19 An example application of the framework for determining system, subsystem, and sensor-level performance targets is shown;

[0024] Figure 20 An example application of a framework for selecting the candidate AV architecture with the highest performance from multiple candidate AV architectures is shown; and

[0025] Figure 21 It is a flowchart for determining the interdependencies of the vehicle's subsystems. Detailed Implementation

[0026] In the following description, numerous specific details are set forth for purposes of explanation in order to provide a thorough understanding of this disclosure. However, it will be apparent that the embodiments described herein can be practiced without these specific details. In some instances, well-known constructions and apparatuses are illustrated in block diagram form to avoid unnecessarily obscuring aspects of this disclosure.

[0027] In the accompanying drawings, for ease of description, specific arrangements or orders of schematic elements (such as those representing systems, devices, modules, instruction blocks, and / or data elements) are illustrated. However, those skilled in the art will understand that, unless explicitly described, the specific order or arrangement of schematic elements in the drawings is not intended to imply a requirement for a particular processing order or sequence, or separation of processes. Furthermore, unless explicitly described, the inclusion of schematic elements in the drawings is not intended to imply that such elements are required in all embodiments, nor is it intended to imply that features represented by such elements cannot be included in some embodiments or cannot be combined with other elements in some embodiments.

[0028] Furthermore, in the accompanying drawings, connecting elements (such as solid or dashed lines or arrows) are used to illustrate connections, relationships, or associations between or among two or more other schematic elements. The absence of any such connecting element does not imply that connections, relationships, or associations cannot exist. In other words, some connections, relationships, or associations between elements are not illustrated in the drawings so as not to obscure the content of this disclosure. Additionally, for ease of illustration, a single connecting element may be used to represent multiple connections, relationships, or associations between elements. For example, if a connecting element represents communication of signals, data, or instructions (e.g., "software instructions"), those skilled in the art will understand that such an element may represent one or more signal paths (e.g., a bus) that may be necessary to influence the communication.

[0029] Although the terms "first," "second," and / or "third," etc., are used to describe various elements, these elements should not be limited by these terms. The terms "first," "second," and / or "third" are used only to distinguish one element from another. For example, without departing from the scope of the described embodiments, a first contact may be referred to as a second contact, and similarly, a second contact may be referred to as a first contact. Both the first contact and the second contact are contacts, but they are not the same contact.

[0030] The terminology used in the description of the various embodiments described herein is included for the purpose of describing particular embodiments only and is not intended to be limiting. As used in the description of the various embodiments described and the appended claims, the singular forms “a,” “an,” and “the” are also intended to include the plural forms and may be used interchangeably with “one or more” or “at least one” unless the context clearly indicates otherwise. It will also be understood that the term “and / or” as used herein refers to and includes any and all possible combinations of one or more of the associated listed items. It will also be understood that when the terms “comprising,” “including,” “possessing,” and / or “having” are used in this specification, they specifically indicate the presence of the stated features, integers, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0031] As used herein, the terms "communication" and "to communicate" refer to at least one of the following: receiving, receiving, transmitting, conveying, and / or providing information (or information represented by, for example, data, signals, messages, instructions, and / or commands). For a unit (e.g., an apparatus, system, component of an apparatus or system, and / or combinations thereof) that wants to communicate with another unit, this means that the unit is able to receive information directly or indirectly from the other unit and / or send (e.g., transmit) information to the other unit. This can refer to a direct or indirect connection that is essentially wired and / or wireless. Furthermore, two units can communicate with each other even if the transmitted information can be modified, processed, relayed, and / or routed between the first and second units. For example, the first unit can communicate with the second unit even if it passively receives information and does not actively transmit information to the second unit. As another example, the first unit can communicate with the second unit if at least one intermediary unit (e.g., a third unit located between the first and second units) processes information received from the first unit and transmits the processed information to the second unit. In some embodiments, a message may refer to a network packet that includes data (e.g., a data packet, etc.).

[0032] As used herein, depending on the context, the term "if" may optionally be interpreted as "when," "in," "in response to being determined," and / or "in response to being detected," etc. Similarly, depending on the context, the phrases "if determined" or "if [the stated condition or event] is detected" may optionally be interpreted as "in response to being determined," "in response to being determined," "or" "in response to being detected," and / or "in response to being detected," etc. Furthermore, as used herein, the terms "have," "possess," or "own," etc., are intended to be open-ended terms. Additionally, unless explicitly stated otherwise, the phrase "based on" is intended to mean "at least partially based on."

[0033] Reference will now be made in detail to embodiments, examples of which are illustrated in the accompanying drawings. Numerous specific details are set forth in the following detailed description in order to provide a thorough understanding of the various embodiments described. However, it will be apparent to those skilled in the art that the various embodiments described can be practiced without these specific details. In other instances, well-known methods, processes, components, circuits, and networks have not been described in detail so as not to unnecessarily obscure aspects of the embodiments.

[0034] General Overview

[0035] In some aspects and / or embodiments, the systems, methods, and computer program products described herein include and / or implement a functional architecture for modeling AV subsystems and their interdependencies. The disclosed functional framework enables developers to better understand the impact of AV subsystem performance as defined by the rulebook, identify subsystem-level performance improvements that will result in improved AV behavior, and provide quantitative empirical models for creating performance requirements based on incorporating rule violation scores into subsystem performance metrics.

[0036] In this embodiment, the framework is applied to different AV driving scenarios in Monte Carlo simulations to evaluate AV performance across various scenarios (e.g., determining the number of rule violations by the AV). Users can modify or update the model to improve AV performance for a given driving scenario and run the simulation again with the updated model to see if performance has improved or decreased. Given (e.g., by a rulebook) quantitative performance expectations, the system model provides tools that enable AV technology stack developers to explore the AV design space to find a design that best meets the performance expectations. The framework uses a set of equations representing the performance of each subsystem in the AV technology stack and their interactions and interdependencies with other subsystems in the AV stack.

[0037] In this embodiment, the framework provides a high-level model of the AV stack, encompassing the main subsystems driving performance. The framework takes a functional architecture as input, including architectural variations, expert opinions on which effects drive performance most at the subsystem level, a list of relevant scenarios, and a set of rules for evaluating AV performance in a given scenario. The framework outputs performance requirements for the subsystems and a rulebook performance comparison between different AV stack designs. For example, using the same rulebook, the framework can determine whether a particular design of the AV stack will cause fewer rule violations compared to other designs of the AV stack. The framework can also test high-level subsystem concepts and whether they affect the rulebook score in a given scenario.

[0038] In one embodiment, a method includes: using at least one processor to select a scenario for simulating a vehicle and at least one intelligent agent in a virtual world environment; using the at least one processor to select a set of subsystem component models for the vehicle, wherein the subsystem component models include a positioning subsystem component model, a perception subsystem component model, a trajectory proposer subsystem component model, and a trajectory selector subsystem component model; using the at least one processor to simulate the scenario in the virtual world environment using the selected subsystem component models, wherein the simulation includes: estimating the attitude of the vehicle and the at least one intelligent agent based on the positioning subsystem component model, and estimating the attitude of the vehicle and the at least one intelligent agent based on the perception subsystem component model, the estimated attitude of the vehicle, and the at least one intelligent agent. The system uses at least one agent to estimate the attitude and a latency model of a message queuing network for communicating data between subsystems to determine the detection probability of the at least one agent utilizing at least one sensor of the vehicle; uses at least one processor to generate a set of candidate trajectories for the vehicle to navigate the at least one agent based on the trajectory proposer subsystem component model; uses at least one processor to evaluate the trajectory based on rule violations associated with each trajectory; uses at least one processor to select a trajectory from the set of trajectories based on the number of rule violations for each candidate trajectory; and uses at least one processor to analyze the output of the subsystem component model to determine the interdependencies between two or more subsystems of the vehicle.

[0039] In one embodiment, the simulation includes a Monte Carlo simulation.

[0040] In one embodiment, the latency model includes performing discrete event simulations of the message queuing network and selecting simulated events based on the probability distribution of the events.

[0041] In an embodiment, the positioning subsystem component model includes a covariance matrix for vehicle attitude error, computation time, and task cycle.

[0042] In an embodiment, the perception subsystem component model includes recall as a function of the distance between the vehicle and the at least one agent and the agent's detection at a previous time step in the simulation, a list of classes, a confusion matrix between classes, and a covariance matrix with respect to the agent's pose.

[0043] In this embodiment, recall as a function of distance is based on a Markov model.

[0044] In this embodiment, the set of candidate trajectories is randomly generated.

[0045] In one embodiment, the trajectory selector subsystem component model includes a rule manual for ranking the set of candidate trajectories and selecting a specific trajectory from the set of candidate trajectories based on the ranking.

[0046] In one embodiment, analyzing the output of the subsystem component model to determine the interdependencies between two or more subsystem component models in the subsystem component model further includes comparing rule violations with the probability of detection.

[0047] In one embodiment, analyzing the output of the subsystem component model to determine the interdependencies between two or more subsystem component models in the subsystem component model further includes comparing the perception period of the perception subsystem with the recall error slope.

[0048] In one embodiment, analyzing the output of the subsystem component model to determine the interdependence between two or more subsystem component models in the subsystem component model further includes comparing the positioning error with the selection time period of the trajectory selector subsystem.

[0049] In one embodiment, analyzing the output of the subsystem component model to determine the interdependencies between two or more subsystem component models in the subsystem component model further includes comparing the localization error with the recall error slope.

[0050] In one embodiment, analyzing the output of the subsystem component model to determine the interdependencies between two or more subsystem component models in the subsystem component model further includes generating a correlation heatmap that associates subsystem model parameters with the number of rule violations.

[0051] In one embodiment, analyzing the output of the subsystem component model to determine the interdependencies between two or more subsystem component models in the subsystem component model further includes comparing the degree of violation of at least two rules.

[0052] In one embodiment, a system includes: at least one processor; and a memory for storing instructions that, when executed by the at least one processor, cause the at least one processor to perform any of the methods described above.

[0053] In one embodiment, a non-transitory computer-readable storage medium stores instructions that, when executed by one or more processors, cause the one or more processors to perform any of the methods described above.

[0054] The implementation of the systems, methods, and computer program products described herein offers at least the following advantages. The disclosed embodiments enable rating AV stack designs based on expectations of good driving by utilizing a rulebook to guide which rules are applicable to all scenarios and which violations are most critical. The disclosed embodiments also enable the use of sampling techniques to analyze AV stack components in relevant scenarios, construct abstract models of the AV to achieve behavior given subsystem performance, and run sensitivity analyses and Monte Carlo simulations to obtain statistically significant (random) rule violations given the AV model and scenario. The disclosed embodiments output curves or surfaces instead of a single data point to provide a deeper understanding of subsystem interdependencies. The framework can be implemented in a computer as a tool that decision-makers can use to weigh trade-offs between different design configurations, constraints, etc. Another advantage of the framework is that it allows for modeling and testing of design solutions for new subsystems. For example, the question, “What if the perception algorithm had 99.99% recall?” can be answered. Using the disclosed framework, developers can see the impact of 99.99% recall on the perception system even if perception code that meets the specification does not exist.

[0055] Now for reference Figure 1 Example environment 100 is illustrated, in which vehicles including autonomous systems and vehicles not including autonomous systems operate. As illustrated, environment 100 includes vehicles 102a-102n, objects 104a-104n, routes 106a-106n, area 108, vehicle-to-infrastructure (V2I) device 110, network 112, remote AV system 114, queue management system 116, and V2I system 118. Vehicles 102a-102n, vehicle-to-infrastructure (V2I) device 110, network 112, AV system 114, queue management system 116, and V2I system 118 are interconnected via wired connections, wireless connections, or a combination of wired and wireless connections (e.g., establishing connections for communication, etc.). In some embodiments, objects 104a-104n are interconnected with at least one of vehicles 102a-102n, vehicle-to-infrastructure (V2I) devices 110, network 112, AV system 114, queue management system 116, and V2I system 118 via wired connection, wireless connection, or a combination of wired and wireless connection.

[0056] Vehicles 102a-102n (specifically referred to as vehicle 102 and collectively as vehicle 102) include at least one device configured to transport goods and / or people. In some embodiments, vehicle 102 is configured to communicate with V2I device 110, remote AV system 114, queue management system 116 and / or V2I system 118 via network 112. In some embodiments, vehicle 102 includes cars, buses, trucks and / or trains, etc. In some embodiments, vehicle 102 is associated with vehicle 200 described herein (see Figure 2 The vehicles 102 are the same as or similar to autonomous vehicles 202. In some embodiments, vehicles 200 in a group of vehicles 200 are associated with an autonomous queue manager. In some embodiments, as described herein, vehicles 102 travel along corresponding routes 106a-106n (each individually referred to as route 106 and collectively as route 106). In some embodiments, one or more vehicles 102 include an autonomous system (e.g., an autonomous system that is the same as or similar to autonomous system 202).

[0057] Objects 104a-104n (each individually referred to as object 104 and collectively as object 104) include, for example, at least one vehicle, at least one pedestrian, at least one cyclist, and / or at least one structure (e.g., a building, a sign, a fire hydrant, etc.). Each object 104 (e.g., located at a fixed location and for a period of time) is either stationary or (e.g., having a speed and associated with at least one trajectory) moving. In some embodiments, object 104 is associated with a corresponding location in area 108.

[0058] Routes 106a-106n (each individually referred to as Route 106 and collectively as Route 106) are each associated with (e.g., defining) a series of actions (also referred to as trajectories) along which the connecting AV can navigate. Each Route 106 begins with an initial state (e.g., a state corresponding to a first spatiotemporal location and / or speed, etc.) and ends with a final target state (e.g., a state corresponding to a second spatiotemporal location different from the first spatiotemporal location) or a target area (e.g., a subspace of an acceptable state (e.g., a termination state)). In some embodiments, a first state includes a location where one or more individuals will board the AV, and a second state or area includes a location where one or more individuals boarding the AV will disembark. In some embodiments, Route 106 includes multiple acceptable state sequences (e.g., multiple spatiotemporal location sequences) associated with multiple trajectories (e.g., defining multiple trajectories). In the example, Route 106 includes only high-level actions or imprecise state locations, such as a series of connecting roads indicating a change of direction at a roadway intersection. Additionally or alternatively, route 106 may include more precise actions or states, such as, for example, specific target lanes or precise locations within a lane area and target rates at those locations. In the example, route 106 includes multiple precise state sequences along at least one high-level action with a finite look-ahead horizon leading to an intermediate target, wherein the cumulative combination of successive iterations of the finite horizon state sequences corresponds to multiple trajectories that collectively form a high-level route terminating at a final target state or region.

[0059] Region 108 includes a physical area (e.g., a geographic region) that the vehicle 102 can navigate. In the example, region 108 includes at least one state (e.g., a country, a province, a single state among multiple states included in a country, etc.), at least a portion of a state, at least one city, at least a portion of a city, etc. In some embodiments, region 108 includes at least one named arterial road (referred to herein as a "road"), such as a highway, interstate highway, park road, city street, etc. Additionally or alternatively, in some examples, region 108 includes at least one unnamed road, such as a driving lane, a section of a parking lot, a section of vacant land and / or undeveloped area, dirt road, etc. In some embodiments, a road includes at least one lane (e.g., a portion of the road that the vehicle 102 can traverse). In the example, a road includes at least one lane associated with at least one lane marking (e.g., identified based on at least one lane marking).

[0060] The Vehicle-to-Infrastructure (V2I) device 110 (sometimes referred to as a Vehicle-to-Everything (V2X) device) includes at least one device configured to communicate with vehicle 102 and / or V2I system 118. In some embodiments, V2I device 110 is configured to communicate with vehicle 102, remote AV system 114, queue management system 116, and / or V2I system 118 via network 112. In some embodiments, V2I device 110 includes radio frequency identification (RFID) devices, signs, cameras (e.g., two-dimensional (2D) and / or three-dimensional (3D) cameras), lane markings, streetlights, parking meters, etc. In some embodiments, V2I device 110 is configured to communicate directly with vehicle 102. Additionally or alternatively, in some embodiments, V2I device 110 is configured to communicate with vehicle 102, remote AV system 114, and / or queue management system 116 via V2I system 118. In some embodiments, V2I device 110 is configured to communicate with V2I system 118 via network 112.

[0061] Network 112 includes one or more wired and / or wireless networks. In the example, network 112 includes cellular networks (e.g., Long Term Evolution (LTE) networks, third-generation (3G) networks, fourth-generation (4G) networks, fifth-generation (5G) networks, Code Division Multiple Access (CDMA) networks, etc.), Public Land Mobile Networks (PLMNs), Local Area Networks (LANs), Wide Area Networks (WANs), Metropolitan Area Networks (MANs), telephone networks (e.g., Public Switched Telephone Networks (PSTN)), private networks, self-organizing networks, intranets, the Internet, fiber-based networks, cloud computing networks, etc., and / or combinations of some or all of these networks.

[0062] The remote AV system 114 includes at least one device configured to communicate with the vehicle 102, V2I device 110, network 112, queue management system 116, and / or V2I system 118 via network 112. In examples, the remote AV system 114 includes a server, server group, and / or other similar devices. In some embodiments, the remote AV system 114 is located in the same location as the queue management system 116. In some embodiments, the remote AV system 114 participates in the installation of some or all of the components of the vehicle, including autonomous systems, autonomous vehicle computing, and / or software implemented by autonomous vehicle computing. In some embodiments, the remote AV system 114 maintains (e.g., updates and / or replaces) these components and / or software during the lifespan of the vehicle.

[0063] The queue management system 116 includes at least one device configured to communicate with vehicle 102, V2I device 110, remote AV system 114, and / or V2I system 118. In examples, the queue management system 116 includes servers, server groups, and / or other similar devices. In some embodiments, the queue management system 116 is associated with a ride-sharing company (e.g., an organization for controlling the operation of multiple vehicles (e.g., vehicles including autonomous systems and / or vehicles not including autonomous systems)).

[0064] In some embodiments, the V2I system 118 includes at least one device configured to communicate with the vehicle 102, the V2I device 110, the remote AV system 114, and / or the queue management system 116 via a network 112. In some examples, the V2I system 118 is configured to communicate with the V2I device 110 via a connection different from the network 112. In some embodiments, the V2I system 118 includes a server, a server group, and / or other similar devices. In some embodiments, the V2I system 118 is associated with a municipality or private entity (e.g., a private entity maintaining the V2I device 110).

[0065] supply Figure 1 The number and arrangement of the elements are shown as examples. (and) Figure 1 Compared to the illustrated elements, there may be additional elements, fewer elements, different elements, and / or elements arranged differently. Additionally or alternatively, at least one element of environment 100 may be described as being composed of… Figure 1 One or more functions performed by at least one different element of environment 100. Additionally or alternatively, at least one group of elements of environment 100 may perform one or more functions described as performed by at least one different group of elements of environment 100.

[0066] Now for reference Figure 2 The vehicle 200 includes an autonomous system 202, a powertrain control system 204, a steering control system 206, and a braking system 208. In some embodiments, the vehicle 200 and the vehicle 102 (see...) Figure 1The vehicle 200 is similar to or the same as the vehicle in question. In some embodiments, the vehicle 200 has autonomous capabilities (e.g., implementing at least one function, feature, and / or device that enables the vehicle 200 to operate partially or fully without human intervention, including but not limited to fully autonomous vehicles (e.g., vehicles that abandon human intervention) and / or highly autonomous vehicles (e.g., vehicles that abandon human intervention in certain situations)). For a detailed description of fully autonomous and highly autonomous vehicles, refer to SAE International's standard J3016: Taxonomy and Definitions for Terms Related to On-Road Motor Vehicle Automated Driving Systems, the entire contents of which are incorporated herein by reference. In some embodiments, the vehicle 200 is associated with an autonomous queue manager and / or a ride-sharing company.

[0067] Autonomous system 202 includes a sensor suite comprising one or more devices such as camera 202a, LiDAR sensor 202b, radar sensor 202c, and microphone 202d. In some embodiments, autonomous system 202 may include more or fewer devices and / or different devices (e.g., ultrasonic sensors, inertial sensors, GPS receivers (discussed below), and / or odometer sensors for generating data associated with an indication of the distance traveled by vehicle 200). In some embodiments, autonomous system 202 uses one or more devices included in autonomous system 202 to generate data associated with environment 100 as described herein. The data generated by one or more devices of autonomous system 202 may be used by one or more systems as described herein to observe the environment in which vehicle 200 is located (e.g., environment 100). In some embodiments, autonomous system 202 includes communication device 202e, autonomous vehicle computing 202f, and safety controller 202g.

[0068] Camera 202a includes components configured to communicate with communication device 202e, autonomous vehicle computing 202f, and / or safety controller 202g via a bus (e.g., with...). Figure 3At least one means of communicating with the same or similar bus as bus 302. Camera 202a includes at least one camera (e.g., a digital camera using a light sensor such as a charge-coupled device (CCD), a thermal camera, an infrared (IR) camera, and / or an event camera, etc.) for capturing images of physical objects (e.g., cars, buses, curbs, and / or people, etc.). In some embodiments, camera 202a generates camera data as output. In some examples, camera 202a generates camera data including image data associated with an image. In this example, the image data may specify at least one parameter corresponding to the image (e.g., image characteristics such as exposure, brightness, etc., and / or image timestamp, etc.). In such examples, the image may be in a format (e.g., RAW, JPEG, and / or PNG, etc.). In some embodiments, camera 202a includes multiple independent cameras configured (e.g., positioned on) a vehicle to capture images for stereoscopic imaging (stereoscopic vision). In some examples, camera 202a includes generating image data and transmitting the image data to an autonomous vehicle computing 202f and / or a queue management system (e.g., with...). Figure 1 The queue management system 116 (same as or similar to a queue management system) has multiple cameras. In such an example, the autonomous vehicle calculation 202f determines the depth of one or more objects in the fields of view of at least two of the multiple cameras based on image data from at least two cameras. In some embodiments, camera 202a is configured to capture images of objects within a distance relative to camera 202a (e.g., up to 100 meters and / or up to 1 kilometer, etc.). Therefore, camera 202a includes features such as sensors and lenses optimized for sensing objects at one or more distances relative to camera 202a.

[0069] In embodiments, camera 202a includes at least one camera configured to capture one or more images associated with one or more traffic lights, street signs, and / or other physical objects providing visual navigation information. In some embodiments, camera 202a generates traffic light data associated with one or more images. In some examples, camera 202a generates TLD data associated with one or more images, including formats such as RAW, JPEG, and / or PNG. In some embodiments, camera 202a, which generates TLD data, differs from other systems containing cameras described herein in that camera 202a may include one or more cameras with a wide field of view (e.g., a wide-angle lens, a fisheye lens, and / or a lens with an angle of view of about 120 degrees or greater) to generate images associated with as many physical objects as possible.

[0070] The LiDAR sensor 202b includes components configured to communicate with the communication device 202e, the autonomous vehicle computing device 202f, and / or the safety controller 202g via a bus (e.g., with...). Figure 3 At least one device that communicates with the same or similar bus (bus 302). The LiDAR sensor 202b includes a system configured to emit light from a emitter (e.g., a laser emitter). The light emitted by the LiDAR sensor 202b includes light outside the visible spectrum (e.g., infrared light, etc.). In some embodiments, during operation, the light emitted by the LiDAR sensor 202b encounters a physical object (e.g., a vehicle) and is reflected back to the LiDAR sensor 202b. In some embodiments, the light emitted by the LiDAR sensor 202b does not penetrate the physical object it encounters. The LiDAR sensor 202b also includes at least one photosensor that detects the light after it has encountered a physical object. In some embodiments, at least one data processing system associated with the LiDAR sensor 202b generates an image (e.g., point cloud and / or combined point cloud, etc.) representing objects included in the field of view of the LiDAR sensor 202b. In some examples, at least one data processing system associated with the LiDAR sensor 202b generates an image representing the boundaries of a physical object and / or the surface of the physical object (e.g., the topology of the surface). In such examples, the image is used to determine the boundaries of the physical object within the field of view of the LiDAR sensor 202b.

[0071] The radio detection and ranging (radar) sensor 202c includes components configured to communicate with the communication device 202e, the autonomous vehicle computing 202f, and / or the safety controller 202g via a bus (e.g., with...). Figure 3 At least one device that communicates with the same or similar bus (bus 302). The radar sensor 202c includes a system configured to emit (pulsed or continuous) radio waves. The radio waves emitted by the radar sensor 202c include radio waves within a predetermined spectrum. In some embodiments, during operation, the radio waves emitted by the radar sensor 202c encounter a physical object and are reflected back to the radar sensor 202c. In some embodiments, the radio waves emitted by the radar sensor 202c are not reflected by some objects. In some embodiments, at least one data processing system associated with the radar sensor 202c generates a signal representing objects included in the field of view of the radar sensor 202c. For example, at least one data processing system associated with the radar sensor 202c generates an image representing the boundaries of physical objects and / or the surfaces of physical objects (e.g., surface topology). In some examples, this image is used to determine the boundaries of physical objects in the field of view of the radar sensor 202c.

[0072] Microphone 202d includes components configured to communicate with communication device 202e, autonomous vehicle computing 202f, and / or safety controller 202g via a bus (e.g., with...). Figure 3 At least one device that communicates with the same or similar bus as bus 302. Microphone 202d includes one or more microphones (e.g., array microphones and / or external microphones, etc.) that capture audio signals and generate data associated with (e.g., representing) the audio signals. In some examples, microphone 202d includes transducer devices and / or similar devices. In some embodiments, one or more systems described herein can receive data generated by microphone 202d and determine the position (e.g., distance, etc.) of an object relative to vehicle 200 based on the audio signal associated with the data.

[0073] The communication device 202e includes at least one device configured to communicate with a camera 202a, a LiDAR sensor 202b, a radar sensor 202c, a microphone 202d, an autonomous vehicle computing system 202f, a safety controller 202g, and / or a drive-by-wire (DBW) system 202h. For example, the communication device 202e may include communication with… Figure 3 The communication device 202e is the same as or similar to the communication interface 314. In some embodiments, the communication device 202e includes a vehicle-to-vehicle (V2V) communication device (e.g., a device for enabling wireless communication of data between vehicles).

[0074] The autonomous vehicle computing 202f includes at least one device configured to communicate with a camera 202a, a LiDAR sensor 202b, a radar sensor 202c, a microphone 202d, a communication device 202e, a security controller 202g, and / or a DBW system 202h. In some examples, the autonomous vehicle computing 202f includes devices such as client devices, mobile devices (e.g., cellular phones and / or tablets) and / or servers (e.g., computing devices including one or more central processing units and / or graphics processing units). In some embodiments, the autonomous vehicle computing 202f is the same as or similar to the autonomous vehicle computing 400 described herein. Additionally or alternatively, in some embodiments, the autonomous vehicle computing 202f is configured to communicate with an autonomous vehicle system (e.g., with...). Figure 1 Remote AV systems 114 are the same as or similar to autonomous vehicle systems), queue management systems (e.g., with...). Figure 1 The queue management system 116 is the same as or similar to the queue management system 116), and V2I devices (e.g., with Figure 1V2I devices (same as or similar to V2I devices 110) and / or V2I systems (e.g., with V2I devices 110) Figure 1 The V2I system 118 communicates with the same or similar V2I system.

[0075] The safety controller 202g includes at least one device configured to communicate with a camera 202a, a LiDAR sensor 202b, a radar sensor 202c, a microphone 202d, a communication device 202e, an autonomous vehicle computing system 202f, and / or a DBW system 202h. In some examples, the safety controller 202g includes one or more controllers (electrical controllers and / or electromechanical controllers, etc.) configured to generate and / or transmit control signals to operate the vehicle 200 (e.g., powertrain control system 204, steering control system 206, and / or braking system 208, etc.). In some embodiments, the safety controller 202g is configured to generate control signals that take precedence over (e.g., override) the control signals generated and / or transmitted by the autonomous vehicle computing system 202f.

[0076] The DBW system 202h includes at least one device configured to communicate with the communication device 202e and / or the autonomous vehicle computing 202f. In some examples, the DBW system 202h includes one or more controllers (e.g., electrical controllers and / or electromechanical controllers, etc.) configured to generate and / or transmit control signals to operate the vehicle 200, including one or more devices (e.g., powertrain control system 204, steering control system 206, and / or braking system 208, etc.). Additionally or alternatively, one or more controllers of the DBW system 202h are configured to generate and / or transmit control signals to operate at least one different device (e.g., turn signals, headlights, door locks, and / or windshield wipers, etc.) of the vehicle 200.

[0077] The powertrain control system 204 includes at least one device configured to communicate with the DBW system 202h. In some examples, the powertrain control system 204 includes at least one controller and / or actuator, etc. In some embodiments, the powertrain control system 204 receives control signals from the DBW system 202h, and the powertrain control system 204 causes the vehicle 200 to start moving forward, stop moving forward, start moving backward, stop moving backward, accelerate in a certain direction, decelerate in a certain direction, make a left turn and / or make a right turn, etc. In examples, the powertrain control system 204 increases, keeps the same, or decreases the energy (e.g., fuel and / or electricity, etc.) supplied to the motor of the vehicle, thereby causing at least one wheel of the vehicle 200 to rotate or not rotate.

[0078] The steering control system 206 includes at least one device configured to rotate one or more wheels of the vehicle 200. In some examples, the steering control system 206 includes at least one controller and / or actuator, etc. In some embodiments, the steering control system 206 causes the two front wheels and / or the two rear wheels of the vehicle 200 to turn left or right, thereby causing the vehicle 200 to turn left or right.

[0079] The braking system 208 includes at least one device configured to actuate one or more brakes to decelerate and / or keep the vehicle 200 stationary. In some examples, the braking system 208 includes at least one controller and / or actuator configured to close one or more calipers associated with one or more wheels of the vehicle 200 on the respective rotor of the vehicle 200. Additionally or alternatively, in some examples, the braking system 208 includes an automatic emergency braking (AEB) system and / or a regenerative braking system, etc.

[0080] In some embodiments, the vehicle 200 includes at least one platform sensor (not explicitly illustrated) for measuring or inferring the nature of the state or conditions of the vehicle 200. In some examples, the vehicle 200 includes platform sensors such as a Global Positioning System (GPS) receiver, an Inertial Measurement Unit (IMU), a wheel rate sensor, a wheel brake pressure sensor, a wheel torque sensor, an engine torque sensor, and / or a steering angle sensor.

[0081] Now for reference Figure 3 A schematic diagram of device 300 is illustrated. As illustrated, device 300 includes a computer processor 304, a memory 306, a storage component 308, an input interface 310, an output interface 312, a communication interface 314, and a bus 302. In some embodiments, device 300 corresponds to: at least one device of vehicle 102 (e.g., at least one device of the system of vehicle 102); and / or at least one device of network 112 and / or one or more devices (e.g., one or more devices of the system of network 112). In some embodiments, one or more devices of vehicle 102 (e.g., one or more devices of the system of vehicle 102), and / or one or more devices of network 112 (e.g., one or more devices of the system of network 112) include at least one device 300 and / or at least one component of device 300. Figure 3 As shown, the device 300 includes a bus 302, a computer processor 304, a memory 306, a storage component 308, an input interface 310, an output interface 312, and a communication interface 314.

[0082] Bus 302 includes components for communication between the components of the licensed device 300. In some embodiments, the computer processor 304 is implemented in hardware, software, or a combination of hardware and software. In some examples, the computer processor 304 includes a computer processor (e.g., a central processing unit (CPU), a graphics processing unit (GPU), and / or an accelerated processing unit (APU), etc.), a microphone, a digital signal processor (DSP), and / or any processing component that can be programmed to perform at least one function (e.g., a field-programmable gate array (FPGA) and / or an application-specific integrated circuit (ASIC), etc.). Memory 306 includes random access memory (RAM), read-only memory (ROM), and / or another type of dynamic and / or static storage device (e.g., flash memory, magnetic memory, and / or optical memory, etc.) that stores data and / or instructions for use by the computer processor 304.

[0083] Storage component 308 stores data and / or software related to the operation and use of device 300. In some examples, storage component 308 includes hard disks (e.g., magnetic disks, optical disks, magneto-optical disks, and / or solid-state disks), compact discs (CDs), digital versatile discs (DVDs), floppy disks, cassette tapes, magnetic tapes, CD-ROMs, RAM, PROMs, EPROMs, FLASH-EPROMs, NV-RAMs, and / or other types of computer-readable media, and corresponding drives.

[0084] Input interface 310 includes components that enable the device 300 to receive information, such as via user input (e.g., a touchscreen display, keyboard, keypad, mouse, buttons, switches, microphone, and / or camera). Additionally or alternatively, in some embodiments, input interface 310 includes sensors for sensing information (e.g., a Global Positioning System (GPS) receiver, accelerometer, gyroscope, and / or actuator). Output interface 312 includes components for providing output information from device 300 (e.g., a display, speaker, and / or one or more light-emitting diodes (LEDs)).

[0085] In some embodiments, the communication interface 314 includes transceiver-like components (e.g., a transceiver and / or separate receivers and transmitters) that enable the licensing device 300 to communicate with other devices via a wired connection, a wireless connection, or a combination of wired and wireless connections. In some examples, the communication interface 314 enables the licensing device 300 to receive information from and / or provide information to another device. In some examples, the communication interface 314 includes an Ethernet interface, an optical interface, a coaxial interface, an infrared interface, a radio frequency (RF) interface, a universal serial bus (USB) interface, etc. Interfaces and / or cellular network interfaces, etc.

[0086] In some embodiments, device 300 performs one or more of the processes described herein. Device 300 performs these processes based on software instructions stored in a computer-readable medium, such as memory 306 and / or storage component 308, executed by computer processor 304. Computer-readable medium (e.g., non-transitory computer-readable medium) is defined herein as a non-transitory memory device. A non-transitory memory device includes storage space located within a single physical storage device or storage space distributed across multiple physical storage devices.

[0087] In some embodiments, software instructions are read from another computer-readable medium or from another device via communication interface 314 into memory 306 and / or storage component 308. When executed, the software instructions stored in memory 306 and / or storage component 308 cause computer processor 304 to perform one or more processes described herein. Additionally or alternatively, hard-wired circuitry may be used in place of or in combination with software instructions to perform one or more processes described herein. Therefore, unless explicitly stated otherwise, the embodiments described herein are not limited to any particular combination of hardware circuitry and software.

[0088] The memory 306 and / or storage component 308 include a data storage unit or at least one data structure (e.g., a database). The device 300 is capable of receiving information from the data storage unit or at least one data structure in the memory 306 or storage component 308, storing the information in the data storage unit or at least one data structure, communicating information to the data storage unit or at least one data structure, or searching for information stored in the data storage unit or at least one data structure. In some examples, the information includes network data, input data, output data, or any combination thereof.

[0089] In some embodiments, device 300 is configured to execute software instructions stored in memory of memory 306 and / or another device (e.g., another device identical or similar to device 300). As used herein, the term "module" refers to at least one instruction stored in memory of memory 306 and / or another device that, when executed by computer processor 304 and / or a computer processor of another device (e.g., another device identical or similar to device 300), causes device 300 (e.g., at least one component of device 300) to perform one or more processes as described herein. In some embodiments, modules are implemented as software, firmware, and / or hardware, etc.

[0090] supply Figure 3 The number and arrangement of components are illustrated as examples. In some embodiments, with Figure 3Compared to the illustrated components, device 300 may include additional components, fewer components, different components, or components arranged differently. Additionally or alternatively, a group of components of device 300 (e.g., one or more components) may perform one or more functions described as being performed by another component or another group of components of device 300.

[0091] Now for reference Figure 4 The diagram illustrates an example block diagram of an autonomous vehicle computing 400 (sometimes referred to as an "AV stack"). As illustrated, the autonomous vehicle computing 400 includes a perception system 402 (sometimes referred to as a perception module), a planning system 404 (sometimes referred to as a planning module), a positioning system 406 (sometimes referred to as a positioning module), a control system 408 (sometimes referred to as a control module), and a database 410. In some embodiments, the perception system 402, planning system 404, positioning system 406, control system 408, and database 410 are included in and / or implemented in the vehicle's automatic navigation system (e.g., the autonomous vehicle computing 202f of vehicle 200). Additionally or alternatively, in some embodiments, the perception system 402, planning system 404, positioning system 406, control system 408, and database 410 are included in one or more separate systems (e.g., one or more systems that are the same as or similar to the autonomous vehicle computing 400, etc.). In some examples, the perception system 402, planning system 404, positioning system 406, control system 408, and database 41 are included in one or more independent systems located within the vehicle and / or at least one remote system as described herein. In some embodiments, any and / or all of the systems included in the autonomous vehicle computing 400 are implemented in software (e.g., software instructions stored in memory), computer hardware (e.g., via microprocessors, microcontrollers, application-specific integrated circuits (ASICs), and / or field-programmable gate arrays (FPGAs), etc.), or a combination of computer software and computer hardware. It will also be understood that in some embodiments, the autonomous vehicle computing 400 is configured to communicate with remote systems (e.g., autonomous vehicle systems identical or similar to remote AV system 114, queue management systems identical or similar to queue management systems 116, and / or V2I systems identical or similar to V2I system 118, etc.).

[0092] In some embodiments, the perception system 402 receives data associated with at least one physical object in the environment (e.g., data used by the perception system 402 to detect at least one physical object) and classifies the at least one physical object. In some examples, the perception system 402 receives image data captured by at least one camera (e.g., camera 202a) that is associated with one or more physical objects within the field of view of the at least one camera (e.g., representing the one or more physical objects). In such examples, the perception system 402 classifies at least one physical object based on one or more groups of physical objects (e.g., bicycles, vehicles, traffic signs, and / or pedestrians, etc.). In some embodiments, based on the classification of physical objects by the perception system 402, the perception system 402 transmits data associated with the classification of the physical objects to the planning system 404.

[0093] In some embodiments, the planning system 404 receives data associated with a destination and generates data associated with at least one route (e.g., route 106) along which a vehicle (e.g., vehicle 102) can travel toward the destination. In some embodiments, the planning system 404 periodically or continuously receives data from the sensing system 402 (e.g., the data associated with the classification of physical objects described above), and the planning system 404 updates at least one trajectory or generates at least one different trajectory based on the data generated by the sensing system 402. In some embodiments, the planning system 404 receives data associated with the updated location of the vehicle (e.g., vehicle 102) from the positioning system 406, and the planning system 404 updates at least one trajectory or generates at least one different trajectory based on the data generated by the positioning system 406.

[0094] In some embodiments, positioning system 406 receives data associated with (e.g., representing) a location of a vehicle (e.g., vehicle 102) in an area. In some examples, positioning system 406 receives LiDAR data associated with at least one point cloud generated by at least one LiDAR sensor (e.g., LiDAR sensor 202b). In some examples, positioning system 406 receives data associated with at least one point cloud from multiple LiDAR sensors, and positioning system 406 generates a composite point cloud based on the individual point clouds. In these examples, positioning system 406 compares the at least one point cloud or composite point cloud with a two-dimensional (2D) and / or three-dimensional (3D) map of the area stored in database 410. Then, based on the comparison of the at least one point cloud or composite point cloud with the map, positioning system 406 determines the location of the vehicle in the area. In some embodiments, the map includes a composite point cloud of the area generated prior to navigation of the vehicle. In some embodiments, the map includes, but is not limited to, a high-precision map of the geometry of the roadway, a map describing the connectivity of the road network, a map describing the physical properties of the roadway (such as traffic speed, traffic flow, the number of vehicle and bicycle lanes, lane width, lane traffic direction, or the type and location of lane markings, or combinations thereof), and a map describing the spatial locations of road features (such as pedestrian crossings, traffic signs, or various types of other traffic lights). In some embodiments, the map is generated in real time based on data received by the sensing system.

[0095] In another example, positioning system 406 receives Global Navigation Satellite System (GNSS) data generated by a Global Positioning System (GPS) receiver. In some examples, positioning system 406 receives GNSS data associated with the location of a vehicle in an area, and positioning system 406 determines the latitude and longitude of the vehicle in the area. In such examples, positioning system 406 determines the location of the vehicle in the area based on the latitude and longitude of the vehicle. In some embodiments, positioning system 406 generates data associated with the location of the vehicle. In some examples, based on the location of the vehicle determined by positioning system 406, positioning system 406 generates data associated with the location of the vehicle. In such examples, the data associated with the location of the vehicle includes data associated with one or more semantic properties corresponding to the location of the vehicle.

[0096] In some embodiments, the control system 408 receives data associated with at least one trajectory from the planning system 404, and the control system 408 controls the operation of the vehicle. In some examples, the control system 408 receives data associated with at least one trajectory from the planning system 404, and the control system 408 controls the operation of the vehicle by generating and transmitting control signals to operate the powertrain control system (e.g., DBW system 202h and / or powertrain control system 204, etc.), the steering control system (e.g., steering control system 206), and / or the braking system (e.g., braking system 208). In an example, where the trajectory includes a left turn, the control system 408 transmits control signals to cause the steering control system 206 to adjust the steering angle of the vehicle 200, thereby causing the vehicle 200 to turn left. Additionally or alternatively, the control system 408 generates and transmits control signals to change the state of other devices of the vehicle 200 (e.g., headlights, turn signals, door locks, and / or windshield wipers, etc.).

[0097] In some embodiments, the perception system 402, planning system 404, positioning system 406, and / or control system 408 implement at least one machine learning model (e.g., at least one multilayer perceptron (MLP), at least one convolutional neural network (CNN), at least one recurrent neural network (RNN), at least one autoencoder, and / or at least one transformer, etc.). In some examples, the perception system 402, planning system 404, positioning system 406, and / or control system 408 implement at least one machine learning model individually or in combination with one or more of the aforementioned systems. In some examples, the perception system 402, planning system 404, positioning system 406, and / or control system 408 implement at least one machine learning model as part of a pipeline (e.g., a pipeline for identifying one or more objects located in the environment, etc.).

[0098] Database 410 stores data transmitted to, received from, and / or updated by the sensing system 402, planning system 404, positioning system 406, and / or control system 408. In some examples, database 410 includes storage components for storing operation-related data and / or software, and for computing 400 using autonomous vehicles (e.g., with...). Figure 3(The storage component 308 is the same as or similar to the storage component 308). In some embodiments, database 410 stores data associated with 2D and / or 3D maps of at least one area. In some examples, database 410 stores data associated with 2D and / or 3D maps of a part of a city, multiple parts of multiple cities, multiple cities, counties, states, and / or countries (e.g., countries). In such examples, a vehicle (e.g., the same as or similar to vehicle 102 and / or vehicle 200) can drive along one or more drivable areas (e.g., single-lane roads, multi-lane roads, highways, remote roads, and / or off-road roads, etc.) and causes at least one LiDAR sensor (e.g., the same as or similar to LiDAR sensor 202b) to generate data associated with images representing objects included in the field of view of the at least one LiDAR sensor.

[0099] In some embodiments, database 410 may be implemented across multiple devices. In some examples, database 410 includes a vehicle (e.g., a vehicle identical or similar to vehicle 102 and / or vehicle 200), an autonomous vehicle system (e.g., an autonomous vehicle system identical or similar to remote AV system 114), and a queue management system (e.g., with...). Figure 1 Queue management system 116 (same as or similar to queue management system) and / or V2I system (e.g., with Figure 1 Among the V2I systems (118 similar to or similar V2I systems), etc.

[0100] System Model Overview

[0101] Figure 5An example framework 500 for modeling subsystems of an AV system and an overview of the impact of these subsystems on AV performance are provided. Framework 500 includes computer simulations of a core AV model 504 using three iterative loops: a scenario loop 501, a design loop 502, and a Monte Carlo loop 503. For each iteration of scenario loop 501, a new scenario (e.g., lane changing, collision avoidance) is selected for simulation. For each iteration of design loop 502 within scenario loop 501, a design configuration of the core AV model 504 (e.g., selecting a specific planning, localization, and perception subsystem design) is selected for simulation using the selected scenario. For each iteration of Monte Carlo loop 503 within design loop 502 and scenario loop 501, Monte Carlo simulations are performed on the self-propelled vehicle and one or more agents in a virtual world environment. Monte Carlo simulations are used to predict the probabilities of different outcomes when uncertainties exist in the selected design configuration, to account for the impact of risk and uncertainty on the AV performance of the selected scenario. In an embodiment, Monte Carlo simulations are used to identify statistically significant rule violations in the selected design configuration and scenario.

[0102] Figure 6 An example core AV model 504 is shown. In this embodiment, the core AV model 504 is a high-level system model designed to capture the relationships between subsystems at a sufficiently deep level to enable AV stack developers to make informative design decisions and ensure that a sufficient number of potential solutions are explored in the design space. In this embodiment, the core AV model 504 includes a perception subsystem 600, a positioning subsystem 601, a trajectory proposer 602, a trajectory selector 603, and a predictor 604. The perception subsystem 600 and the positioning subsystem 601 may, for example, be a reference... Figure 4 The aforementioned sensing system 402 and positioning system 406. The combination of trajectory proposer 602 and trajectory selector 603 can be, for example, a combination of planning system 404 and control system 408.

[0103] The perception subsystem 600 takes as input ground real-time information the pose (e.g., position, heading, velocity, acceleration) and class (e.g., pedestrian, driver, cyclist) of at least one agent, and outputs an estimated current agent pose based on a specified error distribution of agent class and pose. In embodiments, the performance of the perception subsystem 600 is characterized by recall over distance, a confusion matrix (e.g., class error distribution), an error matrix with respect to pose estimation, and the latency of communication between subsystems. Recall is defined as the probability that an agent is detected using at least one sensor of a self-propelled vehicle. (See reference...) Figure 7The delay is defined as the difference between the time it takes for the perception subsystem 600 to detect the agent and the time it takes to send a message with the agent's pose and class from the perception subsystem 600 to the planner subsystem (605, 605).

[0104] In this embodiment, the perception subsystem 600 comprises three distinct functions: detection, classification, and estimation. The detection function follows a Markov process with only two states (detected or not detected, as described) at each time step of the simulation. The classification and estimation functions are activated only when an object is detected, and error models are used to track their respective error matrices. The estimated pose is sampled using a multivariate Gaussian distribution centered on the agent's ground-real pose, where the covariance matrix provides the pose variance. Some example parameters of the perception subsystem 600 include, but are not limited to: recall (defined above as a function of distance and object detection at previous time steps), a list of agent classes and a confusion matrix between classes, a covariance matrix of object states used for state estimation, computation time, and computation cycle (the time taken to complete the task).

[0105] The positioning subsystem 601 is modeled as a "black box" and takes ground-based real-time self-propelled vehicle attitude (e.g., position, heading, velocity, acceleration) as input and outputs an estimated current self-propelled vehicle attitude based on a specified error distribution of the self-propelled vehicle attitude. The self-propelled vehicle attitude is characterized by an error matrix relating the attitude estimate and a time delay (as defined above). The estimated attitude is sampled using a multivariate Gaussian distribution centered on the ground-based real-time attitude, where the covariance matrix provides the variance. In another embodiment, instead of directly using the covariance matrix, it can be reconstructed using eigenvalues ​​representing the size of an ellipsoid of uncertainty surrounding the attitude estimate and eigenvectors representing the orientation of the axes of that ellipsoid. For example, the covariance Sigma can be reconstructed as: Sigma = P^-1DP, where P is a matrix of eigenvectors (in order, each eigenvector being a column), and D is a diagonal matrix of eigenvalues ​​(in order).

[0106] In this embodiment, the parameters of the positioning subsystem 601 include, but are not limited to, the covariance matrix of the autonomous vehicle's attitude error, computation time, and computation cycle. In this embodiment, the positioning subsystem 601 acquires sensor errors and outputs a positioning performance metric (e.g., accuracy or uncertainty around attitude estimation). In this embodiment, the positioning performance metric can be approximated using a factor graph representation, where each factor represents a sensor pipeline (e.g., IMU, LiDAR).

[0107] In this embodiment, trajectory proposer 602 and trajectory selector 603 together provide a "propose and select" architecture for proposing multiple candidate trajectories and selecting one trajectory based on a score determined by the number of rulebook violations made by the self-propelled vehicle due to the trajectory. More specifically, trajectory proposer 602 takes an estimated current self-propelled vehicle pose and an estimated current agent pose and class as input and proposes N candidate trajectories for the self-propelled vehicle to take in a specific driving scenario. The "quality" of a particular trajectory proposer is measured by the degree of violation of the proposed best candidate trajectory (hereinafter also referred to as the "rule violation score"), which can be assumed to be a fixed number K. Trajector 602 is also associated with latency, which is the difference between the time of receiving messages from positioning subsystem 601 and trajectory proposer 602 and the time when the trajectory is proposed. The more trajectories proposed, the better the k best trajectories will be, but the slower trajectory selector 603 will be in selecting a trajectory as the execution path.

[0108] For reference Figure 8 More specifically, trajectory selector 603 takes N candidate trajectories and an estimated future agent position provided by predictor 604 as input, and selects one of the N candidate trajectories as the execution path for the autonomous vehicle for a specific driving scenario based on rule violation scores. Given its imperfect input, trajectory selector 603 uses rule violation scores to determine which of the proposed k candidate trajectories to choose as the execution path for the autonomous vehicle. In an embodiment, in the first iteration, lexicographical comparison is used to select the trajectory with the fewest violations of the highest-level rules in the total order rulebook (such as the rulebook described in Liability, Ethics, and Culture-Aware Behavior Specification Using Rulebooks, https: / / arxiv.org / pdf / 1902.09455).

[0109] Figure 7A latency model 700 is shown for modeling latency in the AV stack. In this embodiment, discrete event simulation is used to model latency in the message queuing network, where different event types are: “message creation / task completion computation,” “message transmission completion,” “task start using the message,” and “task completion using the message and creating a new message.” The goal of latency model 700 is to model the latency of the AV stack at a sufficiently deep level to evaluate the impact of different task cycles for each task on rule violations in a given scenario. The clock advances each time an event (e.g., perception start, selection end, message arrival, etc.) occurs. For the perception, localization, and trajectory proposer subsystems, each time one of these functions is activated, the event is randomly selected from a given probability distribution (e.g., using the Metropolis algorithm).

[0110] Figure 8 An example rulebook 800 is shown for determining rule violation scores for candidate trajectories. The performance of a self-propelled vehicle can be defined in different ways based on the exemplary rulebook 800. In one embodiment, the rules in the rulebook are ordered hierarchically by importance, with the highest (i.e., most important rules) R1a, R1b, R1c, and R1d used to avoid and mitigate collisions, and the lowest or least important rule R4b used to meet a minimum speed limit. In some cases, two or more rules may have the same level of importance, such as R5 (smooth driving) and R3C (lane driving), etc. The k candidate trajectories output by the trajectory proposer 602 are each evaluated using the rulebook by counting the number of rule violations and assigning rule violation scores to each candidate trajectory, where a higher score indicates a better candidate trajectory, and the candidate trajectory with the highest rule violation score is the execution path. In another embodiment, for example, a lexicographical comparison can be used to rank the number of rule violations without assigning scores.

[0111] Figure 9 and Figure 10 An example scenario is shown where the autonomous vehicle 901 must avoid collisions with pedestrian 902 and maintain a specified distance. The perception subsystem 600 generates a perception estimate 904, and the positioning subsystem 601 generates a positioning estimate 903.

[0112] Figure 11A The modeling of the perception subsystem recall (agent detection) using Markov chains is shown, where P1 is the probability of recognizing a traveler and P2 is the probability of not recognizing traveler 902.

[0113] Figure 11BThis is a plot of the probability P of identifying pedestrian 902 at the current and previous steps in the simulation relative to the distance (in meters) to the self-propelled vehicle. Ground reality recall is modeled by line 1001, where data points 1002 and 1003 (the probabilities of identifying pedestrian 902 at times t and t-1) are given by the following formula:

[0114] P(Recognized(t)|Recognized(t-1)), and

[0115] P(Recognized(t)|NotRecognized(t-1)).

[0116] Figure 12 The diagram illustrates ground-based real-time recall model 1001 and recall models 1003 and 1004. Recall models 1003 and 1004 model the probability of recognizing pedestrian 902 at the current time step if pedestrian 902 was identified at a previous time step, and the probability of recognizing pedestrian 902 at the current time step if pedestrian 902 was not identified at a previous time step. In this example, as expected, if pedestrian 902 is detected at t-1, P(X=1) increases by 0.40027073, and if pedestrian 902 is not identified at t-1, P(X=1) decreases by 0.5213696, where X is the distance relative to pedestrian 902.

[0117] Note that the model used for detecting the agent / object is the slope of lines 1003 and 1004 (i.e., a linear equation). At each time step of the simulation, the distance relative to pedestrian 902 is calculated and used in models 1003 and 1004 (linear equations) to determine the probability of detecting pedestrian 902. The model used depends on whether pedestrian 902 was detected in a previous time step of the simulation. For example, if pedestrian 902 was not detected at t-1, model 1003 will be used to determine the detection probability at the current time step. If pedestrian 902 was detected at t-1, model 1004 will be used to determine the detection probability at the current time step.

[0118] Figure 13The modeling of trajectory proposer 602 and trajectory selector 603 is shown, which together are used to model planner subsystem 404. The input to trajectory proposer subsystem 602 is the perturbation output from sensing subsystem 600 and positioning subsystem 601. In this embodiment, trajectory proposals are randomized. One parameter is the number of proposed trajectories N, which can be the number of "good" trajectories k, where "good" is relative to the best rulebook trajectory computed offline using ideal parameters. Other parameters of the model are computation time and task cycle. In the example shown, N = 5, and candidate trajectories are labeled 1-5. Candidate trajectory 2 is selected as the best trajectory by trajectory selector 603.

[0119] Figure 14A This is a bar graph showing the violation score of Rule R2 relative to the detection probability at 50 meters, where Rule R2 is used to maintain distance from pedestrians in scenarios where pedestrians are in the lane of a self-driving vehicle. Figure 8 In this example, the trajectory selection period is 300ms, the perception task period is 400ms, and the longitudinal positioning error is negligible. Figure 14A The bar chart shows that there is not much gain between 0.56 and 0.95 (all other things being equal), where it is desirable to be at or above 0.56 to ensure that rule R2 is not violated.

[0120] Figure 14B This is a plot of the perception period (ms) versus the recall error slope for a pedestrian spacing scenario. This plot attempts to answer the question of whether a system with fast perception is better than one with higher recall. Data points are shaded based on the number of R2 violations, with darker shades representing a higher number of R2 violations compared to lighter shades. In this example scenario, a mild trend is observed where a worse error slope and a higher perception period correlate with a higher number of R2 violations. This is evident in the density of darker data points in the upper right corner of the plot. However, since other parameters also vary, this example plot shows that AV performance is revealed based on the intersection of all system performance metrics.

[0121] Figure 14C This is an example plot of positioning error versus selection period (ms). This plot attempts to answer the question of whether AV performance improves with more accurate positioning or a faster selector. From this plot, it can be concluded that AV performance depends on more than just these two parameters. Therefore, it is unclear whether AV performance will improve with more accurate positioning or a faster selector.

[0122] Figure 14DThis is an example plot of the slope of localization error relative to recall error. This plot attempts to answer the question of whether engineering resources should be used to improve perception by X% or localization by Y%. From this plot, it can be concluded that AV performance depends on more than just these two parameters, as there is no clear trend in the data. Therefore, for example, it is unclear whether more engineering resources should be allocated to improve perception by X% or localization by Y%.

[0123] Figure 15 A heatmap showing the correlations of design variables is presented. The plot shows that the two design variables most correlated with high R2 rule violations (darker gray shading) are recall error and perceived activation cycle.

[0124] Figure 16 This is an example plot of rule R1a violation relative to pedestrian spacing violation on road R2 (i.e., the severity of rule R1 violation as a function of R2 violation). The plot illustrates the rule trade-offs where a design that results in a low R2 violation also results in a low R1 violation.

[0125] Figure 17 This is an example plot of the severity / degree of R2 violation (pedestrian spacing) as a function of the severity / degree of rule R5 violation (smooth driving). In this example, most candidate trajectories do not have an R5 violation. Therefore, complying with rule R2 does not necessarily mean violating rule R5. Thus, a trajectory that avoids a collision with pedestrian 902 with appropriate spacing can also be executed by smoothly maneuvering the AV.

[0126] Figure 18 This is an example plot of the severity / degree of the R109 violation (lane separation) as a function of the severity / degree of the R6 violation. In this example, spacing, lane separation, and destination arrival cannot be maintained simultaneously. Therefore, the desired architecture has no spacing violation and a low destination arrival violation.

[0127] Example Application

[0128] Some example applications of the framework described in this paper include, but are not limited to: determining where to place research and development engineering resources in AV stack development, technology roadmaps for future products, platform approaches that minimize rework when serving different market segments, optimization of AV performance taking into account cost / energy constraints, and real-time health monitoring of AVs to identify when subsystem performance drops below requirements and assess the resulting impact on driving behavior performance.

[0129] Figure 19 This illustrates an application of a framework for determining system, subsystem, and sensor-level performance targets. For example... Figure 19As shown, each subsystem (planning, localization, sensing, control) can be decomposed into lower-level models to answer questions such as which sensors to choose and which processor can produce lower latency at the system level. In the example shown, the framework includes a high-level system model 1901 (reference). Figures 5 to 18 The high-level system model 1901 can be used to determine perception performance targets 1902, positioning performance targets 1903, planning and control performance targets 1904, and latency performance targets 1905. A lower-level individual function model 1906 can then be simulated to meet performance targets 1902 to 1905. Based on the lower-level individual function model 1906, sensor performance and individual electronic control unit (ECU) latency targets 1907 can be determined.

[0130] Figure 20 This illustrates an application of a framework for selecting the AV architecture with the highest performance from multiple candidate AV architectures. The high-level system model 1901 can be used to evaluate the AV performance of various AV stack designs, thereby enabling the selection of candidate architectures with the highest AV performance (e.g., collision avoidance) among, for example, higher-level rules, allowing for the plotting of trade-offs and Pareto curves between achievable rule violations (e.g., comfort relative to time to destination).

[0131] Figure 21 This is a flowchart of process 2100 used to determine the interdependencies of the vehicle's subsystems. Process 2100 can be referenced as follows. Figure 5 and Figure 6 Implement it as described.

[0132] In an embodiment, processing 2100 includes the following steps: selecting a scenario for simulating a vehicle and at least one agent in a virtual world environment (2101); selecting a set of subsystem component models for the vehicle (2102), wherein these subsystems include a positioning subsystem component model, a perception subsystem component model, a trajectory proposer subsystem component model, and a trajectory selector subsystem component model; and using the selected subsystem component models to simulate the scenario in the virtual world environment (2103), wherein the simulation includes: estimating the attitude of the vehicle and at least one agent based on the positioning subsystem component model; and estimating the attitude of the vehicle and at least one agent based on the perception subsystem component model, the estimated attitude of the vehicle, and the estimated attitude of the at least one agent. The vehicle's attitude is calculated, and a latency model is used to model the latency of a message queuing network for communication data between subsystems to determine the probability of detecting at least one agent using at least one sensor of the vehicle; a set of candidate trajectories for the vehicle is generated based on a trajectory proposer subsystem component model to avoid collisions with at least one agent; the trajectory is evaluated based on the number of rule violations associated with each trajectory; a trajectory is selected from the set of trajectories based on the number of rule violations of each candidate trajectory, based on a trajectory selector subsystem component model (2104); and the output of the subsystem component model is analyzed to determine the interdependencies between two or more subsystems of the vehicle (2105). Reference Figures 5 to 20 Each step in these steps is described in detail.

[0133] In the preceding description, aspects and embodiments of this disclosure have been described with reference to numerous specific details, which may vary from implementation to implementation. Therefore, the specification and drawings should be considered illustrative rather than restrictive. The sole and exclusive indication of the scope of this invention, and what the applicant expects to be the scope of this invention, is the literal and equivalent scope of the claims published from this application in the specific form of the published claims, including any subsequent amendments. Any definitions of terms expressly set forth herein for inclusion in such claims should be taken as meaning as such terms are used in the claims. Furthermore, when the term “comprising” is used in the preceding specification or appended claims, what follows that phrase may be an additional step or entity, or a sub-step / sub-entity of a previously stated step or entity.

Claims

1. A method for determining the interdependencies between subsystems of a vehicle, comprising: Using at least one processor, select a scenario for simulating a vehicle and at least one intelligent agent in a virtual world environment; Using the at least one processor, a set of subsystem component models are selected for the vehicle, wherein the subsystem component models include a positioning subsystem component model, a sensing subsystem component model, a trajectory proposer subsystem component model, and a trajectory selector subsystem component model; Using the at least one processor, the scene is simulated in the virtual world environment using a selected subsystem component model, wherein the simulation includes: The attitude of the vehicle and the at least one intelligent agent is estimated based on the positioning subsystem component model, and Based on the perception subsystem component model, the estimated attitude of the vehicle, the estimated attitude of the at least one agent, and a delay model that models the delay of the message queuing network used for communicating data between subsystems, the detection probability of the at least one agent utilizing at least one sensor of the vehicle is determined. Using the at least one processor, a set of candidate trajectories is generated for the vehicle based on the trajectory proposer subsystem component model to avoid collisions with the at least one agent; Using the at least one processor, the trajectory is evaluated based on the number of rule violations associated with each trajectory; Using the at least one processor, a trajectory is selected from the set of candidate trajectories based on the number of rule violations for each candidate trajectory; and Using the at least one processor, the output of the subsystem component model is analyzed to determine the interdependencies between two or more subsystems of the vehicle.

2. The method of claim 1, wherein, The simulations include Monte Carlo simulations.

3. The method of claim 1, wherein, The latency model includes performing discrete event simulations of the message queuing network and selecting simulated events based on the probability distribution of the events.

4. The method according to claim 1, wherein, The positioning subsystem component model includes the covariance matrix for the vehicle's attitude error, computation time, and task cycle.

5. The method according to claim 1, wherein, The perception subsystem component model includes recall as a function of the distance between the vehicle and the at least one agent and the agent's detection at a previous time step in the simulation, a list of classes, a confusion matrix between classes, and a covariance matrix with respect to the agent's pose.

6. The method according to claim 5, wherein, Recall as a function of distance is based on Markov models.

7. The method according to claim 1, wherein, The set of candidate trajectories is randomly generated.

8. The method according to claim 1, wherein, The trajectory selector subsystem component model includes a rule manual, which is used to rank the set of candidate trajectories and select a specific trajectory from the set of candidate trajectories based on the ranking.

9. The method according to claim 1, wherein, Analyzing the output of the subsystem component model to determine the interdependencies between two or more subsystem component models in the subsystem component model, further includes at least one of the following: Compare the number of rule violations with the detection probability; Compare the sensing period of the sensing subsystem with the recall error slope; The positioning error is compared with the selection time period of the trajectory selector subsystem; Compare the positioning error with the recall error slope; Generate a correlation heatmap that links subsystem model parameters to the number of rule violations; or Compare the degree of violation of at least two rules.

10. A system for determining the interdependencies between subsystems of a vehicle, comprising: At least one processor; as well as A memory for storing instructions that, when executed by the at least one processor, cause the at least one processor to perform the method according to any one of claims 1 to 9.

11. A non-transitory computer-readable storage medium storing instructions that, when executed by one or more processors, cause the one or more processors to perform the method according to any one of claims 1 to 9.

12. A computer program product comprising a computer program configured to perform the method according to any one of claims 1-9 when executed by a processor.

Images