Method of operating a vehicle, autonomous vehicle and medium

Abstract

The present invention relates to a method of operating a vehicle, an autonomous vehicle, and a medium. The method for operating a vehicle using a behavioral rule model includes receiving sensor data from a first set of sensors and a second set of sensors. The sensor data represents the operation of the vehicle for one or more objects. A violation of the behavioral model for the operation of the vehicle is determined based on the sensor data. A first risk level for one or more violations is determined based on an event distribution of the operation of the vehicle for one or more objects. In response to the first risk level being greater than a threshold risk level, a trajectory is generated. The trajectory has a second risk level lower than the threshold risk level. The vehicle is operated based on the trajectory to avoid collision of the vehicle with the one or more objects.

Description

Technical Field

[0001] The present description relates generally to vehicle operation and, more particularly, to vehicle operation using a behavioral rule model. Background Art

[0002] The operation of a vehicle from an initial location to a final destination typically requires a user or a decision system of the vehicle to select a route from the initial location through a road network to the final destination. The route may involve meeting objectives, such as not exceeding a maximum driving time, etc. Complex routes may require many decisions, making traditional algorithms for autonomous driving impractical. Summary of the invention

[0003] Methods, systems and devices for vehicle operation using a behavioral rule model are disclosed. In an embodiment, one or more processors of a vehicle operating in an environment receive first sensor data from a first set of sensors of the vehicle and second sensor data from a second set of sensors of the vehicle. The first sensor data represents the operation of the vehicle, and the second sensor data represents one or more objects located in the environment. The one or more processors determine one or more violations of a stored behavioral model for the operation of the vehicle based on the first sensor data and the second sensor data. The one or more violations are determined for one or more objects located in the environment. The one or more processors determine a first risk level for the one or more violations based on a stored event distribution of the operation of the vehicle relative to the one or more objects. In response to the first risk level being greater than a threshold risk level, the one or more processors generate a trajectory of the vehicle. The trajectory has a second risk level lower than the threshold risk level. The second risk level is determined for the one or more objects. The one or more processors operate the vehicle based on the trajectory to avoid collision of the vehicle with the one or more objects.

[0004] In an embodiment, the first set of sensors includes at least one of an accelerometer, a steering wheel angle sensor, a wheel sensor, and a brake sensor.

[0005] In an embodiment, the first sensor data includes at least one of a velocity of the vehicle, an acceleration of the vehicle, a heading of the vehicle, an angular velocity of the vehicle, and a torque of the vehicle.

[0006] In an embodiment, the second set of sensors includes at least one of LiDAR, RADAR, camera and microphone.

[0007] In an embodiment, the second sensor data includes at least one of an image of the one or more objects, a velocity of the one or more objects, an acceleration of the one or more objects, and a lateral distance between the one or more objects and the vehicle.

[0008] In an embodiment, the one or more processors determine the second risk level based on a distribution of stored events of the trajectory and operation of the vehicle relative to the one or more objects.

[0009] In an embodiment, the storage event distribution comprises a log-normal probability distribution of independent random variables. Each random variable represents a risk level of danger of operation of the vehicle.

[0010] In an embodiment, the stored behavior model of the operation of the vehicle comprises a plurality of operation rules. Each operation rule has a priority relative to each other operation rule. The priority represents the risk level of one or more violations of the stored behavior model.

[0011] In an embodiment, the one or more violations of the stored behavioral model of operation of the vehicle include a lateral distance between the vehicle and the one or more objects falling below a threshold lateral distance.

[0012] In an embodiment, the priority of the operating rules is adjusted based on the frequency of violations.

[0013] In an embodiment, the motion planning process of the vehicle is adjusted to reduce the second risk level based on the frequency of one or more violations of the stored behavior model.

[0014] In an embodiment, a risk level of a motion planning process for a vehicle is determined based on a frequency of one or more violations of stored behavior models.

[0015] In an embodiment, one or more processors of a vehicle operating in an environment generate a trajectory based on first sensor data from a first set of sensors of the vehicle and second sensor data from a second set of sensors of the vehicle. The first sensor data represents the operation of the vehicle, and the second sensor data represents one or more objects located in the environment. The one or more processors determine whether the trajectory causes one or more violations of a stored behavior model for the operation of the vehicle. The one or more violations are determined for one or more objects located in the environment. In response to determining that the trajectory causes one or more violations of the stored behavior model, the one or more processors determine a first risk level for the one or more violations based on a stored event distribution of the operation of the vehicle relative to the one or more objects. The one or more processors generate an alternative trajectory for the vehicle. The one or more processors determine that the alternative trajectory has a second risk level higher than the first risk level. The second risk level is determined for the one or more objects. The one or more processors operate the vehicle based on the trajectory to avoid collision between the vehicle and the one or more objects.

[0016] In an embodiment, the stored behavior model of the operation of the vehicle comprises multiple layers, each layer having a respective position corresponding to a violation of the one or more violations.

[0017] In an embodiment, a collision of the vehicle with the one or more objects occurs when respective positions of respective ones of the plurality of layers are aligned.

[0018] In an embodiment, the motion planning process of the vehicle is designed so that the probability that the corresponding positions of each of the plurality of layers are aligned is less than a threshold probability.

[0019] In an embodiment, a violation of the one or more violations indicates that a deceleration of the vehicle exceeds a threshold deceleration.

[0020] In an embodiment, a violation of the one or more violations indicates that a lateral distance from said vehicle to said one or more objects falls below a threshold lateral distance.

[0021] In an embodiment, the first set of sensors includes at least one of an accelerometer, a steering wheel angle sensor, a wheel sensor, and a brake sensor.

[0022] In an embodiment, the first sensor data includes at least one of a velocity of the vehicle, an acceleration of the vehicle, a heading of the vehicle, an angular velocity of the vehicle, and a torque of the vehicle.

[0023] In an embodiment, the second set of sensors includes at least one of LiDAR, RADAR, camera and microphone.

[0024] In an embodiment, the second sensor data includes at least one of an image of the one or more objects, a velocity of the one or more objects, an acceleration of the one or more objects, and a lateral distance between the one or more objects and the vehicle.

[0025] In an embodiment, the one or more processors determine the second risk level based on the candidate trajectories and a stored event distribution of operations of the vehicle with respect to the one or more objects.

[0026] In an embodiment, the stored event distribution comprises a log-normal probability distribution of independent random variables, each random variable representing a risk level of hazardousness of the operation of the vehicle.

[0027] In an embodiment, a storage behavior model of the operation of a vehicle includes a plurality of operation rules. Each operation rule in the plurality of operation rules has a priority relative to each other operation rule in the plurality of operation rules. The priority represents a risk level of one or more violations of the storage behavior model.

[0028] In an embodiment, a violation of the one or more violations of the stored behavioral model of operation of the vehicle comprises a lateral distance between the vehicle and one or more objects falling below a threshold lateral distance.

[0029] In an embodiment, the priority of the operating rules is adjusted based on the frequency of violations.

[0030] In an embodiment, the motion planning process of the vehicle is adjusted to reduce the second risk level based on the frequency of one or more violations of the stored behavior model.

[0031] In an embodiment, a risk level of a motion planning process for a vehicle is determined based on a frequency of one or more violations of stored behavior models.

[0032] These and other aspects, features, and implementations may be expressed as methods, apparatus, systems, components, program products, means or steps for performing the functions, and in other ways.

[0033] These and other aspects, features and implementations will become apparent from the following description, including the claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 is a block diagram illustrating an example of an autonomous vehicle (AV) having autonomous capabilities in accordance with one or more embodiments.

[0035] Figure 2is a block diagram illustrating an example "cloud" computing environment in accordance with one or more embodiments.

[0036] Figure 3 is a block diagram illustrating a computer system according to one or more embodiments.

[0037] Figure 4 is a block diagram illustrating an example architecture of an AV in accordance with one or more embodiments.

[0038] Figure 5 is a block diagram illustrating examples of inputs and outputs that may be used by a perception module in accordance with one or more embodiments.

[0039] Figure 6 is a block diagram illustrating an example of a LiDAR system according to one or more embodiments.

[0040] Figure 7 is a block diagram illustrating a LiDAR system in operation according to one or more embodiments.

[0041] Figure 8 is a block diagram illustrating additional details of the operation of a LiDAR system according to one or more embodiments.

[0042] Fig. 9 is a block diagram illustrating the relationship between inputs and outputs of a planning module in accordance with one or more embodiments.

[0043] Fig.10 A directed graph for use in path planning is illustrated in accordance with one or more embodiments.

[0044] Fig.11 is a block diagram illustrating inputs and outputs of a control module according to one or more embodiments.

[0045] Fig.12 is a block diagram illustrating inputs, outputs, and components of a controller according to one or more embodiments.

[0046] Fig.13 is a flow chart illustrating an example process for determining whether a trajectory violates a stored behavioral model of operation of a vehicle in accordance with one or more embodiments.

[0047] Fig.14 Example stored behavior models illustrating operation of a vehicle in accordance with one or more embodiments.

[0048] Fig.15 Example frequencies of violations of a stored behavioral model for operation of a vehicle are illustrated in accordance with one or more embodiments.

[0049] Fig.16Example stored behavior models illustrating operation of a vehicle in accordance with one or more embodiments.

[0050] Fig.17 An example storage event distribution is illustrated for operation of a vehicle relative to one or more objects in accordance with one or more embodiments.

[0051] Fig.18 An example storage event distribution is illustrated for operation of a vehicle relative to one or more objects in accordance with one or more embodiments.

[0052] Fig.19 An example storage event distribution is illustrated for operation of a vehicle relative to one or more objects in accordance with one or more embodiments.

[0053] Fig. 20 An example storage event distribution is illustrated for operation of a vehicle relative to one or more objects in accordance with one or more embodiments.

[0054] Fig.21 An example storage event distribution is illustrated for operation of a vehicle relative to one or more objects in accordance with one or more embodiments.

[0055] Fig. 22 is a flow chart illustrating an example process for vehicle operation using a behavioral rule model in accordance with one or more embodiments.

[0056] Fig.23 is a flow chart illustrating an example process for vehicle operation using a behavioral rule model in accordance with one or more embodiments. DETAILED DESCRIPTION

[0057] In the following description, for the purpose of explanation, many specific details are set forth in order to provide a thorough understanding of the present invention. However, it will be apparent that the present invention can be implemented without these specific details. In other examples, well-known configurations and devices are shown in block diagram form to avoid unnecessarily obscuring the present invention.

[0058] In the accompanying drawings, for ease of description, the specific arrangement or order of schematic elements (such as those representing devices, modules, instruction blocks, and data elements) is shown. However, it should be understood by those skilled in the art that the specific order or arrangement of the schematic elements in the accompanying drawings is not intended to mean that a specific processing order or sequence, or separation of processing processes is required. In addition, the inclusion of schematic elements in the accompanying drawings is not intended to mean that such elements are required in all embodiments, nor is it intended to mean that the features represented by such elements cannot be included in some embodiments or cannot be combined with other elements in an embodiment.

[0059] In addition, in the accompanying drawings, connecting elements, such as solid or dotted lines or arrows, are used to illustrate the connection, relationship or association between two or more other schematic elements, and there is no such connecting element and is not intended to mean that there can be no connection, relationship or association. In other words, the connection, relationship or association between some elements are not shown in the accompanying drawings, so as not to make the present disclosure vague. In addition, for the convenience of illustration, a single connecting element is used to represent multiple connections, relationships or associations between elements. For example, if the connecting element represents the communication of signal, data or instruction, it will be understood by those skilled in the art that this element represents one or more signal paths (for example, bus) that may be needed to affect the communication.

[0060] Reference will now be made in detail to the embodiments, examples of which are illustrated in the accompanying drawings. In the following detailed description, numerous specific details are set forth in order to provide a thorough understanding of the various embodiments described. However, it will be apparent to one of ordinary skill in the art that the various embodiments described may be implemented without these specific details. In other cases, well-known methods, procedures, components, circuits, and networks are not described in detail in order not to unnecessarily obscure aspects of the embodiments.

[0061] Several features described below can each be used independently of one another, or can be used together with any combination of other features. However, any individual feature may not solve any of the problems discussed above, or may only solve one of the problems discussed above. Some of the problems discussed above may not be fully solved by any one of the features described herein. Although titles are provided, information related to a specific title but not found in the section with the title may also be found elsewhere in this specification. This article describes an embodiment according to the following summary:

[0062] 1. General Overview

[0063] 2. System Overview

[0064] 3. Autonomous Vehicle Architecture

[0065] 4. Autonomous Vehicle Input

[0066] 5. Autonomous vehicle planning

[0067] 6. Autonomous Vehicle Control

[0068] 7. Autonomous Vehicle Operations Using Behavioral Planning Models

[0069] General Overview

[0070] This article describes methods, systems and devices for vehicle operation using behavioral rule models. Road safety is a major public health issue (global road traffic deaths exceed 1 million by 2020) and is currently the seventh leading cause of death in the United States in terms of years of life lost. However, a key challenge in evaluating the impact of road safety interventions is that individual human drivers rarely crash, so an impractical amount of driving data is required for direct comparison of crash rates for different road safety interventions. This problem is widely applicable when estimating any number of strategies, technologies, or educational interventions to improve road safety. Since human factors are the key cause of most motor vehicle crashes, methods that identify behaviors that lead to higher crash risks can create a path to prevent traffic deaths. In recent years, the challenge of measuring the safety of autonomous vehicles (AVs) relative to human driving baselines has resurfaced long-standing issues about effectively measuring driving safety. Road safety assessments increasingly rely on methods used for other complex safety-critical systems such as aviation and industrial safety; similarly, new methods for evaluating road safety are also applicable to other complex systems.

[0071] Embodiments disclosed herein implement rule-based tools to estimate the performance of a machine driver or human driver, estimate risk factors, and estimate the performance of an AV system or subsystem (such as a motion planning module, etc.). The implementation of behavior-based driving assessment disclosed herein is based on the observation that good drivers always follow behavioral rules. Rules are derived from safety considerations, traffic regulations, or publicly accepted best practices. Driving rulemaking can be used to quantitatively estimate how actual driving by humans or automated systems matches expected driving behavior.

[0072] Advantages and benefits of the embodiments described herein include estimates of driving performance of both autonomous vehicle systems and human drivers. Using the embodiments, specific autonomous driving behaviors can be estimated. In addition, the embodiments are useful for insurance companies (which may reward improvements in risk assessments). In addition, the embodiments disclosed herein can inform various regulatory and standard processes that increasingly require specific AV behaviors and promote industry collaboration on defining good AV driving behaviors.

[0073] System Overview

[0074] Figure 1 is a block diagram illustrating an example of an autonomous vehicle 100 having autonomous capabilities according to one or more embodiments.

[0075] As used herein, the term "autonomous capability" refers to a function, feature, or facility that enables a vehicle to operate partially or fully without real-time human intervention, including but not limited to fully autonomous vehicles, highly autonomous vehicles, and conditionally autonomous vehicles.

[0076] As used herein, an autonomous vehicle (AV) is a vehicle with autonomous capabilities.

[0077] As used herein, "vehicle" includes a mode of transport of goods or people. For example, a car, bus, train, airplane, drone, truck, boat, ship, submersible, spacecraft, etc. An unmanned car is an example of a vehicle.

[0078] As used herein, a "trajectory" refers to a path or route by which an AV is operated from a first spatiotemporal location to a second spatiotemporal location. In embodiments, the first spatiotemporal location is referred to as an initial location or starting location, and the second spatiotemporal location is referred to as a destination, final location, target, target location, or target location. In some examples, a trajectory consists of one or more road segments (e.g., several sections of a road), and each road segment consists of one or more blocks (e.g., a portion of a lane or intersection). In embodiments, a spatiotemporal location corresponds to a real-world location. For example, a spatiotemporal location is a pickup or drop-off location to get people or cargo on or off the vehicle.

[0079] As used herein, "sensor(s)" includes one or more hardware components for detecting information about the environment surrounding the sensor. Some hardware components may include sensing components (e.g., image sensors, biometric sensors), transmission and / or reception components (e.g., laser or radio frequency wave transmitters and receivers), electronic components (such as analog-to-digital converters), data storage devices (such as RAM and / or non-volatile memory), software or firmware components and data processing components (such as application specific integrated circuits), microprocessors and / or microcontrollers.

[0080] As used herein, a “scene description” is a data structure (e.g., a list) or data stream that includes one or more categorized or labeled objects detected by one or more sensors on an AV vehicle, or provided by a source external to the AV.

[0081] As used herein, a "road" is a physical area that can be traversed by a vehicle and may correspond to a named passage (e.g., a city street, an interstate highway, etc.) or may correspond to an unnamed passage (e.g., a driveway within a house or office building, a section of a parking lot, a section of a vacant parking lot, a dirt passage in a rural area, etc.). Because some vehicles (e.g., four-wheel drive pickup trucks, off-road vehicles (SUVs), etc.) are able to traverse a variety of physical areas that are not particularly suitable for vehicle travel, a "road" may be a physical area that is not formally defined as a passage.

[0082] As used herein, a "lane" is a portion of a road that can be traversed by vehicles, and may correspond to most or all of the space between lane markings, or to only some of the space between lane markings (e.g., less than 50%). For example, a road with lane markings that are far apart may accommodate two or more vehicles between the markings, such that one vehicle can pass another without crossing the lane markings, and thus may be interpreted as having a lane that is narrower than the space between the lane markings, or as having two lanes between the lanes. Lanes may also be interpreted in the absence of lane markings. For example, lanes may be defined based on physical features of the environment, such as rocks and trees along a pathway in a rural area.

[0083] “One or more” includes a function performed by one element, a function performed by multiple elements, such as in a distributed manner, several functions performed by one element, several functions performed by several elements, or any combination of the above.

[0084] It will also be understood that although in some cases, the terms "first," "second," etc. are used herein to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another. For example, without departing from the scope of the various described embodiments, a first contact may be referred to as a second contact, and similarly, a second contact may be referred to as a first contact. Both the first contact and the second contact are contacts, but they are not the same contact.

[0085] The terms used in the specification of the various embodiments described herein are only used for the purpose of describing specific embodiments and are not intended to be limiting. As used in the specification of the various embodiments described and the appended claims, the singular forms "a", "an" and "the" are also intended to include plural forms, unless the context clearly indicates otherwise. It will also be understood that "and / or" as used herein refers to and includes any and all possible combinations of one or more related list items. It will also be understood that when the terms "include", "comprises", "have" and / or "have" are used in this specification, the stated features, integers, steps, operations, elements and / or components are specifically stated, but the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof are not excluded.

[0086] As used herein, the term "if" is alternatively understood to mean "when" or "at the time" or "in response to determining that" or "in response to detecting," depending on the context. Similarly, the phrases "if it has been determined" or "if [the stated condition or event] has been detected" are alternatively understood to mean "upon determination" or "in response to determining that" or "upon detecting [the stated condition or event]" or "in response to detecting [the stated condition or event]," depending on the context.

[0087] As used herein, an AV system refers to an AV and the array of hardware, software, stored data, and real-time generated data that supports the operation of the AV. In an embodiment, the AV system is incorporated into the AV. In an embodiment, the AV system is distributed across several locations. For example, some software of the AV system is in a system similar to the following about Figure 2 The described cloud computing environment 200 is implemented in a cloud computing environment.

[0088] In general, this document describes technologies applicable to any vehicle having one or more autonomous capabilities, including fully autonomous vehicles, highly autonomous vehicles, and conditionally autonomous vehicles, such as so-called Level 5, Level 4, and Level 3 vehicles, respectively (see SAE International Standard J3016: Classification and Definitions of Terms Relating to Automated Driving Systems for On-Road Motor Vehicles, which is incorporated by reference in its entirety into this document for more details on vehicle autonomy levels). The technologies described in this document are also applicable to partially autonomous vehicles and driver-assisted vehicles, such as so-called Level 2 and Level 1 vehicles (see SAE International Standard J3016: Classification and Definitions of Terms Relating to Automated Driving Systems for On-Road Motor Vehicles). In embodiments, one or more Level 1, Level 2, Level 3, Level 4, and Level 5 vehicle systems may automatically perform certain vehicle operations (e.g., steering, braking, and using maps) under certain operating conditions based on processing of sensor inputs. The technologies described in this document may benefit any level of vehicle ranging from fully autonomous vehicles to human-operated vehicles.

[0089] refer to Figure 1 , the AV system 120 causes the AV 100 to operate along a trajectory 198, through an environment 190 to a destination 199 (sometimes referred to as a final location), while avoiding objects (e.g., natural obstacles 191, vehicles 193, pedestrians 192, cyclists, and other obstacles) and complying with road rules (e.g., operating rules or driving preferences).

[0090] In an embodiment, the AV system 120 includes a device 101 equipped to receive and operate operating commands from a computer processor 146. In an embodiment, the computing processor 146 is associated with the following reference Figure 3The processor 304 is similarly described. Examples of devices 101 include steering controls 102, brakes 103, gears, an accelerator pedal or other acceleration control mechanism, windshield wipers, side door locks, window controls, and turn indicators.

[0091] In an embodiment, the AV system 120 includes sensors 121 for measuring or inferring attributes of the state or condition of the AV 100, such as the position, linear and angular velocity and acceleration, and heading (e.g., the direction of the front end of the AV 100) of the AV. Examples of sensors 121 are GNSS, an inertial measurement unit (IMU) that measures both linear acceleration and angular rate of the vehicle, wheel sensors for measuring or estimating wheel slip, wheel brake pressure or brake torque sensors, engine torque or wheel torque sensors, and steering angle and angular rate sensors.

[0092] In an embodiment, the sensors 121 also include sensors for sensing or measuring properties of the environment of the AV, such as monocular or stereo cameras 122 in the visible, infrared, or thermal (or both) spectrum, LiDAR 123, RADAR, ultrasonic sensors, time-of-flight (TOF) depth sensors, velocity sensors, temperature sensors, humidity sensors, and precipitation sensors.

[0093] In an embodiment, the AV system 120 includes a data storage unit 142 and a memory 144 for storing machine instructions associated with a computer processor 146 or data collected by the sensor 121. In an embodiment, the data storage unit 142 is associated with the following Figure 3 190. In an embodiment, the memory 144 is similar to the main memory 306 described below. In an embodiment, the data storage unit 142 and the memory 144 store historical, real-time, and / or predictive information about the environment 190. In an embodiment, the stored information includes maps, driving performance, traffic congestion updates, or weather conditions. In an embodiment, data related to the environment 190 is transmitted from the remote database 134 to the AV 100 via a communication channel.

[0094] In an embodiment, the AV system 120 includes a communication device 140 for transmitting measured or inferred properties of the state and condition of other vehicles (such as position, linear and angular velocity, linear and angular acceleration, and linear and angular heading) to the AV 100. These devices include vehicle-to-vehicle (V2V) and vehicle-to-infrastructure (V2I) communication devices and devices for wireless communication through point-to-point or ad hoc networks or both. In an embodiment, the communication device 140 communicates across the electromagnetic spectrum (including radio and optical communications) or other media (e.g., air and acoustic media). The combination of vehicle-to-vehicle (V2V), vehicle-to-infrastructure (V2I) communications (and in an embodiment, one or more other types of communications) is sometimes referred to as vehicle-to-everything (V2X) communications. V2X communications typically comply with one or more communication standards for communication with and between autonomous vehicles.

[0095] In an embodiment, the communication device 140 includes a communication interface. For example, a wired, wireless, WiMAX, Wi-Fi, Bluetooth, satellite, cellular, optical, near field, infrared, or radio interface. The communication interface transmits data from the remote database 134 to the AV system 120. In an embodiment, the remote database 134 is embedded in a Figure 2 The communication interface 140 transmits data collected from the sensor 121 or other data related to the operation of the AV 100 to the remote database 134. In an embodiment, the communication interface 140 transmits information related to teleoperation to the AV 100. In an embodiment, the AV 100 communicates with other remote (e.g., "cloud") servers 136.

[0096] In an embodiment, the remote database 134 also stores and transmits digital data (e.g., stores data such as roads and street locations). Such data is stored in the memory 144 on the AV 100 or transmitted from the remote database 134 to the AV 100 via a communication channel.

[0097] In an embodiment, the remote database 134 stores and transmits historical information (e.g., speed and acceleration profiles) related to driving attributes of vehicles that have previously traveled along the trajectory 198 at similar times of day. In one implementation, such data may be stored in the memory 144 on the AV 100 or transmitted from the remote database 134 to the AV 100 via a communication channel.

[0098] The computing device 146 located on the AV 100 algorithmically generates control actions based on both real-time sensor data and a priori information, allowing the AV system 120 to perform its autonomous driving capabilities.

[0099] In an embodiment, the AV system 120 includes a computer peripheral device 132 coupled to the computing device 146 for providing information and reminders to a user of the AV 100 (e.g., an occupant or a remote user) and receiving input from the user. In an embodiment, the peripheral device 132 is similar to the one described below with reference to Figure 3 Display 312, input device 314 and cursor control 316 are discussed. The coupling may be wireless or wired. Any two or more of the interface devices may be integrated into a single device.

[0100] Sample cloud computing environment

[0101] Figure 2 is a block diagram illustrating an example "cloud" computing environment according to one or more embodiments. Cloud computing is a service delivery model for enabling convenient, on-demand access to a shared pool of configurable computing resources (e.g., networks, network bandwidth, servers, processing, memory, storage, applications, virtual machines, and services) over a network. In a typical cloud computing system, one or more large cloud data centers house the machines used to deliver the services provided by the cloud. Now referring to Figure 2 , cloud computing environment 200 includes cloud data centers 204a, 204b, and 204c interconnected by cloud 202. Data centers 204a, 204b, and 204c provide cloud computing services for computer systems 206a, 206b, 206c, 206d, 206e, and 206f connected to cloud 202.

[0102] The cloud computing environment 200 includes one or more cloud data centers. Figure 2 The cloud data center 204a shown in FIG. 2 refers to a cloud (eg, Figure 2 The physical arrangement of servers in a cloud 202 or a specific portion of a cloud (as shown in FIG. 202 or a specific portion of a cloud). For example, servers are physically arranged into rooms, groups, rows, and racks in a cloud data center. A cloud data center has one or more zones, which include one or more server rooms. Each room has one or more rows of servers, and each row includes one or more racks. Each rack includes one or more individual server nodes. In some implementations, servers in zones, rooms, racks, and / or rows are arranged into groups based on the physical infrastructure requirements of the data center facility, including power, energy, heat, heat sources, and / or other requirements. In an embodiment, server nodes are similar to Figure 3 The data center 204a has many computing systems distributed across multiple racks.

[0103] Cloud 202 includes cloud data centers 204a, 204b, and 204c and networks and network resources (e.g., network devices, nodes, routers, switches, and network cables) used to connect cloud data centers 204a, 204b, and 204c and help facilitate access to cloud computing services by computing systems 206a-f. In an embodiment, the network represents any combination of one or more local networks, wide area networks, or internetworks coupled by wired or wireless links deployed using terrestrial or satellite connections. Data exchanged through the network is transmitted using a variety of network layer protocols (such as Internet Protocol (IP), Multi-Protocol Label Switching (MPLS), Asynchronous Transfer Mode (ATM), Frame Relay, etc.). In addition, in embodiments where the network represents a combination of multiple subnetworks, different network layer protocols are used on each underlying subnetwork. In an embodiment, the network represents one or more interconnected internetworks (such as the public Internet, etc.).

[0104] Computing systems 206a-f or cloud computing service consumers are connected to the cloud 202 through network links and network adapters. In an embodiment, computing systems 206a-f are implemented as various computing devices, such as servers, desktops, laptops, tablets, smart phones, Internet of Things (IoT) devices, autonomous vehicles (including cars, drones, shuttles, trains, buses, etc.) and consumer electronics. In an embodiment, computing systems 206a-f are implemented in other systems or as part of other systems.

[0105] Computer Systems

[0106] Figure 3 300 is a block diagram illustrating a computer system 300 according to one or more embodiments. In implementation, the computer system 300 is a special-purpose computing device. The special-purpose computing device is hard-wired to perform these techniques, or includes a digital electronic device such as one or more application-specific integrated circuits (ASICs) or field programmable gate arrays (FPGAs) that are permanently programmed to perform the above-mentioned techniques, or may include one or more general-purpose hardware processors that are programmed to perform these techniques according to program instructions in firmware, memory, other memory, or a combination. Such a special-purpose computing device can also combine customized hard-wired logic, ASICs or FPGAs with customized programming to complete these techniques. In various embodiments, the special-purpose computing device is a desktop computer system, a portable computer system, a handheld device, a network device, or any other device that contains hard-wired and / or program logic to implement these techniques.

[0107] In an embodiment, computer system 300 includes a bus 302 or other communication mechanism for communicating information, and a hardware processor 304 coupled to bus 302 to process information. Hardware processor 304 is, for example, a general-purpose microprocessor. Computer system 300 also includes a main memory 306, such as a random access memory (RAM) or other dynamic storage device, coupled to bus 302 to store information and instructions, which are executed by processor 304. In one implementation, main memory 306 is used to store temporary variables or other intermediate information during the execution of instructions to be executed by processor 304. When these instructions are stored in a non-transitory storage medium accessible to processor 304, computer system 300 becomes a special-purpose machine that is customized to perform the operations specified in the instructions.

[0108] In an embodiment, computer system 300 also includes a read only memory (ROM) 308 or other static storage device coupled to bus 302 for storing static information and instructions for processor 304. A storage device 310, such as a magnetic disk, optical disk, solid state drive, or three-dimensional cross point memory, is provided and coupled to bus 302 to store information and instructions.

[0109] In an embodiment, the computer system 300 is coupled to a display 312 such as a cathode ray tube (CRT), a liquid crystal display (LCD), a plasma display, a light emitting diode (LED) display, or an organic light emitting diode (OLED) display for displaying information to a computer user via bus 302. An input device 314 including alphanumeric and other keys is coupled to bus 302 for communicating information and command selections to processor 304. Another type of user input device is a cursor controller 316, such as a mouse, a trackball, a touch display, or cursor direction keys, for communicating direction information and command selections to processor 304 and for controlling movement of a cursor on display 312. Such input devices typically have two degrees of freedom in two axes, a first axis (e.g., an x-axis) and a second axis (e.g., a y-axis) that allow the device to specify a position on a plane.

[0110] According to one embodiment, the techniques herein are performed by computer system 300 in response to processor 304 executing one or more sequences of one or more instructions contained in main memory 306. These instructions are read into main memory 306 from another storage medium, such as storage device 310. Execution of the sequences of instructions contained in main memory 306 causes processor 304 to perform the process steps described herein. In alternative embodiments, hard-wired circuitry is used in place of or in combination with software instructions.

[0111] As used herein, the term "storage medium" refers to any non-temporary medium that stores data and / or instructions that cause a machine to operate in a particular manner. Such storage media include non-volatile media and / or volatile media. Non-volatile media include, for example, optical disks, magnetic disks, solid-state drives, or three-dimensional cross-point memories such as storage device 310. Volatile media include dynamic memories such as main memory 306. Common forms of storage media include, for example, floppy disks, floppy disks, hard disks, solid-state drives, tapes or any other magnetic data storage media, CD-ROMs, any other optical data storage media, any physical media with a hole pattern, RAM, PROM and EPROM, FLASH-EPROM, NV-RAM, or any other memory chip or storage box.

[0112] Storage media are distinct from transmission media, but can be used in conjunction with transmission media. Transmission media participate in the transmission of information between storage media. For example, transmission media include coaxial cables, copper wires, and optical fibers, including wires that provide bus 302. Transmission media can also take the form of sound waves or light waves, such as those generated during radio wave and infrared data communications.

[0113] In an embodiment, various forms of media are involved in carrying one or more sequences of one or more instructions to the processor 304 for execution. For example, the instructions are initially executed on a disk or solid-state drive of a remote computer. The remote computer loads the instructions into its dynamic memory and sends the instructions over a telephone line using a modem. The local modem of the computer system 300 receives the data on the telephone line and uses an infrared transmitter to convert the data to an infrared signal. An infrared detector receives the data carried in the infrared signal, and appropriate circuitry places the data on the bus 302. The bus 302 carries the data to the main memory 306, from which the processor 304 retrieves and executes the instructions. The instructions received by the main memory 306 may optionally be stored on the storage device 310 before or after execution by the processor 304.

[0114] Computer system 300 also includes a communication interface 318 coupled to bus 302. Communication interface 318 provides a two-way data communication coupled to a network link 320 connected to a local network 322. For example, communication interface 318 is an integrated services digital network (ISDN) card, a cable modem, a satellite modem, or a modem for providing a data communication connection with a corresponding type of telephone line. As another example, communication interface 318 is a local area network (LAN) card for providing a data communication connection with a compatible LAN. In some implementations, a wireless link is also implemented. In any such implementation, communication interface 318 sends and receives an electrical, electromagnetic or optical signal carrying a digital data stream representing various types of information.

[0115] The network link 320 typically provides data communication to other data devices through one or more networks. For example, the network link 320 provides a connection to a host computer 324 or to a cloud data center or device operated by an Internet Service Provider (ISP) 326 through a local network 322. The ISP 326, in turn, provides data communication services through a worldwide packet data communication network now commonly referred to as the "Internet" 328. Both the local network 322 and the Internet 328 use electrical, electromagnetic, or optical signals that carry digital data streams. The signals through the various networks and the signals on the network link 320 and through the communication interface 318 are example forms of transmission media, where these signals carry digital data to and from the computer system 300. In an embodiment, the network 320 includes the cloud 202 described above or a portion of the cloud 202.

[0116] Computer system 300 sends messages and receives data, including program code, through network(s), network link 320, and communication interface 318. In an embodiment, computer system 300 receives code for processing. The received code is executed by processor 304 as it is received, and / or stored in storage device 310, or other non-volatile storage for later execution.

[0117] Autonomous Vehicle Architecture

[0118] Figure 4 is an example of a method for an autonomous vehicle (e.g., Figure 1 100). The architecture 400 includes a sensing module 402 (sometimes referred to as sensing circuitry), a planning module 404 (sometimes referred to as planning circuitry), a control module 406 (sometimes referred to as control circuitry), a positioning module 408 (sometimes referred to as positioning circuitry), and a database module 410 (sometimes referred to as database circuitry). Each module plays a role in the operation of the AV 100. Collectively, the modules 402, 404, 406, 408, and 410 may be Figure 1 4 is a portion of the AV system 120 shown. In an embodiment, any of the modules 402, 404, 406, 408, and 410 is a combination of computer software (e.g., executable code stored on a computer-readable medium) and computer hardware (e.g., one or more microprocessors, microcontrollers, application specific integrated circuits [ASICs], hardware memory devices, other types of integrated circuits, other types of computer hardware, or a combination of any or all of these).

[0119] In use, the planning module 404 receives data representing a destination 412 and determines data representing a trajectory 414 (sometimes referred to as a route) that the AV 100 may travel in order to reach (e.g., arrive at) the destination 412. In order for the planning module 404 to determine the data representing the trajectory 414, the planning module 404 receives data from the perception module 402, the positioning module 408, and the database module 410.

[0120] The perception module 402 uses, for example, Figure 1 One or more sensors 121 are shown to identify nearby physical objects. Objects are classified (e.g., grouped into types such as pedestrians, bicycles, cars, traffic signs, etc.), and a scene description including the classified objects 416 is provided to the planning module 404.

[0121] The planning module 404 also receives data representing the AV position 418 from the positioning module 408. The positioning module 408 determines the AV position by using data from the sensor 121 and data from the database module 410 (e.g., geographic data) to calculate the position. For example, the positioning module 408 uses data from a global navigation satellite system (GNSS) unit and geographic data to calculate the longitude and latitude of the AV. In an embodiment, the data used by the positioning module 408 includes a high-precision map with lane geometry attributes, a map describing the road network connection attributes, a map describing the physical attributes of the lane (such as traffic speed, traffic volume, the number of vehicles and bicycle lanes, lane width, lane traffic direction, or lane marking type and location, or a combination thereof), and a map describing the spatial location of road features (such as intersections, traffic signs or various types of other driving signals, etc.).

[0122] The control module 406 receives data representing the trajectory 414 and data representing the AV position 418, and operates control functions 420a-420c (e.g., steering, throttle, brakes, ignition) of the AV in a manner that will cause the AV 100 to travel the trajectory 414 to reach the destination 412. For example, if the trajectory 414 includes a left turn, the control module 406 will operate the control functions 420a-420c in the following manner: the steering angle of the steering function will cause the AV 100 to turn left, and the throttle and brakes will cause the AV 100 to pause and wait for a passing pedestrian or vehicle before making the turn.

[0123] Autonomous Vehicle Input

[0124] Figure 5 is an example of a perception module 402 ( Figure 4 ) used by the inputs 502a-502d (e.g., Figure 11) and outputs 504a-504d (e.g., sensor data). One input 502a is a LiDAR (Light Detection and Ranging) system (e.g., Figure 1 123). LiDAR is a technology that uses light (e.g., a beam of light such as infrared light) to obtain data about physical objects in its line of sight. The LiDAR system produces LiDAR data as output 504a. For example, LiDAR data is a collection of 3D or 2D points (also called a point cloud) used to construct a representation of the environment 190.

[0125] Another input 502b is a RADAR (radar) system. RADAR is a technology that uses radio waves to obtain data related to nearby physical objects. RADAR can obtain data related to objects that are not within the line of sight of the LiDAR system. The RADAR system 502b produces RADAR data as output 504b. For example, RADAR data is one or more radio frequency electromagnetic signals used to construct a representation of the environment 190.

[0126] Another input 502c is a camera system. The camera system uses one or more cameras (e.g., a digital camera using a light sensor such as a charge coupled device [CCD]) to obtain information about nearby physical objects. The camera system generates camera data as output 504c. The camera data is typically in the form of image data (e.g., data in an image data format such as RAW, JPEG, PNG, etc.). In some examples, the camera system has multiple independent cameras, such as for the purpose of stereoscopic imaging (stereoscopic vision), which enables the camera system to perceive depth. Although the objects perceived by the camera system are described here as "nearby", this is relative to the AV. In use, the camera system can be configured to "see" distant objects (e.g., as far as 1 km or more in front of the AV). Therefore, the camera system can have features such as sensors and lenses that are optimized for perceiving distant objects.

[0127] Another input 502d is a traffic light detection (TLD) system. The TLD system uses one or more cameras to obtain information about traffic lights, street signs, and other physical objects that provide visual operational information. The TLD system generates TLD data as an output 504d. The TLD data often takes the form of image data (e.g., data in an image data format such as RAW, JPEG, PNG, etc.). The TLD system differs from a system that includes a camera in that the TLD system uses a camera with a wide field of view (e.g., using a wide-angle lens or a fisheye lens) to obtain information about as many physical objects that provide visual operational information as possible, so that the AV 100 can access all relevant operational information provided by these objects. For example, the viewing angle of the TLD system can be about 120 degrees or greater.

[0128] In an embodiment, the outputs 504a-504d are combined using sensor fusion techniques. Thus, the individual outputs 504a-504d are provided to other systems of the AV 100 (e.g., to Figure 4 The combined output may be provided to other systems in the form of a single combined output or multiple combined outputs of the same type (e.g., using the same combining technique or combining the same outputs, or both) or a single combined output or multiple combined outputs of different types (e.g., using different individual combining techniques or combining different individual outputs, or both). In an embodiment, an early fusion technique is used. The early fusion technique is characterized in that the outputs are combined before one or more data processing steps are applied to the combined output. In an embodiment, a late fusion technique is used. The late fusion technique is characterized in that the outputs are combined after one or more data processing steps are applied to the individual outputs.

[0129] Figure 6 is an example of a LiDAR system 602 (eg, Figure 56a-604c). The LiDAR system 602 emits light 604a-604c from a light emitter 606 (e.g., a laser emitter). The light emitted by the LiDAR system is typically not in the visible spectrum; for example, infrared light is often used. Some of the emitted light 604b encounters a physical object 608 (e.g., a vehicle) and reflects back to the LiDAR system 602. (The light emitted from the LiDAR system typically does not penetrate a physical object, such as a physical object that is solid in form.) The LiDAR system 602 also has one or more light detectors 610 for detecting the reflected light. In an embodiment, one or more data processing systems associated with the LiDAR system generate an image 612 representing a field of view 614 of the LiDAR system. The image 612 includes information representing a boundary 616 of the physical object 608. In this way, the image 612 is used to determine the boundary 616 of one or more physical objects near the AV.

[0130] Figure 7 is a block diagram illustrating a LiDAR system 602 in operation according to one or more embodiments. In the scenario shown in the figure, the AV 100 receives both the camera system output 504c in the form of an image 702 and the LiDAR system output 504a in the form of LiDAR data points 704. In use, the data processing system of the AV 100 compares the image 702 with the data points 704. In particular, the physical objects 706 identified in the image 702 are also identified in the data points 704. In this way, the AV 100 perceives the boundaries of the physical objects based on the contours and density of the data points 704.

[0131] Figure 8 is a block diagram illustrating additional details of the operation of the LiDAR system 602 according to one or more embodiments. As described above, the AV 100 detects the boundaries of a physical object based on the characteristics of the data points detected by the LiDAR system 602. Figure 8 As shown, a flat object such as the ground 802 will reflect light 804a-804d emitted from the LiDAR system 602 in a consistent manner. In other words, because the LiDAR system 602 emits light using consistent intervals, the ground 802 will reflect light back to the LiDAR system 602 at the same consistent intervals. As the AV 100 drives over the ground 802, the LiDAR system 602 will continue to detect light reflected by the next valid ground point 806 if nothing is blocking the road. However, if an object 808 is blocking the road, the light 804e-804f emitted by the LiDAR system 602 will reflect from points 810a-810b in a manner that is inconsistent with the expected consistent manner. Based on this information, the AV 100 can determine that the object 808 is present.

[0132] Path Planning

[0133] Fig. 9 is an example of a method according to one or more embodiments (for example, Figure 4 900 of the relationship between the input and output of the planning module 404 (shown in FIG. 1 ). In general, the output of the planning module 404 is a route 902 from a starting point 904 (e.g., a source location or initial location) to an end point 906 (e.g., a destination or final location). The route 902 is typically defined by one or more road segments. For example, a road segment refers to a distance to be traveled on at least a portion of a street, road, highway, lane, or other physical area suitable for automobile travel. In some examples, for example, if the AV 100 is an off-road capable vehicle such as a four-wheel drive (4WD) or all-wheel drive (AWD) car, SUV, or pickup truck, the route 902 includes "off-road" road segments such as unpaved paths or open fields.

[0134] In addition to the route 902, the planning module also outputs lane-level route planning data 908. The lane-level route planning data 908 is used to drive through the road segments of the route 902 at a specific time based on the conditions of the road segments. For example, if the route 902 includes a multi-lane highway, the lane-level route planning data 908 includes trajectory planning data 910, wherein the AV 100 can use the trajectory planning data 910 to select a lane from the multiple lanes based on, for example, whether an exit is approaching, whether there are other vehicles in one or more of the multiple lanes, or other factors that change over the course of a few minutes or less. Similarly, in some implementations, the lane-level route planning data 908 includes a speed constraint 912 that is specific to a road segment of the route 902. For example, if the road segment includes pedestrians or unexpected traffic, the speed constraint 912 can limit the AV 100 to a travel speed that is slower than the expected speed, such as a speed based on the speed limit data of the road segment.

[0135] In an embodiment, inputs to the planning module 404 include (e.g., Figure 4 ) database data 914, current location data 916 (e.g., Figure 4 AV position 418 shown), (e.g., for Figure 4 Destination data 918 and object data 920 (e.g., such as the destination 412 shown) Figure 44 (a) and (b) a schematic diagram of ...

[0136] Fig.10 In path planning (eg, by planning module 404 ( Figure 4 )) uses a directed graph 1000. In general, if Fig.10 A directed graph 1000 such as the one shown is used to determine a path between any origin 1002 and destination 1004. In the real world, the distance separating the origin 1002 and destination 1004 may be relatively large (e.g., in two different metropolitan areas), or may be relatively small (e.g., two intersections adjacent to a city block or two lanes of a multi-lane road).

[0137] In an embodiment, directed graph 1000 has nodes 1006a-1006d representing different locations that AV 100 may occupy between starting point 1002 and end point 1004. In some examples, for example, when starting point 1002 and end point 1004 represent different metropolitan areas, nodes 1006a-1006d represent sections of a road. In some examples, for example, when starting point 1002 and end point 1004 represent different locations on the same road, nodes 1006a-1006d represent different locations on the road. In this way, directed graph 1000 includes information at different granularity levels. In an embodiment, a directed graph with high granularity is also a subgraph of another directed graph with a larger scale. For example, most of the information for a directed graph where the start point 1002 and the end point 1004 are far apart (e.g., many miles apart) is at a low granularity and the directed graph is based on stored data, but the directed graph also includes some high granularity information for a portion of the directed graph that represents a physical location in the field of view of the AV 100.

[0138] Nodes 1006a-1006d are different from objects 1008a-1008b that cannot overlap with nodes. In an embodiment, at low granularity, objects 1008a-1008b represent areas that cars cannot pass through, such as areas without streets or roads. At high granularity, objects 1008a-1008b represent physical objects in the field of view of AV 100, such as other cars, pedestrians, or other entities with which AV 100 cannot share physical space. In an embodiment, some or all of objects 1008a-1008b are static objects (e.g., objects that do not change position, such as street lights or telephone poles, etc.) or dynamic objects (e.g., objects that can change position, such as pedestrians or other cars, etc.).

[0139] Nodes 1006a-1006d are connected by edges 1010a-1010c. If two nodes 1006a-1006b are connected by edge 1010a, the AV 100 can travel between one node 1006a and another node 1006b, for example, without having to travel to an intermediate node before reaching the other node 1006b. (When referring to the AV 100 traveling between nodes, it means that the AV 100 travels between two physical locations represented by the corresponding nodes.) Edges 1010a-1010c are generally bidirectional, in the sense that the AV 100 travels from a first node to a second node, or from a second node to a first node. In an embodiment, edges 1010a-1010c are unidirectional, in the sense that the AV 100 can travel from a first node to a second node, but the AV 100 cannot travel from a second node to the first node. Edges 1010a-1010c are unidirectional where they represent, for example, a one-way street, a single lane of a street, road, or highway, or other features that can only be traversed in one direction due to legal or physical constraints.

[0140] In an embodiment, the planning module 404 uses the directed graph 1000 to identify a path 1012 consisting of nodes and edges between the start point 1002 and the end point 1004 .

[0141] Edges 1010a-1010c have associated costs 1014a-1014b. Costs 1014a-1014b are values ​​representing resources that would be expended if the AV 100 selected that edge. A typical resource is time. For example, if the physical distance represented by one edge 1010a is twice the physical distance represented by another edge 1010b, the associated cost 1014a of the first edge 1010a may be twice the associated cost 1014b of the second edge 1010b. Other factors that affect time include expected traffic, number of intersections, speed limits, etc. Another typical resource is fuel economy. Two edges 1010a-1010b may represent the same physical distance, but one edge 1010a requires more fuel than the other edge 1010b, for example, due to road conditions, expected weather, etc.

[0142] When the planning module 404 identifies a path 1012 between the start point 1002 and the end point 1004, the planning module 404 typically selects a path that is optimized for cost, eg, a path that has a minimum total cost when the individual costs of the edges are added together.

[0143] Autonomous Vehicle Control

[0144] Fig.11 is an example of a method according to one or more embodiments (for example, Figure 4 1100 of the inputs and outputs of the control module 406 (shown in FIG. 110). The control module operates according to a controller 1102, which includes, for example: one or more processors similar to the processor 304 (e.g., one or more computer processors such as a microprocessor or a microcontroller or both); short-term and / or long-term data storage devices similar to the main memory 306, ROM 308, and storage devices 310 (e.g., memory, random access memory, or flash memory or both); and instructions stored in the memory that, when executed (e.g., by the one or more processors), perform the operations of the controller 1102.

[0145] In an embodiment, the controller 1102 receives data representing a desired output 1104. The desired output 1104 typically includes speed, such as velocity and heading. The desired output 1104 may be based on, for example, Figure 4104 . Based on the desired output 1104, the controller 1102 generates data that can be used as a throttle input 1106 and a steering input 1108. The throttle input 1106 represents the amount by which the throttle of the AV 100 (e.g., an acceleration control) should be engaged to achieve the desired output 1104, such as by engaging a steering pedal or engaging another throttle control. In some examples, the throttle input 1106 also includes data that can be used to engage the brakes of the AV 100 (e.g., a deceleration control). The steering input 1108 represents a steering angle, such as the angle at which the steering control of the AV (e.g., a steering wheel, a steering angle actuator, or other function for controlling the steering angle) should be positioned to achieve the desired output 1104.

[0146] In an embodiment, the controller 1102 receives feedback used in adjusting the inputs provided to the throttle and steering. For example, if the AV 100 encounters a disturbance 1110, such as a hill, the measured velocity 1112 of the AV 100 drops below the desired output rate. In an embodiment, any measured output 1114 is provided to the controller 1102 so that the required adjustments can be made, for example, based on the difference 1113 between the measured velocity and the desired output. The measured output 1114 includes measured position 1116, measured velocity 1118 (including velocity and heading), measured acceleration 1120, and other outputs that the sensors of the AV 100 can measure.

[0147] In an embodiment, information about the disturbance 1110 is detected in advance, for example, by a sensor such as a camera or LiDAR sensor, and provided to the predictive feedback module 1122. The predictive feedback module 1122 then provides information to the controller 1102 that the controller 1102 can use to adjust accordingly. For example, if the sensors of the AV 100 detect ("see") a hill, the controller 1102 can use this information to prepare to engage the throttle at an appropriate time to avoid significant deceleration.

[0148] Fig.12 1 is a block diagram 1200 illustrating inputs, outputs, and components of a controller 1102 according to one or more embodiments. The controller 1102 has a rate analyzer 1204 that affects the operation of a throttle / brake controller 1204. For example, the rate analyzer 1202 instructs the throttle / brake controller 1204 to use a throttle / brake 1206 to accelerate or decelerate based on, for example, feedback received by the controller 1102 and processed by the rate analyzer 1202.

[0149] Controller 1102 also has a lateral tracking controller 1208 that affects the operation of steering wheel controller 1210. For example, lateral tracking controller 1208 instructs steering wheel controller 1210 to adjust the position of steering angle actuator 1212 based on feedback received by controller 1102 and processed by lateral tracking controller 1208, for example.

[0150] The controller 1102 receives several inputs used to determine how to control the throttle / brake 1206 and the steering angle actuator 1212. The planning module 404 provides information used by the controller 1102, for example, to select a heading for the AV 100 to begin operation and to determine which road segment to cross when the AV 100 reaches an intersection. The positioning module 408 provides information describing the current location of the AV 100 to the controller 1102, for example, so that the controller 1102 can determine whether the AV 100 is in the expected location based on the way the throttle / brake 1206 and the steering angle actuator 1212 are being controlled. In an embodiment, the controller 1102 receives information from other inputs 1214, such as information received from a database, a computer network, etc.

[0151] Vehicle Operations Using Behavioral Rule Models

[0152] Fig.13 is a flow chart illustrating a process for determining whether a trajectory violates a stored behavioral model of the operation of the AV 100 according to one or more embodiments. Figure 1 The AV 100 is illustrated and described in more detail. The AV 100 uses a stored behavioral model of the operation of the AV 100 to provide feedback on the AV driving performance. The stored behavioral model is sometimes referred to as a rule book. In some embodiments, feedback is provided in a pass-fail manner. Fig.13 The processing is designed to recognize when the AV 100 generates a trajectory that violates the rules, when a trajectory is available for the AV 100 that is significantly better than the generated trajectory, and the preferred trajectory.

[0153] AV 100 operates in environment 190. Reference Figure 1 The environment 190 is illustrated and described in more detail. In an embodiment, one or more processors 146 of the AV 100 generate the trajectory 198. Figure 1 The processor 146 and the trajectory 198 are illustrated and described in further detail. The trajectory 198 is generated based on first sensor data from a first set of sensors (e.g., sensors 121) of the AV 100 and second sensor data from a second set of sensors (e.g., sensors 122) of the AV 100. Figure 1The sensors 121, 122 are illustrated and described in further detail. The first sensor data represents the operation of the AV 100, and the second sensor data represents one or more objects 146 located in the environment 190. Figure 4 The objects are illustrated and described in more detail. In an embodiment, the first set of sensors 121 includes at least one of an accelerometer, a steering wheel angle sensor, a wheel sensor, and a brake sensor. The first sensor data includes at least one of a speed of the vehicle, an acceleration of the vehicle, a heading of the vehicle, an angular velocity of the vehicle, and a torque of the vehicle.

[0154] In an embodiment, one or more processors determine whether trace 198 causes one or more violations of a stored behavior model of the operation of AV 100. Fig.14 An example storage behavior model is illustrated and described in more detail. One or more violations are determined for one or more objects 416 located in the environment 190. For example, a criterion is defined for marking a trajectory 198 as a possible failure. A simple criterion is a violation of a single rule, and other criterion formulations are also possible. For example, given a possible or actual trajectory 198 generated by the planning module 404 of the AV 100, Fig.13 The processing provides feedback on the trajectory 198 in terms of the appropriateness of the driving behavior. Figure 4 The planning module 404 is illustrated and described in further detail.

[0155] In an embodiment, in response to determining that the trajectory 198 causes one or more violations of the storage behavior model, the one or more processors 146 determine a first risk level of the one or more violations based on the operation of the AV 100 relative to the storage event distribution of the one or more objects 416. Events are sometimes referred to as "hazards". Fig.15 Storing event distributions is illustrated and described in further detail. The one or more processors 146 generate candidate trajectories for the AV 100. For example, a set of candidate trajectories is created to be evaluated against the labeled trajectories 198.

[0156] In an embodiment, the one or more processors 146 determine that the alternative trajectory has a second risk level that is higher than the first risk level. The second risk level is determined for the one or more objects 416. For example, possible results of feedback from the risk determination are "pass" (i.e., the trajectory 198 is satisfactory, or there is no better alternative trajectory), or "fail" (i.e., the AV trajectory 198 does not meet the rule book behavior specification and there is a substantially better alternative trajectory available). If a substantially better trajectory is identified, the trajectory 198 is considered to be a "fail". A formalization of what constitutes a substantially better trajectory is used. Fig.13 The processing is not used to identify minor or mild improvements.

[0157] Fig.13 The processing is designed to prevent a "mildly satisfactory" trajectory (i.e., a trajectory where the AV 100 will stop or will not reach its target) from being considered a better solution than a trajectory 198 that reaches the target with some rule violations. The rules for "reaching the target" are explicitly built into the rule book. The one or more processors 146 operate the AV 100 based on the trajectory 198 to avoid collisions between the AV 100 and one or more objects 416. For example, referring to Figure 4 The control module 406 illustrated and described in greater detail operates the AV 100 .

[0158] Fig.14 A storage behavior model illustrating the operation of the AV 100 according to one or more embodiments. Figure 1 The AV 100 is illustrated and described in more detail. In an embodiment, the one or more processors 146 of the AV 100 receive first sensor data from a first set of sensors 121 of the AV 100 and second sensor data from a second set of sensors 122 of the AV 100. Figure 1 The processor 146 and sensors 121, 122 are illustrated and described in further detail. The first sensor data represents the operation of the AV 100, and the second sensor data represents one or more objects 416 located in the environment 190. Figure 4 Object 416 is illustrated and described in more detail. Figure 1 The environment 190 is illustrated and described in more detail. In an embodiment, the second set of sensors 122 includes at least one of LiDAR, RADAR, camera, and microphone. The second sensor data includes at least one of an image of one or more objects 416, a velocity of one or more objects 416, an acceleration of one or more objects 416, or a lateral distance between one or more objects 416 and the AV 100.

[0159] In an embodiment, the one or more processors 146 determine one or more violations of a stored behavioral model of operation of the AV 100 based on the first sensor data and the second sensor data. The one or more violations are determined for one or more objects 416 located in the environment 190. For example, vehicle-based sensors or external sensors such as sensors configured on the AV 100 record information related to the scene in which the AV 100 is involved and the driver's responsive driving behavior, including but not limited to speed, heading, neighboring objects, or route.

[0160] In an embodiment, the one or more processors 146 determine a first risk level for one or more violations based on the operation of the AV 100 relative to the storage event distribution of the one or more objects 416. Fig.15An example storage event distribution is illustrated and described in more detail. For example, given a record, a rule book (rule making and rule evaluation) is used to determine whether the driving behavior follows or violates the rules. Rule violations may be related to safety outcomes in both human driving systems and autonomous driving systems.

[0161] In response to the first risk level being greater than the threshold risk level, the one or more processors generate a trajectory 198 for the AV 100. Figure 1 The trajectory 198 is illustrated and described in further detail. The trajectory has a second risk level that is lower than a threshold risk level. The second risk level is determined for one or more objects 416. The one or more processors 146 operate the AV 100 based on the trajectory 198 to avoid collision of the AV 100 with the one or more objects 416.

[0162] In an embodiment, the motion planning processing of the planning module 404 is adjusted based on the frequency of one or more violations of the stored behavior models to reduce the second risk level. For example, a validated rule book is applied to design and implement an automated vehicle system or to perform a "risk score" of a human driver for insurance or public safety purposes. In the case of a machine driver that typically has a system model, the driving performance can use the rule book to estimate the AV driving performance. In an embodiment, the risk level of the motion planning processing of the AV 100 is determined based on the frequency of one or more violations of the stored behavior models. For example, if Fig.14 As shown, the impact of system design and subsystem performance on the planned trajectory is modeled. Based on the system design and subsystem performance, the planned trajectory is scored to measure the overall driving performance. (Sub)system requirements are derived from the behavior specifications (rules), performance is optimized, and resources are prioritized.

[0163] Fig.15 100 illustrates the violation frequency of a storage behavior model for the operation of AV 100 according to one or more embodiments. Figure 1 AV 100 is illustrated and described in more detail. Fig.14 An example storage behavior model of operation is illustrated and described in more detail. In an embodiment, the one or more processors 146 determine a second risk level based on the trajectory 198 and the storage event distribution of the operation of the AV 100 relative to the one or more objects 416. Figure 1 The one or more processors 146 and the trace 198 are illustrated and described in further detail. Figure 4 One or more objects 416 are illustrated and described in further detail.

[0164] Fig.15The violation frequency of the stored behavior model for the operations exemplified in models the way the AV 100 should behave on public roads. Complex scenarios require trade-offs between different behaviors when regulations are frequently unspecified and appropriate behaviors for all scenarios are difficult to enumerate even given perfect information. Fig.14 The stored behavioral model of operation exemplified in is used to measure AV system-level performance relative to human drivers for crashes and safety envelope violations. For example, there is an inverse relationship between the frequency and severity of events. There is a verifiable quantitative relationship associated with events of different severity. Events originate from generating reference Fig.16 A single generation process of a distribution may be characterized as illustrated and described in more detail.

[0165] The embodiments disclosed herein enable the analysis of severity distributions using few continuous measures of crash severity. While there are detailed statistics regarding the relative prevalence of crashes resulting in death, injury, and property damage only, these categories are discrete in nature and quantitative scaling limit analysis of crash severity distributions is lacking. The disclosed embodiments thus consider a continuous crash severity distribution. Four data sets using different proxies for severity are used to test whether the severity distribution of safety critical road events is consistent with the Fig.13 and 15 The model in is consistent.

[0166] The first example data set used is the National Automobile Sampling System's Crashworthiness Data System ("NASS CDS"). This is an ongoing data collection effort to study, reconstruct, and classify random samples of all reported collisions in the United States that are severe enough to require a tow truck. One of the reported collision characteristics is Delta-V, which is defined as "the change in speed between the pre- and post-collision trajectories of the vehicle" and is an authoritative measure of accident severity, widely considered to be the best predictor of injuries and deaths in vehicle collisions. Unlike the collision severity rating, Delta-V can take a continuum of values. For example, a data set of 6,286 collisions between 2000 and 2011 was analyzed, with records from the event data recorders of the vehicles involved. Delta-V is determined by taking the Euclidean norm of the maximum Delta-V reported during the collision event in the lateral and longitudinal directions. Many events in this data set report a Delta-V of 0 miles per hour, which appears to be false because the collision implies some rate differences. Partly to eliminate this possible spurious data, incidents where the Delta-V was less than 5 miles per hour were discarded, which were unlikely to result in tow-away collisions and represented well below 10% of the dataset values. Other example datasets analyzed include insurance claims datasets.

[0167] Fig.16 A storage behavior model illustrating the operation of the AV 100 according to one or more embodiments. Figure 1 AV 100 is illustrated and described in further detail. In an embodiment, a storage behavior model of the operation of AV 100 includes a plurality of layers. Each layer has a respective position corresponding to a particular violation of the one or more violations. Fig.16 Illustrate a framework for relating safety incidents of varying severity based on the observation of an almost constant ratio of high-severity incidents to lower-severity incidents. The implication is that a focus on reducing minor incidents, near misses, and dangerous conditions results in a proportional reduction in major incidents (accidents).

[0168] In an embodiment, a collision of the AV 100 with the one or more objects 416 occurs when the corresponding positions of the layers in the plurality of layers are aligned. Fig.16 The storage behavior model considers the safety of complex systems to be composed of multiple layers, albeit with some holes in the layers that represent faults (failures). Fig.16 The model shows that accidents only occur when the holes in the various safety layers are aligned. This means that it is necessary to design a safety system with multiple safety layers with few holes (i.e., low failure probability) in each safety layer.

[0169] In an embodiment, the motion planning process of the AV 100 is designed so that the probability that the corresponding positions of each of the multiple layers are aligned is less than a threshold probability. For example, alternative safety metrics measure potential driving conflicts or behaviors that do not result in a collision but represent some degree of danger. Although a wide range of technologies exist, alternative safety metrics are specific to narrowly defined scenarios (e.g., assessing the safety of a subset of unsignalized intersections). Telematic services represent a commercial demonstration of the practical value of alternative safety metrics, tracking groups of drivers who brake or accelerate heavily frequently and assigning them a higher degree of collision risk.

[0170] In an embodiment, the stored behavioral model of the operation of the AV 100 includes a plurality of operating rules. Each operating rule has a priority relative to each other operating rule. The priority represents a risk level of one or more violations of the stored behavioral model. For example, alternative safety metrics are used to evaluate AV safety. Thus, alternative safety metrics are used to more quickly estimate road safety and integrate the concept into the overall theoretical framework. In an embodiment, a violation of the one or more violations of the stored behavioral model of the operation of the AV 100 includes a lateral distance between the AV 100 and one or more objects 416 falling below a threshold. For example, Fig.16The storage behavior model of is formalized into a framework for accidents (distribution of accident severity), which, if validated, means that the safety of complex systems can be inferred by observing alternative safety metrics. In an embodiment, the priority of the operating rules is adjusted based on the frequency of violations. For example, empirical evidence from human driver data is used to support the Fig.16 Storage behavior models applied to road safety.

[0171] Fig.17 4 illustrates a storage event distribution of an operation of the AV 100 on one or more objects 416 according to one or more embodiments. Figure 1 and 4 AV 100 and object 416 are illustrated and described in more detail. In an embodiment, the storage event distribution includes a log-normal probability distribution of independent random variables. Each random variable represents a risk level of danger of operation of AV 100. For example, Fig.17 The storage event distribution of implies that the collision events follow a log-normal distribution. Therefore, observation of low-severity events including behavioral metrics reveals the frequency of high-severity events. Therefore, Fig.17 The storage event distribution exemplified in enables AV design with redundant systems and resistance to single point failures. For example, the storage event distribution can be used to formalize the predictable relationship between the frequency and severity of safety-critical driving events based on the existing theoretical framework of accidents (accidents).

[0172] In an embodiment, mathematical analysis is used to determine a storage behavior model for accident causes and accident severity (see Fig.16 ) implies a specific distribution of accident severity. Fig.16 The storage behavior model is Fig.17 The severity of safety-critical driving events caused by hazards is formalized as a mathematical expression. Safety-critical driving incidents (events) are modeled as events that contain risk-increasing factors but may or may not lead to a crash. The central limit theorem is Fig.17 is used in to show that this mathematical expression implies a log-normal distribution of the severity of safety-critical events.

[0173] In an embodiment, the distribution patterns of four different data sets approximating the severity of motor vehicle events can be further analyzed. Mathematical Analysis (see Fig.17 ) shows that the storage behavior model is consistent with the log-normal distribution of accident severity. Fig.17 The empirical analysis shown in confirms that all five data sets fit the log-normal distribution closely. The sixth example data set shows a significantly increased relationship between near collisions and collisions. Fig.17The experiments illustrated in support the use of high frequency, low severity events to more quickly assess the safety of a motor vehicle or individual driver. In addition, complex systems (including autonomous vehicles) designed to be robust to single point failures are consistent with the same theoretical framework, allowing safer technologies to be deployed more quickly using the embodiments disclosed herein.

[0174] In an embodiment, several common "heavy-tailed" candidate distributions (power law, exponential, lognormal) are used to model the severity of the collision. The power law and exponential distributions have monotonically decreasing density functions (i.e., they do not have a left tail). Therefore, they only fit the right tail of the data. To ensure a fair comparison between the three candidate distributions, each data set discards the left tail relative to its peak, which is obtained by dividing the data set into 100 percentiles and taking the low end of the percentile containing the peak number of samples ( Fig.19 ). Since the lognormal distribution has a left tail, this treatment favors the other two candidate distributions. Not considering the left tail effectively ignores collisions of very low severity (i.e., low claim amount or Delta-V) (which are most affected by false negatives).

[0175] In an embodiment, the empirical data set is evaluated whether it follows a lognormal distribution, follows a different candidate distribution, or does not follow a candidate distribution. In the experiment, the Python powerlaw package is used, which uses maximum likelihood estimation to obtain the best fit for each candidate distribution. For each data set and candidate distribution, the Kolmogorov-Smirnov (KS) distance is determined, which is defined as the maximum difference between the cumulative empirical distribution function and the candidate fitting cumulative distribution function. The KS distance provides a measure of how well each individual candidate distribution fits the data. In order to more directly determine whether the lognormal distribution provides a better fit than the other two candidate distributions, the p value is considered for the significance of the log-likelihood ratio of the data from the lognormal distribution compared to each other candidate distribution. Small p values ​​provide evidence in favor of the lognormal distribution.

[0176] In the experiment, the trip records with the safety critical event data set were combined to determine the total number of near collisions that each driver had during the study, and the drivers were divided into groups based on how many near collisions the driver experienced during the study period. For a larger number of near collisions, the data set contained very few drivers. These drivers were combined into the same groups until the number of miles driven in the group exceeded one million miles, and the groups were assigned the average number of near collisions for all drivers added to the group. Spearman's rank correlation, which measures the strength of a monotonic (not necessarily linear) relationship, and the importance of the monotonic relationship for investigating whether groups of drivers with higher near collision rates tend to experience higher collision and severe collision rates, was determined.

[0177] Fig.18 4 illustrates the operation of AV 100 with respect to the storage event distribution of one or more objects 416 according to one or more embodiments. Figure 1 and 4 The AV 100 and the object 416 are illustrated and described in more detail. In an embodiment, a violation in the one or more violations indicates that the deceleration of the AV 100 exceeds a threshold deceleration. For example, the risk level is assessed based on non-crash results such as the frequency of near-crash and encountering dangerous driving situations. To evaluate whether a single framework can combine pre-crash and crash behaviors, two example data sets are analyzed.

[0178] The first example data set represents readings from a mobile device installed in a consumer vehicle for analyzing driver safety. The first data set is a randomly selected sample of hard braking events above a threshold deceleration. Hard braking is an evasive maneuver associated with an elevated risk of collision. To assess whether hard braking events lie on the same outcome continuum as collisions, a lognormal fit is evaluated using the same method used for the collision data.

[0179] In an embodiment, a violation in one or more violations indicates that the lateral distance from the AV 100 to one or more objects 416 falls below a threshold lateral distance (near-miss). For example, the second data set is derived from the second Strategic Highway Research Project (SHRP-2) naturalistic driving study, the largest study of real-world driving behavior to date. Two extracted data sets are used - the first has the trip records of the drivers participating in the study (3,546 drivers, 5.4 million trips captured, and 32 million fully recorded miles traveled), and the second has records of safety-critical events (8,717 collisions and near-collisions). The SHRP-2 study classifies safety-critical events into one of five categories (collision severity 1-4 and near-collisions). For the purpose of analysis, collisions with property damage, injury, or death are classified as "severe" (286 events). In addition to these being classified as level 4, any other contact events (i.e., tires running off the road or curb impacts not involving risk factors) are classified as "minor" (775 events). Level 4 collisions and non-collision incidents requiring evasive action were classified as “near collisions” (7,656 events).

[0180] Fig.19 An example storage event distribution illustrating operation of the AV 100 with respect to one or more objects 416 according to one or more embodiments. Figure 1 and 4 AV 100 and object 416 are illustrated and described in greater detail. Fig.16The stored behavior model of is conceptualized as a series of independent factors that interact with dangerous driving situations to amplify or mitigate dangers. For example, a hazard may occur due to the actions of other vehicles on the road. Whether a collision occurs depends on many other factors, such as the driver's reaction time, road geometry, weather, speed, vehicle capabilities, and maintenance status. If one or more factors are highly favorable, the hazard is eliminated and no safety-critical events occur. In reality, factors are not completely decisive and cannot completely eliminate hazards; factors may even aggravate hazards and thus become cumulatively added factors (e.g., bad weather), so factors may also act as hazards (e.g., a poorly maintained car may be a hazard, or a factor that aggravates another hazard). If other factors are only slightly effective in mitigating hazards, a near collision or a minor collision will occur, and if other factors are largely ineffective in mitigating hazards, a severe collision may occur.

[0181] In an embodiment, Fig.19 The storage event distribution of comprises a log-normal probability distribution of independent random variables. Each random variable represents a risk level of danger of the operation of the vehicle. For example, each collision has a danger at its main root cause. The distribution of collision severity is determined as in equation (1).

[0182] S = ∑ i w i S i (1)

[0183] Here, S i is a random variable representing the severity of the safety-critical event associated with the occurrence of hazard i, and w i is the proportion of safety-critical events due to hazard i. For a single type of hazard, the distribution of the results is determined as in equation (2).

[0184] S i = 〖H i × X〗 1i × X 2i × X 3i . . . X Ni (2)

[0185] Here, H i is a random variable representing the severity of hazard i, and each X ji is a random variable representing the effect of factor j on mitigating (or worsening) a particular hazard i. Taking the logarithm of both sides yields equation (3).

[0186] log S i = log 〖H i + log X〗1i + log X 2i + log X 3i . . . + log X Ni (3)

[0187] Since the right side of equation (3) is the sum of a series of independent random variables, if (1) all H i and X ji are equally distributed or (2)H i and X ji If the Lyapunov or Lindeberg central limit theorem conditions are satisfied, they will converge to a normal distribution. If so, then S i is well approximated by a log-normal probability distribution as in equation (4).

[0188] p(x) = 1 / x exp[-((ln x-μ) 2 ) / (2σ 2 )] (4)

[0189] where μ and σ are the quantities log S of the normal distribution, respectively. i Although completely different hazards and factors are more difficult to model by anything approaching identically distributed random variables, as long as no small subset dominates the random variables, the random variables are mostly independent, and there are a sufficiently large number of random variables, then S i will converge to a lognormal distribution. Therefore, if many largely unrelated factors affect the severity of a collision, such as Fig.16 As assumed by the model illustrated in , the severity distribution will tend to be log-normal.

[0190] Equation (2) enables the severity of all collisions to be captured by a weighted approximate summation of the encountered hazards i. The sum of multiple lognormal distributions remains lognormal and converges to a normal distribution only very slowly. Therefore, a car collision (or one following Fig.16 There will be outliers that represent single failure modes (e.g., the driver falling asleep, whose consequences are biased toward severe outcomes) and will not follow a lognormal distribution, but as long as the system is resilient to single point failures, the distribution of severity will be lognormal.

[0191] As discussed previously, Fig.16 The model of represents a qualitative distribution form for certain classes of safety-critical events characterized by a long right tail. The above analysis shows that the distribution naturally emerges from AV systems that trace the factor combinations for most accidents, and suggests that the true distribution form of the events is a log-normal distribution. Fig.19 The upper part of summarizes the results of the analysis of the distribution of crash severity, confirming that for all data sets: (i) the lognormal fit leads to a large minimum KS distance to the data in the candidate distributions, and (ii) the lognormal fit is highly significantly better than the fits to the other candidate distributions.

[0192] Fig. 20 An example storage event distribution illustrating operation of the AV 100 with respect to one or more objects 416 according to one or more embodiments. Figure 1 and 4 AV 100 and object 416 are illustrated and described in greater detail. Fig. 20 The fits of the two claim data sets are visually represented. Each plot shows the fitted probability distribution function of each candidate distribution relative to the empirical probability mass function on a logarithmic scale. In both, visual inspection shows that the lognormal distribution is likely the best fit, complementing the Fig.19 The numerical results of .

[0193] Fig.21 An example storage event distribution illustrates operations of the AV 100 with respect to one or more objects 416 according to one or more embodiments. Figure 1 and 4 AV 100 and subject 416 are illustrated and described in more detail. Analysis of the SHRP-2 naturalistic driving data shows a strong incremental relationship between a group's near crash rate and its crash rate (Spearman R=0.95, p<0.001). The incremental relationship is slightly weaker but still strong and significant when only the relationship between a group's near crash rate and severe crash rate is considered (Spearman R=0.75, p=0.01). These findings support the concept that the frequency with which a driver is involved in near crashes is a strong signal of driver ability and overall crash rate. Fig.21 The relationship between near crash rates and group crash and severe crash rates for driver groups is visualized with best fit lines and 95% confidence intervals. Due to individual differences and outliers, there is a weak correlation between near crash rate and crash rate only at the individual driver level (not the group level reported above), but the correlation is highly significant for both crashes (Spearman R=0.19, p<0.001) and severe crashes (Spearman R=0.12, p<0.001). Because SHRP-2 has a discrete outcome for "near crash", it is more difficult to estimate the relative severity of different near crashes and whether they follow a log-normal distribution. However, the evasive braking maneuver dataset from CMT provides a measure of the severity of both crashes and safety-critical events in Delta-V. Fig.19 The lower part of shows that, similar to collision events, hard braking events also follow a log-normal distribution.

[0194] Fig. 22 is a flowchart illustrating an example process for AV operation using a behavior rule model according to one or more embodiments. Figure 1 A more detailed description of the AV 100 is given in Fig. 22 In other embodiments, a particular entity (e.g., perception module 402 or planning module 404) performs some or all steps of the process. Furthermore, embodiments may include different and / or additional steps, or perform the steps in a different order. Figure 4 The perception module 402 and the planning module 404 are illustrated and described in further detail.

[0195] The AV 100 receives 2204 first sensor data from the first set of sensors 121 of the AV 100 and second sensor data from the second set of sensors 122 of the AV 100. Figure 1 The first set of sensors 121 and the second set of sensors 122 are described in more detail. The first sensor data represents the operation of the AV 100, and the second sensor data represents one or more objects 416 located in the environment 190. Figure 4 One or more objects 416 are described in more detail. Figure 1 Environment 190 is described in more detail.

[0196] The AV 100 determines 2208 one or more violations of a stored behavioral model of operation of the AV 100 based on the first sensor data and the second sensor data. Fig.14 An example storage behavior model is described in further detail. One or more violations are determined for one or more objects 416 located in environment 190 .

[0197] The AV 100 determines 2212 a first risk level of one or more violations based on the operation of the AV 100 relative to the storage event distribution of the one or more objects 416. Fig.17 An example storage event distribution is described in more detail.

[0198] In response to the first risk level being greater than the threshold risk level, the AV 100 generates 2216 a trajectory 198 for the AV 100. Figure 1 Trajectory 198 is described in more detail. Trajectory 198 has a second risk level that is lower than a threshold risk level. A second risk level is determined for one or more objects 416 .

[0199] The AV 100 operates 2220 based on the trajectory 198 to avoid collision of the AV 100 with one or more objects 416. For example, the control module 406 of the AV 100 is used to operate the AV 100. Figure 4 The control module 406 is described in further detail.

[0200] Fig.23 is a flow chart illustrating an example process for operating the AV 100 using a behavior rule model according to one or more embodiments. Figure 1 AV 100 is illustrated and described in more detail. In an embodiment, Fig.23 The processing is done by reference Figure 4 The planning circuit 404 described in more detail below performs the processing. In other embodiments, other entities (e.g., the sensing module 402 or the control module 406) perform some or all steps of the process. In addition, embodiments may include different and / or additional steps, or perform the steps in a different order. Figure 4 The perception module 402 and the control module 406 are illustrated and described in further detail.

[0201] The AV 100 generates 2304 the trajectory 198 based on the first sensor data from the first set of sensors 121 of the AV 100 and the second sensor data from the second set of sensors 122 of the AV 100. Figure 1 The first set of sensors 121 and the second set of sensors 122 are described in more detail. The first sensor data represents the operation of the AV 100, and the second sensor data represents one or more objects 416 located in the environment 190. Figure 4 One or more objects 416 are described in more detail. Figure 1 Environment 190 is described in more detail.

[0202] The AV 100 determines 2308 whether the trace 198 results in one or more violations of a stored behavior model of the operation of the AV 100. Fig.14 An example storage behavior model is described in further detail. One or more violations are determined relative to one or more objects 416 located in environment 190 .

[0203] In response to determining that the trace 190 causes one or more violations of the storage behavior model, the AV 100 determines 2312 a first risk level for the one or more violations based on the operation of the AV 100 relative to the storage event distribution of the one or more objects 416. Fig.17 The example storage event distribution described in more detail. The AV 100 generates candidate trajectories for the AV 100 .

[0204] The AV 100 determines 2316 that the candidate trajectory has a second risk level that is higher than the first risk level. The second risk level is determined for one or more objects 416 .

[0205] The AV 100 is operated 2320 based on the trajectory 198 to avoid collision of the AV 100 with one or more objects 416. For example, the control module 406 of the AV 100 is used to operate the AV 100. Figure 4 The control module 406 is described in further detail.

[0206] Using the embodiments disclosed herein, safety critical events are shown to result from the same causal mechanisms regardless of whether the event results in a loss (i.e., a collision in the case of a motor vehicle event), and the events are similar to those of the reference Fig.16 In line with the model described in more detail, the severity of the loss is multiplicative in nature, resulting in a log-normal severity distribution. In experiments, near collisions are strongly correlated with collisions (see Fig.21 ). At an aggregate level, experiments using the disclosed embodiments show that the frequency of hard braking events in a group also correlates with the collision rate in that group, because hard braking events are often caused by traffic conflicts that are resolved without collisions. Therefore, aggressive braking is an indicator of more severe collisions and higher collision frequency. Hard braking events follow the same distribution form as collision events using the same variable (Delta-V), which indicates a common causal mechanism. The embodiments disclosed herein can be used to determine that evasive braking actions follow the same form of frequency-severity distribution as the collisions themselves.

[0207] Embodiments disclosed herein can also be used to determine that five different data sets of events that are identified as crash and safety critical events all closely follow a log-normal distribution. Thus, implementations are disclosed for assessing crash risk based on the frequency with which a driver encounters a hazardous situation. While embodiments can speed up the evaluation of any road safety intervention, they are particularly useful in the case of AVs. Safety standards for AVs and other complex systems recommend the use of redundant subsystems with multiple safety measures to minimize single points of failure. These recommendations are consistent with Fig.16 The model is consistent with Fig.17 The corresponding analyses in show that these suggestions yield a predictable relationship between less severe and more severe events, with more severe events occurring with decreasing frequency.

[0208] The AV community is taking steps toward definitions of good behavior for AVs that measure behavioral capabilities beyond avoiding conflicts. Embodiments disclosed herein can be used to estimate how well AVs' ex ante (e.g., pre-collision) metrics predict risk in simulations or through aggregation of real-world data. Ex ante metrics can include hard braking, close following (low collision time), or others. Unlike system-specific software or other metrics, metrics that estimate safe road performance have the advantage of being technology-neutral (i.e., safety can be assessed independently of a specific technology implementation). The analysis described here shows that the product of random variables will converge to a log-normal distribution, even if there is some correlation between the random variables. In addition, formal modeling of accidents using the form of equation (3) helps understand how different factors combine to affect the severity of an event. In addition, considering other events besides near collisions in natural driving studies can eliminate some statistical fluctuations that limit SHRP-2 data analysis. Finally, a comprehensive analysis of safety-critical events involving AVs can be implemented.

[0209] In the previous description, embodiments of the present invention have been described with reference to many specific details, which may be different due to implementation. Therefore, the description and the accompanying drawings should be regarded as illustrative, rather than restrictive. The only and exclusive indication of the scope of the present invention, and the applicant's expectation that the content of the scope of the present invention is the literal and equivalent scope of the claims issued from this application in the specific form of issuing the claims, including any subsequent amendments. Any definition of the terms used to be included in such claims clearly set forth herein should be based on the meaning of such terms as used in the claims. In addition, when the term "also includes" is used in the previous description or the attached claims, the following of the phrase can be an additional step or entity, or a sub-step / sub-entity of the previously described step or entity.

[0210] CROSS-REFERENCE TO RELATED APPLICATIONS

[0211] This application is a conversion of U.S. Provisional Application No. 63 / 078,062 filed on September 14, 2020, and is incorporated herein by reference in its entirety.

Claims

1. A method of operating a vehicle, comprising: receiving, by one or more processors of a vehicle operating in an environment, first sensor data from a first set of sensors of the vehicle and second sensor data from a second set of sensors of the vehicle, the first sensor data being representative of operation of the vehicle and the second sensor data being representative of one or more objects located in the environment; determining, by the one or more processors, one or more violations of a stored behavioral model for operation of the vehicle based on the first sensor data and the second sensor data, the one or more violations being determined for the one or more objects located in the environment; determining, by the one or more processors, a first risk level for the one or more violations based on operation of the vehicle relative to a distribution of storage events for the one or more objects; generating, by the one or more processors, a trajectory for the vehicle, the trajectory having a second risk level lower than the threshold risk level, the second risk level determined for the one or more objects, in response to the first risk level being greater than a threshold risk level; as well as operating, by the one or more processors, the vehicle based on the trajectory to avoid a collision between the vehicle and the one or more objects, Wherein, the storage event distribution comprises a log-normal probability distribution of independent random variables, each random variable representing a risk level of danger of the operation of the vehicle.

2. The method according to claim 1, wherein: The first set of sensors includes at least one of an accelerometer, a steering wheel angle sensor, a wheel sensor, and a brake sensor.

3. The method according to claim 1, wherein: The first sensor data includes at least one of a velocity of the vehicle, an acceleration of the vehicle, a heading of the vehicle, an angular velocity of the vehicle, and a torque of the vehicle.

4. The method according to claim 1, wherein: The second set of sensors includes at least one of LiDAR, RADAR, camera and microphone.

5. The method according to claim 1, wherein: The second sensor data includes at least one of an image of the one or more objects, a velocity of the one or more objects, an acceleration of the one or more objects, and a lateral distance between the one or more objects and the vehicle.

6. The method of claim 1 further comprising determining, by the one or more processors, the second risk level based on a stored event distribution of the trajectory and operation of the vehicle relative to the one or more objects.

7. The method according to claim 1, wherein: A stored behavioral model of operation of the vehicle includes a plurality of operating rules, each operating rule of the plurality of operating rules having a priority relative to each other operating rule of the plurality of operating rules, the priority representing a risk level of one or more violations of the stored behavioral model.

8. The method according to claim 7, wherein: A violation of the one or more violations of the stored behavioral model of operation of the vehicle includes a lateral distance between the vehicle and the one or more objects falling below a threshold lateral distance.

9. The method of claim 8, further comprising adjusting a priority of the operational rule based on the frequency of the violations.

10. The method of claim 1, further comprising adjusting a motion planning process of the vehicle to reduce the second risk level based on a frequency of one or more violations of the stored behavior model.

11. The method of claim 10, further comprising determining a risk level for a motion planning process for the vehicle based on a frequency of one or more violations of the stored behavior model.

12. An autonomous vehicle, comprising: one or more computer processors; as well as one or more non-transitory storage media storing instructions that, when executed by the one or more computer processors, cause operations to be performed, the operations comprising: receiving, by one or more computer processors of the vehicle operating in an environment, first sensor data from a first set of sensors of the vehicle and second sensor data from a second set of sensors of the vehicle, the first sensor data being representative of operation of the vehicle and the second sensor data being representative of one or more objects located in the environment; determining, by one or more computer processors of the autonomous vehicle, one or more violations of a stored behavioral model for operation of the vehicle based on the first sensor data and the second sensor data, the one or more violations being determined for the one or more objects located in the environment; determining, by one or more computer processors of the autonomous vehicle, a first risk level for the one or more violations based on a distribution of stored events for the operation of the vehicle relative to the one or more objects; In response to the first risk level being greater than a threshold risk level, generating, by one or more computer processors of the autonomous vehicle, a trajectory for the vehicle, the trajectory having a second risk level lower than the threshold risk level, the second risk level determined for the one or more objects; and operating, by one or more computer processors of the autonomous vehicle, the vehicle based on the trajectory to avoid a collision between the vehicle and the one or more objects, Wherein, the storage event distribution comprises a log-normal probability distribution of independent random variables, each random variable representing a risk level of danger of the operation of the vehicle.

13. The autonomous vehicle of claim 12, wherein: The first set of sensors includes at least one of an accelerometer, a steering wheel angle sensor, a wheel sensor, and a brake sensor.

14. The autonomous vehicle of claim 12, wherein: The first sensor data includes at least one of a velocity of the vehicle, an acceleration of the vehicle, a heading of the vehicle, an angular velocity of the vehicle, and a torque of the vehicle.

15. The autonomous vehicle of claim 12, wherein: The second set of sensors includes at least one of LiDAR, RADAR, camera and microphone.

16. One or more non-transitory storage media storing instructions that, when executed by one or more computing devices, cause operations to be performed, the operations comprising: receiving, by one or more processors of a vehicle operating in an environment, first sensor data from a first set of sensors of the vehicle and second sensor data from a second set of sensors of the vehicle, the first sensor data being representative of operation of the vehicle and the second sensor data being representative of one or more objects located in the environment; determining, by the one or more processors, one or more violations of a stored behavioral model for operation of the vehicle based on the first sensor data and the second sensor data, the one or more violations being determined for the one or more objects located in the environment; determining, by the one or more processors, a first risk level for the one or more violations based on operation of the vehicle relative to a distribution of storage events for the one or more objects; generating, by the one or more processors, a trajectory for the vehicle, the trajectory having a second risk level lower than the threshold risk level, the second risk level determined for the one or more objects, in response to the first risk level being greater than a threshold risk level; as well as operating, by the one or more processors, the vehicle based on the trajectory to avoid a collision between the vehicle and the one or more objects, Wherein, the storage event distribution comprises a log-normal probability distribution of independent random variables, each random variable representing a risk level of danger of the operation of the vehicle.

17. The one or more non-transitory storage media of claim 16, wherein: The first set of sensors includes at least one of an accelerometer, a steering wheel angle sensor, a wheel sensor, and a brake sensor.

18. The one or more non-transitory storage media of claim 16, wherein: The first sensor data includes at least one of a velocity of the vehicle, an acceleration of the vehicle, a heading of the vehicle, an angular velocity of the vehicle, and a torque of the vehicle.

19. The one or more non-transitory storage media of claim 16, wherein: The second set of sensors includes at least one of LiDAR, RADAR, camera and microphone.

Images