Method for a vehicle, autonomous vehicle, and storage medium

Abstract

A method for a vehicle, an autonomous vehicle, and a storage medium. A method of vehicle operation using behavior rule checking includes receiving first sensor data from a first sensor of a vehicle and second sensor data from a second sensor. The first sensor data represents operation of the vehicle according to a first trajectory. The second sensor data represents at least one object. A first behavior rule is determined to be violated for the first trajectory based on the first sensor data and the second sensor data. The first behavior rule has a first priority. A plurality of alternative trajectories is generated using a control barrier function. A second trajectory is identified that violates a second behavior rule having a second priority that is less than the first priority. In response to identifying the second trajectory, a message is sent to a control circuit of the vehicle to operate the vehicle based on the second trajectory.

Description

TECHNICAL FIELD

[0001] This specification generally relates to operation of a vehicle, and specifically to vehicle operation using behavior rule checking. BACKGROUND

[0002] Operation of a vehicle from an initial location to a final destination often requires a user or a decision system of the vehicle to select a route through a road network from the initial location to the final destination. The route can involve satisfying objectives such as not exceeding a maximum driving time. In addition, the vehicle can be required to satisfy complex specifications imposed by traffic laws and cultural expectations of driving behavior. As a result, operation of an autonomous vehicle can require many decisions, making traditional algorithms for autonomous driving impractical. SUMMARY

[0003] Methods, systems, and devices for vehicle operation using behavior rule checking are disclosed. In an embodiment, at least one processor receives first sensor data from a first set of sensors of a vehicle and second sensor data from a second set of sensors of the vehicle. The first sensor data represents operation of the vehicle according to a first trajectory. The second sensor data represents at least one object. The at least one processor determines, based on the first sensor data and the second sensor data, that a first behavior rule of a hierarchical set of rules of operation of the vehicle is violated for the first trajectory. The first behavior rule has a first priority. The at least one processor generates, based on the first sensor data and the second sensor data, a plurality of alternative trajectories of the vehicle. The at least one processor identifies a second trajectory from the plurality of alternative trajectories. The second trajectory violates a second behavior rule of the hierarchical set of rules. The second behavior rule has a second priority that is less than the first priority. In response to identifying the second trajectory, the at least one processor sends a message to a control circuit of the vehicle to operate the vehicle based on the second trajectory.

[0004] In an embodiment, the framework is a general offline framework. In a general offline framework, pass / fail evaluation of trajectories is performed a posteriori. If it is found that a controller produces trajectories that cause fewer violations of the rule priority structure, a given trajectory is rejected.

[0005] In an embodiment, the framework is a general online framework. In a general online framework, the vehicle has a limited sensing range of the hierarchical set of rules that change operation of the vehicle. Control is generated using a receding horizon (model predictive control) approach.

[0006] In an embodiment, the at least one processor is located within a planning circuit of the vehicle. The at least one processor receives the first sensor data and the second sensor data during operation of the vehicle.

[0007] In an embodiment, the at least one processor adjusts operation of the planning circuit of the vehicle based on the second trajectory. The at least one processor is located on a computer device external to the vehicle. The at least one processor receives the first sensor data and the second sensor data after operation of the vehicle.

[0008] In an embodiment, the first set of sensors includes at least one of an accelerometer, a steering wheel angle sensor, a wheel sensor, and a brake sensor. The first sensor data includes at least one of a speed of the vehicle, an acceleration of the vehicle, a heading of the vehicle, an angular velocity of the vehicle, and a torque of the vehicle.

[0009] In an embodiment, the second set of sensors includes at least one of a LiDAR, a radar, a camera, a microphone, an infrared sensor, a sound navigation and ranging (SONAR) sensor, and the like.

[0010] In an embodiment, the second sensor data is at least one of an image of the at least one object, a speed of the at least one object, an acceleration of the at least one object, a lateral distance between the at least one object and the vehicle, and other kinematic data.

[0011] In an embodiment, the at least one processor selects the second trajectory from the plurality of alternative trajectories using at least one of a minimal violation planning, a model predictive control, and a machine learning, the selection being based on a plurality of hierarchical rules.

[0012] In an embodiment, each behavioral rule in the set of hierarchical rules has a respective priority relative to each other behavioral rule in the set of hierarchical rules. The respective priority indicates a level of risk of violating the each behavioral rule relative to the each other behavioral rule.

[0013] In an embodiment, violating the first behavioral rule includes operating the vehicle such that the lateral distance between the vehicle and the at least one object decreases to less than a threshold lateral distance.

[0014] In an embodiment, violating the first behavioral rule includes operating the vehicle such that the vehicle exceeds a speed limit.

[0015] In an embodiment, violating the first behavioral rule includes operating the vehicle such that the vehicle stops before reaching a destination.

[0016] In an embodiment, violating the first behavioral rule includes operating the vehicle such that the vehicle collides with the at least one object.

[0017] In an embodiment, the at least one processor determines a path of the at least one object based on the second sensor data. Determining that the first trajectory violates the first behavioral rule is further based on the path of the at least one object.

[0018] These and other aspects, features, and implementations can be expressed as methods, devices, systems, components, program products, means, or steps for performing functions and other manners.

[0019] A method for a vehicle includes: receiving, with at least one processor, first sensor data from a first set of sensors of the vehicle and second sensor data from a second set of sensors of the vehicle, the first sensor data representing operation of the vehicle according to a first trajectory, and the second sensor data representing at least one object; determining, with the at least one processor, that the first trajectory violates a first behavioral rule of a plurality of hierarchical rules of operation of the vehicle based on the first sensor data and the second sensor data, the first behavioral rule having a first priority; generating, with the at least one processor, a plurality of alternative trajectories of the vehicle based on the first sensor data and the second sensor data, the plurality of alternative trajectories generated using a control barrier function; identifying, with the at least one processor, a second trajectory from the plurality of alternative trajectories, wherein the second trajectory violates a second behavioral rule of the plurality of hierarchical rules, the second behavioral rule having a second priority less than the first priority; and in response to identifying the second trajectory, sending, with the at least one processor, a message to a control circuit of the vehicle to operate the vehicle based on the second trajectory.

[0020] An autonomous vehicle comprising: one or more processors; and one or more non-transitory storage media storing instructions that, when executed by the one or more processors, cause performance of the above method.

[0021] One or more non-transitory storage media storing instructions that, when executed by one or more computing devices, cause performance of the above method.

[0022] A method for a vehicle comprising performance of machine-executed operations involving instructions that, when executed by one or more computing devices, cause performance of the above method, wherein the machine-executed operations are at least one of sending the instructions, receiving the instructions, storing the instructions, and executing the instructions.

[0023] These and other aspects, features, and implementations will be more fully apparent from the following description (including claims) and appended drawings. BRIEF DESCRIPTION OF DRAWINGS

[0024] Figure 1 is a block diagram illustrating an example of an autonomous vehicle (AV) with autonomous capabilities, in accordance with one or more embodiments.

[0025] Figure 2 is a block diagram illustrating an example "cloud" computing environment, in accordance with one or more embodiments.

[0026] Figure 3 is a block diagram illustrating a computer system, in accordance with one or more embodiments.

[0027] Figure 4 is a block diagram illustrating an example architecture for an AV, in accordance with one or more embodiments.

[0028] Figure 5 is a block diagram illustrating an example of inputs and outputs that can be used by a perception module, in accordance with one or more embodiments.

[0029] Figure 6 is a block diagram illustrating an example of a LiDAR system, in accordance with one or more embodiments.

[0030] Figure 7 is a block diagram illustrating a LiDAR system in operation, in accordance with one or more embodiments.

[0031] Figure 8 is a block diagram illustrating additional details of the operation of a LiDAR system, in accordance with one or more embodiments.

[0032] Figure 9 is a block diagram illustrating relationships between inputs and outputs of a planning module, in accordance with one or more embodiments.

[0033] Figure 10 illustrates a directed graph used in path planning, in accordance with one or more embodiments.

[0034] Figure 11 is a block diagram illustrating inputs and outputs of a control module, in accordance with one or more embodiments.

[0035] Figure 12 is a block diagram illustrating inputs, outputs, and components of a controller, in accordance with one or more embodiments.

[0036] Figure 13A illustrates an example scenario of vehicle operation using behavior rule checking, in accordance with one or more embodiments.

[0037] Figure 13B illustrates an example hierarchical rule set, in accordance with one or more embodiments.

[0038] Figure 14An example flowchart of vehicle operation using behavior rule checking is shown, in accordance with one or more embodiments.

[0039] Figure 15 An example flowchart of vehicle operation using behavior rule checking is shown, in accordance with one or more embodiments.

[0040] Figure 16 An example of behavior rule checking for a vehicle is shown, in accordance with one or more embodiments.

[0041] Figure 17 An example flowchart of vehicle operation using behavior rule checking is shown, in accordance with one or more embodiments.

[0042] Figure 18 An example output of behavior rule checking for a vehicle is shown, in accordance with one or more embodiments.

[0043] Figure 19 An example flowchart of vehicle operation using behavior rule checking is shown, in accordance with one or more embodiments.

[0044] Figure 20 An example hierarchy rule set for vehicle operation using behavior rule checking is shown, in accordance with one or more embodiments. DETAILED DESCRIPTION

[0045] In the following description, for the purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the present application. It will be apparent, however, that the present application can be practiced without

[0046] In the drawings, specific arrangements or orders are shown for the sake of convenience in describing the illustrations, such as those representing devices, modules, instruction blocks, and data elements. However, it will be appreciated by those skilled in the art that the specific order or arrangement of the illustrations shown in the drawings is not intended to imply a particular processing order or sequence, or a separate processing process. Furthermore, inclusion of an illustration in the drawings does not mean that such an element is required in all embodiments, nor does it mean that a feature represented by such an element cannot be included in or combined with other elements in an embodiment.

[0047] Furthermore, in the drawings, connecting elements, such as lines or arrows or the like, are used to illustrate connections, relationships or associations between two or more other illustrative elements, and do not imply that there is any such connection, relationship or association in actual practice. In other words, some elements in the drawings can be shown to be connected, even when not so illustrated in the drawings, and such depiction of connection is not intended to be a given of a necessity of such connection. Rather, it is understood that when a requester requests a connection, relationship or association, the connection, relationship or association can be made in any manner, including without limitation, by wire, wireless, bus or network, including radio frequency, Infrared, Bluetooth or similar communication protocols, and the like, unless requested otherwise. Further, some elements will be shown in several drawings having the same numerical designation, to illustrate similar or related elements among several figures.

[0048] Reference will now be made in detail to embodiments, examples of which are illustrated in the accompanying drawings. In the following detailed description of embodiments, numerous specific details are set forth in order to provide a thorough understanding of the various described embodiments. However, it will be apparent to one skilled in the art that the various described embodiments can be practiced without these specific details. In other instances, well-known methods, procedures, components, circuits, and networks have not been described in detail so as not to unnecessarily obscure aspects of the embodiments.

[0049] Several of the described features can each be used alone or in combination with any of the other features. However, none of the individual features maybe sufficient to solve any of the problems discussed above, or to address any of the other concerns discussed herein. Some of the problems discussed above can not be solved by any of the features described herein. Although the following claims list certain combinations, other combinations are also possible and are contemplated. Although provided with headings, information relating to a particular heading can also be found elsewhere in the specification. Embodiments are described herein in accordance with the following outline:

[0050] 1. OVERALL SUMMARY

[0051] 2. SYSTEM SUMMARY

[0052] 3. AUTONOMOUS VEHICLE ARCHITECTURE

[0053] 4. AUTONOMOUS VEHICLE INPUT

[0054] 5. AUTONOMOUS VEHICLE PLANNING

[0055] 6. AUTONOMOUS VEHICLE CONTROL

[0056] 7. AUTONOMOUS VEHICLE OPERATION USING BEHAVIORAL PLANNING CHECKS

[0057] OVERALL SUMMARY

[0058] Methods, systems, and devices for vehicle operation using behavioral rule checking are presented herein. Road safety is an important public health issue (over 1 million people die in global road traffic every year by 2020) and is currently the seventh leading cause of years of life lost in the United States. The embodiments disclosed herein implement rule-based checking to evaluate the performance of a machine driver, to evaluate risk factors, and to evaluate the trajectory generation capabilities of an AV system or a subsystem such as a motion planning module. The implementation of the behavioral-based driving assessment disclosed herein is based on determining whether an alternative trajectory that is likely to cause fewer violations is available for an autonomous vehicle that violates a particular rule. These rules are derived from safety considerations, traffic laws, and accepted best practices. The driving rules are formulated to quantitatively evaluate how actual driving of an autonomous system matches expected driving behavior.

[0059] Advantages and benefits of the embodiments described herein include improved evaluation of driving performance of autonomous vehicle systems compared to traditional methods. Using these embodiments, specific autonomous driving behaviors can be evaluated more efficiently. The rule-based control method implemented using a control barrier function can be used as a trajectory checker for automated “post-hoc” optimal control evaluation and for execution on an autonomous vehicle for real-time evaluation. Since the implementation has reduced computational complexity, the disclosed embodiments can also be implemented in real-time on an autonomous vehicle as a rule-based planner or controller.

[0060] Further advantages and benefits of the embodiments disclosed herein include consideration of alternative trajectories so that unreasonable expectations are not imposed on autonomous vehicles. Since the rulebook is scenario and technology agnostic, the rulebook can be used for a variety of scenarios, different autonomous vehicle stack builds, different sensor configurations, and different planner algorithms. The disclosed embodiments enable autonomous vehicles to be implemented with greater scalability and avoid judgment calls by a test evaluator. In addition, the embodiments can inform various regulatory and standard processes that increasingly require specific AV behaviors and facilitate industry collaboration on defining good AV driving behavior.

[0061] System Overview

[0062] Figure 1 is a block diagram illustrating an example of an autonomous vehicle 100 with autonomous capabilities in accordance with one or more embodiments.

[0063] As used herein, the term “autonomous capabilities” refers to a function, feature, or facility that enables a vehicle to operate, in part or in whole, without real-time human intervention, including but not limited to fully autonomous vehicles, highly autonomous vehicles, and conditional autonomous vehicles.

[0064] As used herein, an autonomous vehicle (AV) is a vehicle with autonomous capabilities.

[0065] As used herein, a "vehicle" includes a means of transporting goods or people. For example, a car, bus, train, airplane, drone, truck, boat, ship, submersible, spacecraft, etc. A self-driving car is an example of a vehicle.

[0066] As used herein, a "trajectory" refers to a path or route that runs an AV from a first spatiotemporal location to a second spatiotemporal location. In embodiments, the first spatiotemporal location is referred to as an initial location or a starting location, and the second spatiotemporal location is referred to as a destination, a final location, a target, a target location, or a target location. In some examples, a trajectory is composed of one or more segments (e.g., pieces of a road), and each segment is composed of one or more blocks (e.g., a portion of a lane or an intersection). In embodiments, a spatiotemporal location corresponds to a real-world location. For example, a spatiotemporal location is a pickup or drop-off location for a person or a good to board or disembark.

[0067] As used herein, a "sensor(s)" includes one or more hardware components that detect information about the environment surrounding the sensor. Some hardware components can include sensing components (e.g., image sensors, biometric sensors), transmitting and / or receiving components (e.g., laser or radio frequency wave emitters and receivers), electronic components such as analog-to-digital converters, data storage devices such as RAM and / or non-volatile memory, software or firmware components, and data processing components such as application-specific integrated circuits, microprocessors, and / or microcontrollers.

[0068] As used herein, a "scene description" is a data structure (e.g., a list) or data stream that includes one or more classified or labeled objects detected by one or more sensors on an AV vehicle or provided by a source external to the AV.

[0069] As used herein, a "road" is a physical area that can be traversed by a vehicle, and can correspond to a named thoroughfare (e.g., a city street, an interstate highway, etc.) or can correspond to an unnamed thoroughfare (e.g., a driveway within a house or office building, a section of a parking lot, a section of an empty parking lot, a dirt path in a rural area, etc.). Because some vehicles (e.g., four-wheel drive pickup trucks, sport utility vehicles (SUVs), etc.) are capable of traversing a variety of physical areas that are not specifically designed for vehicle travel, a "road" can be any physical area that has not been formally defined as a thoroughfare by a municipality or other government or administrative body.

[0070] As used herein, a "lane" is a portion of a roadway that can be traversed by a vehicle, and can correspond to most or all of the space between lane markings, or only a portion (e.g., less than 50%) of the space between lane markings. For example, a roadway with lane markings that are far apart can accommodate two or more vehicles such that one vehicle can pass another without crossing a lane marking, and thus can be interpreted as a lane being narrower than the space between lane markings, or as two lanes between lanes. Lanes can also be interpreted in the absence of lane markings. For example, a lane can be defined based on physical features of the environment (e.g., rocks in a rural area and trees along a boulevard).

[0071] "one or more" includes a function performed by one element, a function performed by more than one element, e.g., in a distributed manner, a function performed by one element in a number of instances, a function performed by a number of elements in a number of instances, or any combination of the above.

[0072] It will also be understood that, although the terms "first," "second," etc. can be used herein to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another. For example, a first contact could be termed a second contact, and, similarly, a second contact could be termed a first contact, without departing from the scope of the various described embodiments. The first contact and the second contact are both contacts, but they are not the same contact.

[0073] The terminology used in the description of the various described embodiments herein is for the purpose of describing particular embodiments only and is not intended to be limiting. As used in the description of the various described embodiments and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that "and / or" as used herein, refers to and encompasses any and all possible combinations of one or more of the associated items. It will be further understood that the terms "comprises" "comprising," "includes" "including," "has" "having," and / or "has" when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0074] As used herein, the term "if' is, optionally, understood to mean "when" or "upon" or "in response to a determination that" or "in response to a detection that," depending on the context. Similarly, the phrase "if it is determined" or "if [a stated condition or event] is detected" is, optionally, understood to mean "upon determining" or "in response to a determination that" or "upon detecting [a stated condition or event]" or "in response to a detection [of a stated condition or event]," depending on the context.

[0075] As used herein, an AV system refers to an array of AVs and hardware, software, stored data, and real-time generated data that support operation of the AVs. In embodiments, the AV system is incorporated within the AVs. In embodiments, the AV system is distributed across several locations. For example, some software of the AV system is in a cloud computing environment similar to the cloud computing environment 200 described below with respect to FIG. 2. Figure 2 implemented in a cloud computing environment similar to the cloud computing environment 200 described below with respect to FIG. 2.

[0076] In general, this document describes techniques applicable to any vehicle with one or more levels of autonomy, including fully autonomous vehicles, highly autonomous vehicles, and conditionally autonomous vehicles, such as so-called Level 5, Level 4, and Level 3 vehicles, respectively (see SAE International Standard J3016: Taxonomy and Definitions for Terms Related to Driving Automation Systems for On-Road Motor Vehicles, incorporated by reference in its entirety for more detailed information on levels of vehicle autonomy). The techniques described in this document are also applicable to partially autonomous vehicles and driver-assist vehicles, such as so-called Level 2 and Level 1 vehicles (see SAE International Standard J3016: Taxonomy and Definitions for Terms Related to Driving Automation Systems for On-Road Motor Vehicles). In embodiments, one or more Level 1, Level 2, Level 3, Level 4, and Level 5 vehicle systems can automatically perform certain vehicle operations (e.g., steering, braking, and use of a map) under certain operating conditions based on processing of sensor inputs. The techniques described herein can benefit vehicles at any level of autonomy, ranging from fully autonomous vehicles to human-operated vehicles.

[0077] Referring to Figure 1 , the AV system 120 causes the AV 100 to operate along a trajectory 198 through the environment 190 to a destination 199 (sometimes referred to as a final location) while avoiding objects (e.g., natural obstacles 191, vehicles 193, pedestrians 192, cyclists, and other obstacles) and adhering to road rules (e.g., operating rules or driving preferences).

[0078] In embodiments, the AV system 120 includes the apparatus 101 for receiving and operating on operational commands from the computer processor 146. In embodiments, the computer processor 146 is with the AV system 120. Figure 3The described processor 304 is similar. Examples of the apparatus 101 include a steering controller 102, a brake 103, a gear, an accelerator pedal or other acceleration control mechanism, a windshield wiper, a side door lock, a window control, and a turn indicator.

[0079] In embodiments, the AV system 120 includes sensors 121 for measuring or inferring properties of the state or condition of the AV 100, such as the AV's position, linear and angular velocity and acceleration, and heading (e.g., the direction of the AV's 100 front end). Examples of the sensors 121 are GNSS, an inertial measurement unit (IMU) that measures both vehicle linear acceleration and angular rate, wheel sensors for measuring or estimating wheel slip, wheel brake pressure or brake torque sensors, engine torque or wheel torque sensors, and steering angle and angular rate sensors.

[0080] In embodiments, the sensors 121 also include sensors for sensing or measuring properties of the AV's environment. For example, monocular or stereo video cameras 122 in the visible, infrared, or thermal (or both) light spectrum, LiDAR 123, RADAR, ultrasonic sensors, time-of-flight (TOF) depth sensors, speed sensors, temperature sensors, humidity sensors, and precipitation sensors.

[0081] In embodiments, the AV system 120 includes a data storage unit 142 and a memory 144 for storing machine instructions associated with a computer processor 146 or data collected by the sensors 121. In embodiments, the data storage unit 142 is similar to the ROM 308 or storage 310 described below with respect to the AV 100. Figure 3 The described ROM 308 or storage 310 is similar. In embodiments, the memory 144 is similar to the main memory 306 described below. In embodiments, the data storage unit 142 and the memory 144 store historical, real-time, and / or predictive information about the environment 190. In embodiments, the stored information includes maps, driving performance, traffic congestion updates, or weather conditions. In embodiments, data about the environment 190 is transmitted from the remote database 134 to the AV 100 over a communication channel.

[0082] In embodiments, the AV system 120 includes communication devices 140 for transmitting properties of other vehicles' states and conditions, such as position, linear and angular velocity, linear and angular acceleration, and linear and angular heading, measured or inferred to the AV 100. These devices include vehicle-to-vehicle (V2V) and vehicle-to-infrastructure (V2I) communication devices and devices for wireless communication over point-to-point or ad hoc networks or both. In embodiments, the communication devices 140 communicate across the electromagnetic spectrum, including radio and optical communications, or other media (e.g., air and acoustic media). The combination of vehicle-to-vehicle (V2V), vehicle-to-infrastructure (V2I) communication (and, in embodiments, one or more other types of communication) is sometimes referred to as vehicle-to-everything (V2X) communication. V2X communication is generally in compliance with one or more communication standards for communication with and between autonomous vehicles.

[0083] In embodiments, the communication devices 140 include a communication interface. For example, a wired, wireless, WiMAX, Wi-Fi, Bluetooth, satellite, cellular, optical, near field, infrared, or radio interface. The communication interface transmits data from the remote database 134 to the AV system 120. In embodiments, the remote database 134 is embedded in a cloud computing environment 200 as described in Figure 2 In embodiments, the communication interface 140 transmits data collected from the sensors 121 or other data related to AV 100 operation to the remote database 134. In embodiments, the communication interface 140 transmits information related to teleoperation to the AV 100. In embodiments, the AV 100 communicates with other remote (e.g., "cloud") servers 136.

[0084] In embodiments, the remote database 134 also stores and transmits digital data (e.g., storing data such as road and street locations). This data is stored in the memory 144 on the AV 100 or transmitted from the remote database 134 to the AV 100 over a communication channel.

[0085] In embodiments, the remote database 134 stores and transmits historical information related to the driving properties (e.g., velocity and acceleration profiles) of vehicles that have previously traveled along the trajectory 198 at a similar time of day. In one implementation, this data can be stored in the memory 144 on the AV 100 or transmitted from the remote database 134 to the AV 100 over a communication channel.

[0086] The computing devices 146 located on the AV 100 generate control actions algorithmically based on both real-time sensor data and a priori information, allowing the AV system 120 to execute its autonomous driving capabilities.

[0087] In embodiments, the AV system 120 includes a computer peripheral 132 coupled to the computing device 146 for providing information and reminders to a user (e.g., an occupant or a remote user) of the AV 100 and receiving input from the user. In embodiments, the peripheral 132 is similar to the display 312, input device 314, and cursor controller 316 discussed below with reference to Figure 3 The coupling is wireless or wired. Any two or more of the interface devices can be integrated into a single device.

[0088] Example Cloud Computing Environment

[0089] Figure 2 is a block diagram illustrating an example “cloud” computing environment in accordance with one or more embodiments. Cloud computing is a model of service delivery for enabling convenient, on-demand network access to a shared pool of configurable computing resources (e.g. networks, network bandwidth, servers, processing, memory, storage, applications, virtual machines, and services) typically provided by a cloud computing system. In typical cloud computing systems, one or more large cloud data centers house the machines used to deliver the services provided by the cloud. Referring now to Figure 2 , the cloud computing environment 200 includes cloud data centers 204a, 204b, and 204c interconnected by a cloud 202. The data centers 204a, 204b, and 204c provide cloud computing services for computer systems 206a, 206b, 206c, 206d, 206e, and 206f connected to the cloud 202.

[0090] The cloud computing environment 200 includes one or more cloud data centers. In general, a cloud data center (e.g., the cloud data center 204a shown in Figure 2 ) refers to a physical arrangement of servers that make up a cloud (e.g., the cloud 202 shown in Figure 2 or a particular portion of a cloud). For example, the servers are physically arranged in rooms, groups, rows, and racks in a cloud data center. A cloud data center has one or more areas that include one or more server rooms. Each room has one or more rows of servers, and each row includes one or more racks. Each rack includes one or more individual server nodes. In some implementations, the servers in an area, room, rack, and / or row are arranged into groups based on physical infrastructure requirements of the data center facility, including power, energy, heat, heat sources, and / or other requirements. In embodiments, the server nodes are similar to the computer systems described in Figure 3 . The data center 204a has many computing systems distributed across multiple racks.

[0091] The cloud 202 includes cloud data centers 204a, 204b, and 204c and networks and network resources (e.g., network devices, nodes, routers, switches, and network cables) used to connect and facilitate access by the computing systems 206a-f to cloud computing services. In embodiments, the network represents any combination of one or more local networks, wide-area networks, or internetworks coupled by wired or wireless links deployed using terrestrial or satellite connections. Data exchanged over the network is transmitted using a variety of network layer protocols, such as Internet Protocol (IP), Multiprotocol Label Switching (MPLS), Asynchronous Transfer Mode (ATM), Frame Relay, and the like. Moreover, in embodiments where the network represents a combination of multiple sub-networks, different network layer protocols are used on each underlying sub-network. In embodiments, the network represents one or more internetworks, such as the public Internet, and the like.

[0092] The computing systems 206a-f or cloud computing service consumers are connected to the cloud 202 through network links and network adapters. In embodiments, the computing systems 206a-f are implemented as various computing devices, such as servers, desktops, laptops, tablets, smartphones, Internet of Things (IoT) devices, autonomous vehicles (including cars, drones, spacecraft, trains, buses, and the like), and consumer electronics. In embodiments, the computing systems 206a-f are implemented in or as part of other systems.

[0093] Computer system

[0094] Figure 3 is a block diagram illustrating a computer system 300 in accordance with one or more embodiments. In implementations, the computer system 300 is a special-purpose computing device. The special-purpose computing device is either hard-wired to perform the techniques, or includes digital electronic devices such as one or more application-specific integrated circuits (ASICs) or field programmable gate arrays (FPGAs) that are persistently programmed to perform the techniques, or one or more general purpose hardware processors programmed to perform the techniques pursuant to program instructions in firmware, memory, other storage, or a combination. Such special-purpose computing devices can also combine custom hard-wired logic, ASICs, or FPGAs with custom programming to accomplish the techniques. In various embodiments, the special-purpose computing device is a desktop computer system, a portable computer system, a handheld device, a network device, or any other device that incorporates hard-wired and / or program logic to implement the techniques.

[0095] In embodiments, the computer system 300 includes a bus 302 or other communication mechanism for communicating information, and a hardware processor 304 coupled with bus 302 for processing information. The hardware processor 304 is, for example, a general-purpose microprocessor. The computer system 300 also includes a main memory 306, such as a random access memory (RAM) or other dynamic storage device, coupled to bus 302 for storing information and instructions to be executed by processor 304. In one implementation, the main memory 306 is used for storing temporary variables or other intermediate information during execution of instructions to be executed by processor 304. Such instructions can be stored or implemented in non-transitory storage media accessible by processor 304, such as storage media 310, when such instructions are available to the processor 304. The computer system 300 is thus made an entity (being the special purpose machine) that operates to perform the operations specified in the instructions.

[0096] In embodiments, the computer system 300 also includes a read only memory (ROM) 308 or other static storage device coupled to bus 302 for storing static information and instructions for processor 304. A storage device 310, such as a magnetic disk, optical disk, solid-state drive, or three-dimensional cross-point memory, is provided and coupled to bus 302 for storing information and instructions.

[0097] In embodiments, the computer system 300 is coupled via bus 302 to a display 312, such as a cathode ray tube (CRT), liquid crystal display (LCD), plasma display, light emitting diode (LED) display, or organic light emitting diode (OLED) display, for displaying information to a computer user. An input device 314, including alphanumeric and other keys, is coupled to bus 302 for communicating information and command selections to processor 304. Another type of user input device is cursor control 316, such as a mouse, a trackball, a touch display, or cursor direction keys for communicating direction information and command selections to processor 304 and for

[0098] According to one embodiment, the techniques herein are performed by computer system 300 in response to processor 304 executing one or more sequences of one or more instructions contained in main memory 306. Such instructions can be read into main memory 306 from another storage medium, such as storage device 310. Execution of the sequences of instructions contained in main memory 306 causes processor 304 to perform the process steps described herein. In alternative embodiments, hard-wired circuitry can be used in place of or in combination with software instructions.

[0099] The term "storage media" as used herein refers to any non-transitory media that store data and / or instructions that cause a machine to operate in a specific fashion. Such storage media include non-volatile media and / or volatile media. Non-volatile media include, for example, optical or magnetic disks, such as storage device 310. Volatile media include dynamic memory, such as main memory 306. Common forms of storage media include, for example, a floppy disk, a flexible disk, hard disk, solid-state drive, magnetic tape, or any other magnetic data storage medium, a CD-ROM, any other optical data storage medium, any physical medium with patterns of holes, a RAM, a PROM, and EPROM, a FLASH-EPROM, NVRAM, or any other memory chip or cartridge.

[0100] Storage media are distinct from, but can be used in combination with, transmission media. Transmission media participate in transferring information between storage media. For example, transmission media include coaxial cables, copper wire, and optical fibers, including the wires that comprise bus 302. Transmission media can also take the form of acoustic or light waves, such as those generated during radio frequency and infrared data communications.

[0101] In embodiments, various forms of media are involved in carrying one or more sequences of one or more instructions to processor 304 for execution. For example, the instructions can initially be carried on a magnetic disk or solid-state drive of a remote computer. The remote computer loads the instructions into its dynamic memory and sends the instructions over a telephone line using a modem. A local modem in computer system 300 receives the data on the telephone line and uses an infrared transmitter to convert the data to an infrared signal. An infrared detector in computer system 300 receives the data carried in the infrared signal and places the data on bus 302. Bus 302 carries the data to main memory 306, from which processor 304 retrieves and executes the instructions. The instructions received by main memory 306 can optionally be stored on storage device 310 either before or after execution by processor 304.

[0102] Computer system 300 also includes a communication interface 318 coupled to bus 302. Communication interface 318 provides a two-way data communication coupling to a network link 320 that is connected to a local network 322. For example, communication interface 318 is a integrated services digital network (ISDN) card, cable modem, satellite modem, or a modem to provide a data communication connection to a corresponding type of telephone line. As another example, communication interface 318 is a local area network (LAN) card to provide a data communication connection to a compatible LAN. Wireless links are also implemented in some implementations. In any such implementation, communication interface 318 sends and receives electrical, electromagnetic or optical signals that carry digital data streams representing various types of information.

[0103] Network link 320 typically provides data communication to other data devices via one or more networks. For example, network link 320 provides connectivity to host computer 324 or to a cloud data center or device operated by Internet Service Provider (ISP) 326 via local network 322. ISP 326, in turn, provides data communication services via a worldwide packet data communication network now commonly referred to as the "Internet" 328. Both local network 322 and Internet 328 use electrical, electromagnetic, or optical signals that carry digital data streams. Signals through various networks and signals on network link 320 via communication interface 318 are example forms of transmission media carrying digital data entering and leaving computer system 300. In embodiments, network 320 includes the aforementioned cloud 202 or a portion of cloud 202.

[0104] Computer system 300 sends messages and receives data including program code through one or more networks, network links 320, and communication interfaces 318. In an embodiment, computer system 300 receives code for processing. The received code is executed by processor 304 upon receipt and / or stored in storage device 310, or in other non-volatile storage devices for later execution.

[0105] Autonomous Vehicle Architecture

[0106] Figure 4 This illustrates an autonomous vehicle (e.g., according to one or more embodiments) Figure 1 The diagram shows a block diagram of an example architecture 400 for the AV 100. Architecture 400 includes a sensing module 402 (sometimes called a sensing circuit), a planning module 404 (sometimes called a planning circuit), a control module 406 (sometimes called a control circuit), a positioning module 408 (sometimes called a positioning circuit), and a database module 410 (sometimes called a database circuit). Each module plays a role in the operation of the AV 100. Commonly, modules 402, 404, 406, 408, and 410 can be... Figure 1 This is part of the AV system 120 shown. In this embodiment, any of the modules 402, 404, 406, 408, and 410 is a combination of computer software (e.g., executable code stored on a computer-readable medium) and computer hardware (e.g., one or more microprocessors, microcontrollers, application-specific integrated circuits (ASICs), hardware memory devices, other types of integrated circuits, other types of computer hardware, or any or all combinations of these hardware).

[0107] In use, the planning module 404 receives data representing the destination 412 and determines data representing a trajectory 414 (sometimes referred to as a route) that the AV 100 can travel in order to reach (e.g., arrive at) the destination 412. In order for the planning module 404 to determine the data representing the trajectory 414, the planning module 404 receives data from the perception module 402, the localization module 408, and the database module 410.

[0108] The perception module 402 identifies nearby physical objects using one or more sensors 121, for example, as shown in Figure 1 The objects are classified (e.g., grouped into types such as pedestrians, bicycles, cars, traffic signs, etc.) and a scene description including the classified objects 416 is provided to the planning module 404.

[0109] The planning module 404 also receives data representing the AV position 418 from the localization module 408. The localization module 408 determines the AV position by using data from the sensors 121 and data from the database module 410 (e.g., geographic data) to calculate a position. For example, the localization module 408 uses data from a global navigation satellite system (GNSS) unit and geographic data to calculate the longitude and latitude of the AV. In embodiments, the data used by the localization module 408 includes high-precision maps with lane geometry properties, maps describing road network connectivity properties, maps describing lane physical properties such as traffic speed, traffic volume, number of vehicle and bicycle lanes, lane width, lane traffic direction, or lane marking type and location, or combinations thereof, and maps describing spatial locations of road features such as intersections, traffic signs, or other types of travel signals.

[0110] The control module 406 receives the data representing the trajectory 414 and the data representing the AV position 418 and operates the control functions 420a-c (e.g., steering, throttle, brakes, ignition) of the AV in a manner that will cause the AV 100 to travel the trajectory 414 to the destination 412. For example, if the trajectory 414 includes a left turn, the control module 406 will operate the control functions 420a-c in a manner that the steering angle of the steering function will cause the AV 100 to turn left and the throttle and brakes will cause the AV 100 to pause and wait for passing pedestrians or vehicles before making the turn.

[0111] Autonomous vehicle inputs

[0112] Figure 5 FIG. 5 is a diagram illustrating inputs 502a-d (e.g., sensor data) used by the perception module 402 (e.g., perception system) according to one or more embodiments. Figure 4 Figure 1 ​The input 502a is a LiDAR (light detection and ranging) system (e.g., a LiDAR 123 as shown in FIG. 1) that uses light (e.g., a beam of light such as infrared light) to obtain data about physical objects in its line of sight. The LiDAR system produces LiDAR data as output 504a. For example, the LiDAR data is a collection of 3D or 2D points (also referred to as a point cloud) used to construct a representation of the environment 190. Figure 1 The LiDAR is a technology that uses light (e.g., a beam of light such as infrared light) to obtain data about physical objects in its line of sight. The LiDAR system produces LiDAR data as output 504a. For example, the LiDAR data is a collection of 3D or 2D points (also referred to as a point cloud) used to construct a representation of the environment 190.

[0113] The input 502b is a RADAR (RAdio Detection And Ranging) system. RADAR is a technology that uses radio waves to obtain data about physical objects in the vicinity. RADAR can obtain data about objects that are not in the line of sight of the LiDAR system. The RADAR system 502b produces RADAR data as output 504b. For example, the RADAR data is one or more radio frequency electromagnetic signals used to construct a representation of the environment 190.

[0114] The input 502c is a camera system. The camera system uses one or more cameras (e.g., digital cameras that use a photosensor such as a charge-coupled device [CCD]) to acquire information about physical objects in the vicinity. The camera system produces camera data as output 504c. The camera data is typically in the form of image data (e.g., data in an image data format such as RAW, JPEG, PNG, etc.). In some examples, the camera system has multiple independent cameras, for example, for the purpose of stereoscopic imagery (stereo vision), which enables the camera system to perceive depth. Although the objects perceived by the camera system are described here as being in the “vicinity,” this is relative to the AV. In use, the camera system can be configured to “see” objects that are far away (e.g., up to 1 kilometer or more in front of the AV). Thus, the camera system can have features such as sensors and lenses that are optimized for perceiving objects that are far away.

[0115] Another input 502d is a traffic light detection (TLD) system. The TLD system uses one or more cameras to obtain information about traffic lights, street signs, and other physical objects that provide visual operational information. The TLD system produces TLD data as output 504d. The TLD data often takes the form of image data (e.g., data in an image data format such as RAW, JPEG, PNG, etc.). The TLD system differs from systems that include cameras in that the TLD system uses a camera with a wide field of view (e.g., using a wide-angle lens or a fisheye lens) to obtain information about as many physical objects that provide visual operational information as possible, so that the AV 100 has access to all relevant operational information provided by those objects. For example, the TLD system can have a field of view of about 120 degrees or more.

[0116] In embodiments, the outputs 504a-504d are combined using sensor fusion techniques. Thus, the individual outputs 504a-504d are provided to other systems of the AV 100 (e.g., to the planning module 404 as shown), or a single combined output or multiple combined outputs of the same type (e.g., using the same combination technique or combining the same outputs or both) or a single combined output or multiple combined outputs of different types (e.g., using different respective combination techniques or combining different respective outputs or both) can be provided to the other systems. Figure 4 In embodiments, early fusion techniques are used. Early fusion techniques are characterized by combining the outputs before applying one or more data processing steps to the combined output. In embodiments, late fusion techniques are used. Late fusion techniques are characterized by combining the outputs after applying one or more data processing steps to the individual outputs.

[0117] Figure 6 is an example showing a LiDAR system 602 in accordance with one or more embodiments (e.g., the LiDAR system 202 of FIG. 2). Figure 5The block diagram shown is for input 502a). The LiDAR system 602 emits light 604a-604c from a emitter 606 (e.g., a laser emitter). The light emitted by the LiDAR system is typically not in the visible spectrum; for example, infrared light is often used. Some of the emitted light 604b encounters a physical object 608 (e.g., a vehicle) and is reflected back to the LiDAR system 602. (The light emitted from the LiDAR system typically does not penetrate the physical object, e.g., a solid physical object.) The LiDAR system 602 also has one or more photodetectors 610 for detecting the reflected light. In an embodiment, one or more data processing systems associated with the LiDAR system generate an image 612 representing the field of view 614 of the LiDAR system. Image 612 includes information representing the boundary 616 of the physical object 608. Thus, image 612 is used to determine the boundary 616 of one or more physical objects near the AV.

[0118] Figure 7 This is a block diagram illustrating the operation of a LiDAR system 602 according to one or more embodiments. In the scenario shown in the figure, AV 100 receives both a camera system output 504c in the form of an image 702 and a LiDAR system output 504a in the form of LiDAR data points 704. In use, the data processing system of AV 100 compares the image 702 with the data points 704. Specifically, physical objects 706 identified in the image 702 are also identified in the data points 704. Thus, AV 100 perceives the boundaries of physical objects based on the contours and density of the data points 704.

[0119] Figure 8 This is a block diagram illustrating additional details of the operation of a LiDAR system 602 according to one or more embodiments. As described above, the AV 100 detects the boundaries of physical objects based on the characteristics of data points detected by the LiDAR system 602. Figure 8 As shown, a flat object, such as ground 802, will reflect light 804a-804d emitted from LiDAR system 602 in a consistent manner. In other words, because LiDAR system 602 emits light at a consistent interval, ground 802 will reflect light back to LiDAR system 602 at the same consistent interval. As AV 100 travels on ground 802, if nothing obstructs the path, LiDAR system 602 will continue to detect light reflected by the next effective surface point 806. However, if object 808 obstructs the path, light 804e-804f emitted by LiDAR system 602 will be reflected from points 810a-810b in a manner inconsistent with the expected consistency. Based on this information, AV 100 can determine the presence of object 808.

[0120] Path planning

[0121] Figure 9 This illustrates one or more embodiments (e.g., such as...) Figure 4 The diagram 900 illustrates the relationship between the inputs and outputs of the planning module 404. Generally, the output of the planning module 404 is a route 902 from a starting point 904 (e.g., a source location or initial location) to an ending point 906 (e.g., a destination or final location). Route 902 is typically defined by one or more road segments. For example, a road segment refers to the distance to be traveled over at least a portion of a street, road, highway, driveway, or other physical area suitable for vehicle travel. In some examples, such as if AV 100 is an off-road capable vehicle such as a four-wheel drive (4WD) or all-wheel drive (AWD) car, SUV, or pickup truck, route 902 includes “off-road” segments such as unpaved paths or open fields.

[0122] In addition to route 902, the planning module also outputs lane-level route planning data 908. Lane-level route planning data 908 is used to navigate segments of route 902 based on conditions at a specific time. For example, if route 902 comprises a multi-lane highway, lane-level route planning data 908 includes trajectory planning data 910, which AV 100 can use to select a lane from the multiple lanes based on factors such as whether an exit is nearby, whether other vehicles are present in one or more lanes, or other factors that change over a period of minutes or less. Similarly, in some implementations, lane-level route planning data 908 includes a speed constraint 912 specific to a segment of route 902. For example, if the segment includes pedestrians or unexpected traffic, speed constraint 912 can limit AV 100 to a slower speed than expected, such as a speed limit based on the segment's speed limit data.

[0123] In this embodiment, the input to the planning module 404 includes (e.g., from...) Figure 4 The database module 410 shown contains database data 914 and current location data 916 (for example, Figure 4 The AV position shown is 418), (for example, for use with Figure 4 The destination data 918 and object data 920 shown for destination 412 (e.g., as shown) Figure 4The database data 914 includes classified objects 416 perceived by the perception module 402 as shown. In embodiments, the database data 914 includes rules used in planning. The rules are specified using a formal language (e.g., using Boolean logic). At least some of the rules will apply to any given situation encountered by the AV 100. A rule applies to a given situation if the rule has conditions that are satisfied based on information available to the AV 100 (e.g., information about the surrounding environment). Rules can have priorities. For example, the rule "if the highway is a freeway, then move to the leftmost lane" can have a lower priority than "if an exit is nearby within one mile, then move to the rightmost lane."

[0124] Figure 10 A directed graph 1000 used in path planning (e.g., by the planning module 404 (402) is shown in accordance with one or more embodiments. In general, the directed graph 1000 like the one shown is used to determine a path between any start point 1002 and end point 1004. In the real world, the distance separating the start point 1002 and the end point 1004 can be relatively large (e.g., in two different metropolitan areas) or can be relatively small (e.g., two intersections abutting city blocks or two lanes of a multi-lane road). Figure 4 Figure 10 The directed graph 1000 like the one shown is used to determine a path between any start point 1002 and end point 1004. In the real world, the distance separating the start point 1002 and the end point 1004 can be relatively large (e.g., in two different metropolitan areas) or can be relatively small (e.g., two intersections abutting city blocks or two lanes of a multi-lane road).

[0125] In embodiments, the directed graph 1000 has nodes 1006a-1006d representing different locations between the start point 1002 and the end point 1004 that the AV 100 can occupy. In some examples, e.g., when the start point 1002 and the end point 1004 represent different metropolitan areas, the nodes 1006a-1006d represent road segments. In some examples, e.g., when the start point 1002 and the end point 1004 represent different locations on the same road, the nodes 1006a-1006d represent different positions on the road. In this way, the directed graph 1000 includes information at different levels of granularity. In embodiments, a directed graph with high granularity is also a subgraph of another directed graph with greater scale. For example, a directed graph with a start point 1002 and an end point 1004 that are far apart (e.g., many miles apart) has most of its information at low granularity and is based on stored data, but the directed graph also includes some high granularity information for a portion of the directed graph representing physical locations in the field of view of the AV 100.

[0126] ​Nodes 1006a-1006d are different from objects 1008a-1008b, which cannot overlap with nodes. In embodiments, at low granularity, objects 1008a-1008b represent areas that a car cannot pass through, e.g., areas without streets or roads. At high granularity, objects 1008a-1008b represent physical objects in the field of view of AV 100, e.g., other cars, pedestrians, or other entities with which AV 100 cannot share physical space. In embodiments, some or all of objects 1008a-1008b are static objects (e.g., objects that do not change position, such as streetlights or utility poles) or dynamic objects (e.g., objects that can change position, such as pedestrians or other cars).

[0127] Nodes 1006a-1006d are connected by edges 1010a-1010c. If two nodes 1006a-1006b are connected by edge 1010a, AV 100 can travel between one node 1006a and the other node 1006b, e.g., without having to travel to an intermediate node before reaching the other node 1006b. (When referring to AV 100 traveling between nodes, it means that AV 100 travels between two physical locations represented by the respective nodes.) Edges 1010a-1010c are typically bidirectional, in the sense that AV 100 can travel from a first node to a second node, or from the second node to the first node. In embodiments, edges 1010a-1010c are unidirectional, in the sense that AV 100 can travel from a first node to a second node, but AV 100 cannot travel from the second node to the first node. Edges 1010a-1010c are unidirectional in cases where they represent, e.g., one-way streets, individual lanes of a street, road, or highway, or other features that can only be traversed in one direction due to legal or physical constraints.

[0128] In embodiments, planning module 404 uses directed graph 1000 to identify a path 1012 consisting of nodes and edges between start node 1002 and end node 1004.

[0129] Edge 1010a-1010c has associated costs 1014a-1014b. Costs 1014a-1014b represent the value of the resources that would be spent if AV 100 selected that edge. A typical resource is time. For example, if the physical distance represented by one edge 1010a is twice the physical distance represented by another edge 1010b, then the associated cost 1014a of the first edge 1010a can be twice the associated cost 1014b of the second edge 1010b. Other factors affecting time include anticipated traffic, the number of intersections, speed limits, etc. Another typical resource is fuel economy. The two edges 1010a-1010b can represent the same physical distance, but due to factors such as road conditions and anticipated weather, one edge 1010a may require more fuel than the other edge 1010b.

[0130] When the planning module 404 identifies the path 1012 between the starting point 1002 and the ending point 1004, the planning module 404 typically selects the path that is optimized for cost, such as the path that has the minimum total cost when the individual costs of the edges are added together.

[0131] Autonomous Vehicle Control

[0132] Figure 11 This illustrates one or more embodiments (e.g., such as...) Figure 4 The block diagram 1100 shows the inputs and outputs of the control module 406. The control module operates according to a controller 1102, which includes, for example, one or more processors similar to processor 304 (e.g., one or more computer processors such as a microprocessor or microcontroller or both); short-term and / or long-term data storage devices similar to main memory 306, ROM 308 and storage device 310 (e.g., memory, random access memory or flash memory or both); and instructions stored in the memory that, when executed (e.g. by one or more processors), perform the operation of controller 1102.

[0133] In one embodiment, controller 1102 receives data representing a desired output 1104. The desired output 1104 typically includes speed, such as rate and heading. The desired output 1104 may be based, for example, from (e.g., as...) Figure 4The planning module 404 receives data (e.g., as shown) from the sensors 402. The controller 1102 produces data that can be used as a throttle input 1106 and a steering input 1108 based on the desired output 1104. The throttle input 1106 represents the magnitude at which the throttle (e.g., acceleration control) of the AV 100 should be engaged, e.g., by engaging a steering pedal or engaging another throttle control, to achieve the desired output 1104. In some examples, the throttle input 1106 also includes data that can be used to engage the brakes (e.g., deceleration control) of the AV 100. The steering input 1108 represents the angle at which the steering control (e.g., steering wheel, steering angle actuator, or other functionality used to control the steering angle) of the AV should be positioned to achieve the desired output 1104.

[0134] In embodiments, the controller 1102 receives feedback that is used in adjusting the inputs provided to the throttle and steering. For example, if the AV 100 encounters an interference 1110, such as a hill, the measured velocity 1112 of the AV 100 drops below the desired output velocity. In embodiments, any measured outputs 1114 are provided to the controller 1102 so that the necessary adjustments are made, e.g., based on the difference 1113 between the measured velocity and the desired output. The measured outputs 1114 include measured position 1116, measured velocity 1118 (including speed and heading), measured acceleration 1120, and other outputs that are measurable by sensors of the AV 100.

[0135] In embodiments, information about the interference 1110 is detected in advance, e.g., by sensors such as cameras or LiDAR sensors, and the information is provided to a predictive feedback module 1122. The predictive feedback module 1122 then provides information to the controller 1102 that the controller 1102 can use to adjust accordingly. For example, if a sensor of the AV 100 detects (“sees”) a hill, the controller 1102 can use that information to prepare to engage the throttle at the appropriate time to avoid a significant deceleration.

[0136] Figure 12 is a block diagram 1200 showing inputs, outputs, and components of the controller 1102 according to one or more embodiments. The controller 1102 has a velocity analyzer 1202 that influences the operation of a throttle / brake controller 1204. For example, the velocity analyzer 1202 instructs the throttle / brake controller 1204 to use a throttle / brake 1206 to accelerate or to decelerate based on feedback, e.g., received by the controller 1102 and processed by the velocity analyzer 1202.

[0137] The controller 1102 also has a lateral tracking controller 1208 that influences the operation of the steering wheel controller 1210. For example, the lateral tracking controller 1208 instructs the steering wheel controller 1210 to adjust the position of the steering angle actuator 1212 in accordance with feedback, e.g., received by the controller 1102 and processed by the lateral tracking controller 1208.

[0138] The controller 1102 receives several inputs that are used to determine how to control the throttle / brake 1206 and the steering angle actuator 1212. The planning module 404 provides information that the controller 1102 uses, e.g., to select a course for the AV 100 to follow when the AV 100 begins operation and to determine which road segment to cross when the AV 100 reaches an intersection. The localization module 408 provides information that describes the current location of the AV 100 to the controller 1102, e.g., so that the controller 1102 can determine whether the AV 100 is at a location that is expected based on the manner in which the throttle / brake 1206 and the steering angle actuator 1212 are being controlled. In embodiments, the controller 1102 receives information from other inputs 1214, e.g., information received from a database, a computer network, etc.

[0139] Vehicle operation using behavior rule checking

[0140] Figure 13A An example scenario is shown that illustrates operation of the AV 100 using behavior rule checking in accordance with one or more embodiments. Reference is made to Figure 1 The AV 100 is shown and described in more detail. The AV 100 operates in an environment 190, reference is made to Figure 1 The environment 190 is shown and described in more detail. In Figure 13A In the example scenario shown, the AV 100 is operating in a lane 1316 that is a one-way lane. The environment 190 includes another lane 1320 that is adjacent to the lane 1316 and in an opposite direction from the lane 1316. Another vehicle 193 is operating in the lane 1320. Reference is made to Figure 1 The vehicle 193 is shown and described in more detail. There is a double line 1312 that separates the lane 1316 from the lane 1320. However, there is no physical road divider or median that separates the lane 1316 from the lane 1320. Traffic rules in the environment 190 prohibit vehicles from crossing the double line 1312 or exceeding a speed limit of 45 miles per hour to prevent collisions.

[0141] There is a roadblock 1308 due to an accident in the lane 1316 ahead of the AV 100 in the path of the AV 100. A vehicle 1304 has broken down or suffered a collision in the lane 1316 because of the roadblock 1308. The AV 100 is operating in the lane 1316 toward a destination 199 that is also in the lane 1316. Reference is made to Figure 1Destination 199 is shown and described in more detail. Barricade 1308 and vehicle 1304 are referenced Figure 4 An example of a classified object 416 is shown and described in more detail. AV 100 uses its perception module 402 to identify physical objects 1308, 1304 using one or more sensors 121 (e.g., also as Figure 1 shown). Reference is made to Figure 4 Perception module 402 is shown and described in more detail. Objects 1304, 1308 are classified (e.g., grouped into types such as cars, barricades, traffic cones, etc.) and a scene description including the classified objects 1304, 1308 is provided to planning module (or "planning circuit") 404. Reference is made to Figure 1 Planning circuit 404 is shown and described in more detail.

[0142] AV 100 determines that lane 1316 is blocked by objects 1304, 1308. As referenced Figure 8 As shown and described in more detail, AV 100 detects the boundaries of objects 1304, 1308 based on characteristics of the data points (first sensor data) detected by sensors 121. As Figure 8 shown, planar objects such as lane 1316 will reflect light 804a-804d emitted from LiDAR system 602 in a consistent manner. Reference is made to Figure 6 LiDAR system 602 is shown and described in more detail. In other words, since LiDAR system 602 emits light at a consistent spacing, lane 1316 will reflect the light back to LiDAR system 602 at the same consistent spacing. As AV 100 travels on lane 1316, if nothing is blocking lane 1316, LiDAR system 602 will continue to detect light reflected by the next valid ground point. However, if objects 1304, 1308 are blocking lane 1316, light 804e-804f emitted by LiDAR system 602 will reflect from points 810a-810b in a manner that is inconsistent with the expected consistent manner. From this information, AV 100 can determine that objects 1304, 1308 are present.

[0143] To reach destination 199, planning circuit 404 of AV 100 generates trajectory 198. Reference is made to Figure 1Trajectory 198 is shown and described in more detail. Operating AV 100 according to trajectory 198 causes AV 100 to violate a traffic rule and cross double line 1312 to maneuver around objects 1304, 1308 so that AV 100 can reach its destination 199. Trajectory 198 causes AV 100 to cross double line 1312 and enter lane 1320 in the path of vehicle 193. AV 100 uses a hierarchical rule set of operations to provide feedback about the driving performance of AV 100. The hierarchical rule set is sometimes referred to as a stored behavior model or rulebook. In some embodiments, the feedback is provided in a pass-fail manner. Embodiments disclosed herein are designed to detect when AV 100 (e.g., planning circuit 404) generates a trajectory 198 that violates a higher-priority behavior rule, even if AV 100 can have generated an alternative trajectory that would have only violated a lower-priority behavior rule. The occurrence of such a detection represents a failure of the motion planning process. Reference is made to Figure 13B An example hierarchical rule set 1352 is shown and described in more detail.

[0144] At least one processor is used to generate trajectory 198. In a first embodiment, the at least one processor is located within planning circuit 404 of AV 100. For example, the at least one processor is reference Figure 1 Processor 146 is shown and described in more detail. From this, at least one processor (processor 146 on AV 100) receives first sensor data and second sensor data during operation of AV 100. In a first embodiment, as a trajectory checker or as a rule-based planner / controller, a rule-based control method is executed on AV 100 for real-time evaluation. For example, in an online framework, the AV uses a hierarchical rule set of operations to iteratively provide feedback about the driving performance of AV 100 during operation of the AV (e.g., when the AV crosses a double line). In particular, the online framework activates and deactivates rules depending on local sensing of relevant traffic actors or features (e.g., parked car, pedestrian, road divider).

[0145] In a second embodiment, the at least one processor is located on a computer device external to AV 100. For example, the computer device is reference Figure 1The server 136 is shown and described in more detail. In the second embodiment, at least one processor (of the server 136) receives the first sensor data and the second sensor data after operation of the AV 100. Since the rulebook is scene-agnostic and technology-agnostic, the same rulebook can be used for different scenarios and stack builds to post-hoc modify and improve the planning circuit 404. In an example, the offline framework is configured to develop a transparent and reproducible rule-based pass / fail evaluation of AV trajectories in test scenarios. For example, in the offline framework, a given trajectory output by the planning circuit 404 is rejected if it is found to cause a lesser violation of the rule priority structure. The planning circuit is modified and improved based at least in part on the rejected trajectory and data associated with the rejected trajectory.

[0146] The trajectory 198 is generated based on first sensor data from a first set of sensors (e.g., the sensors 121) of the AV 100 and second sensor data from a second set of sensors (e.g., the sensors 122) of the AV 100. In an embodiment, the first sensor data represents operation of the AV 100 and the second sensor data represents the objects 1304, 1308 located in the environment 190. In an example, the first sensor data is received from the AV 100 and the second sensor data is received from a second device (e.g., the server 136). Figure 1 In an example, the first set of sensors 121 includes at least one of an accelerometer, a steering wheel angle sensor, a wheel sensor, and a brake sensor. The first sensor data includes at least one of a velocity of the AV 100, an acceleration of the AV 100, a heading of the AV 100, an angular velocity of the AV 100, and a torque of the AV 100. In an embodiment, the second set of sensors includes at least one of a LiDAR, a radar, a camera, a microphone, an infrared sensor, a sound navigation and ranging (SONAR) sensor, and the like. Figure 1In examples, the second sensor data includes at least one of an image of the object (e.g., the vehicle 193), a velocity of the vehicle 193, an acceleration of the vehicle 193, and a lateral distance between the vehicle 193 and the AV 100. For ease of description, particular sensors in the first set of sensors and the second set of sensors are described. However, the present technology can be implemented using sensors that characterize information associated with the AV, information associated with the object, information associated with the environment, and any combination thereof. Generally, the first set of sensors are dynamic sensors that capture dynamic data. For example, the dynamic data includes centrifugal force, gravitational force, and velocity, among others. Generally, the second set of sensors are kinematic sensors that capture kinematic data. In examples, the kinematic data describes motion of the object relative to the AV. For example, the kinematic data includes at least one of an image of the at least one object, a velocity of the at least one object, an acceleration of the at least one object, a lateral distance between the at least one object and the vehicle, a longitudinal distance between the at least one object and the vehicle, a jerk of the object, among others. In examples, the data captured by the first set of sensors or the second set of sensors is a time derivative, or a rate of change of a value expressed as a function.

[0147] In embodiments, the processor 146 continuously or periodically receives first sensor data from the first set of sensors 121 of the AV 100 and second sensor data from the second set of sensors 122 of the AV 100. The first sensor data and the second sensor data thus represent a particular scenario in which the AV 100 is operating Figure 13A In examples, the particular scenario is used to determine rules that are activated in the online framework. In embodiments, the processor 146 determines that the trajectory 198 violates a first behavioral rule in a hierarchical set of rules governing operation of the AV 100. The processor 146 determines that the trajectory 198 violates the first behavioral rule (crossing the double line 1312 in the presence of traffic (vehicle 193) in the lane 1320) based on the first sensor data and the second sensor data. For example, the first behavioral rule states that the AV 100 should not cross the double line 1312 in the presence of traffic to prevent a collision. (Alternatively, if the trajectory 198 in use does not violate any behavioral rules, the planning circuit 404 and the AV system 120 pass the behavioral verification process.)

[0148] A violation of the hierarchical set of rules for operation of the AV 100 is determined with respect to one or more objects in the environment 190 (e.g., objects 1304, 1308, and the vehicle 193). For example, a criterion is defined for labeling the trajectory 198 as likely to fail. A simple criterion is a violation of a single behavioral rule, and other formulations are possible. For example, given a trajectory 198 (e.g., a potential trajectory, an actual trajectory, or another trajectory) generated by the planning circuit 404 of the AV 100, the embodiments described herein provide feedback about the trajectory 198 in terms of a priority of the violated rule. In an example, the online framework iteratively updates the trajectory as the AV travels through the environment 190. In this example, the given trajectory is part of or a subset of a larger trajectory.

[0149] In an embodiment, the processor determines a path of the moving object (e.g., the vehicle 193) based on the second sensor data. For example, as the vehicle 193 moves, the processor determines a geometric path formed by successive positions of the endpoint of the position vector of the vehicle 193 over time. The processor can represent the coordinates x, y, and z (e.g., x(t), y(t), and z(t)) of the position vector written as a function of time to represent the evolution of the position of the vehicle 193 over time, i.e., the path of the vehicle 193. The processor determines that the first behavioral rule is violated for the first trajectory 198 based on the path of the vehicle 193. For example, the first behavioral rule can be violated if a point on the trajectory 198 is less than a threshold distance from a point on the path.

[0150] The first behavioral rule (i.e., the rule violated by the trajectory 198) has a first priority. In an embodiment, each behavioral rule in the hierarchical set of rules has a respective priority with respect to each other behavioral rule in the hierarchical set of rules. The respective priority represents a level of risk of violating each behavioral rule with respect to each other behavioral rule. The at least one processor generates a plurality of alternative trajectories for the AV 100 based on the first sensor data and the second sensor data. For example, the plurality of alternative trajectories can be based on a position of the AV 100, a velocity of the AV 100, a position of the vehicle 193, or a velocity of the vehicle 193. Each alternative trajectory represents a choice that can have been made by the AV 100 instead of generating the trajectory 198. The plurality of alternative trajectories is generated by the processor 146 in real-time during operation of the AV 100 (as in the first embodiment described above) or post-hoc on the server 136 (as in the second embodiment described above).

[0151] In an embodiment, a plurality of alternative trajectories are generated using a control barrier function (CBF). A barrier function is a continuous function of a point whose value increases to infinity as the point approaches the boundary of the feasible region of an optimization problem. Such a function can be used to replace inequality constraints by a penalty term in an easier to handle objective function. The CBF takes as input the current system state (e.g., data associated with the location of the AV 100, the rate of the AV 100, the acceleration of the AV 100, or the distance of the AV 100 from the objects 1304, 1308) and outputs a real number corresponding to the safety state of the system. As the system approaches an unsafe operating point, the CBF value increases to infinity. The CBF can be composed with a control Lyapunov function (CLF) to provide joint guarantees on stability, performance, and safety. A Lyapunov function V(x) refers to a scalar function that can be used to determine the stability of the equilibrium of an ordinary differential equation. A CLF refers to a Lyapunov function V(x) for a system with control inputs (e.g., the AV system 120 or the planning circuit 404). A regular Lyapunov function can be used to test whether a dynamic system is stable, i.e., whether a system starting in a certain domain D with state x ≠ 0 will remain in D or will eventually return to x = 0 for asymptotic stability. A CLF is used to test whether a system is feedback stabilizable, i.e., whether for a state x there exists a control u(x, t) such that the system can be brought to the zero state by applying the control u. For example, the offline framework implements trajectory tracking by using additional constraints implemented with a CLF. In the online framework, a reference trajectory is tracked by including the tracking error in the cost and by optimizing over a moving horizon (MPC).

[0152] The at least one processor identifies a second trajectory from the plurality of alternative trajectories. For example, according to the second trajectory, the AV 100 is about to stop in the lane 1316 and then cross the double line 1312 after the vehicle 193 has passed. The second trajectory thus only violates a second behavioral rule in the hierarchy of rules set (crossing the double line 1312 when there is no traffic in the lane 1320). The second behavioral rule has a second priority that is less than the first priority. Preventing a collision (crossing the double line 1312 when there is traffic in the lane 1320) has a greater priority than simply crossing the double line 1312 when there is no traffic in the lane 1320. Alternatively, if the second trajectory violates a higher priority rule, the planning circuit 404 and the AV system 120 pass the behavioral verification because no alternative trajectory with a lesser degree of rule violation is found.

[0153] In the first embodiment described above, in response to identifying the second trajectory, the at least one processor 146 sends a message to the control circuit 406 of the AV 100 to operate the AV 100 based on the second trajectory. With reference to Figure 4The control circuit 406 is shown and described in more detail. For example, if the at least one processor 146 is on the planning circuit 404, the AV 100 is operated in real-time based on the second trajectory. If the at least one processor is on the offline server 136, the second trajectory is used to reprogram the planning circuit 404, as in the second embodiment described above. In embodiments, the result of feedback from trajectory validation is either a “pass” (e.g., the trajectory 198 is satisfactory, or an alternative trajectory is not available) or a “fail” (e.g., the AV trajectory 198 does not comply with the rulebook behavior specification, and there is an alternative trajectory available that does not violate the behavior rule or violates a lower-priority behavior rule compared to the trajectory 198). The trajectory 198 is considered to be a “fail” if such an alternative trajectory is identified.

[0154] The embodiments disclosed herein are designed to prevent a “mildly satisfactory” trajectory (e.g., a trajectory in which the AV 100 is about to stop or does not reach its goal 199) from being considered as an alternative solution to a trajectory that reaches the goal with a rule violation. The “reach goal” rule is explicitly constructed into the rulebook. The processor 146 operates the AV 100 based on the trajectory 198 to avoid collisions of the AV 100 with the objects 1304, 1308, and the vehicle 193. For example, referring to Figure 4 The control module 406 that is shown and described in more detail operates the AV 100.

[0155] Figure 13B An example hierarchical rule set 1352 according to one or more embodiments is shown. The stored behavior rules of operation of the AV 100 include a plurality of behavior rules. Referring to Figure 1 、 Figure 13A The AV 100 is shown and described in more detail. Individual behavior rules (e.g., rule 1356) have a priority relative to individual other rules (e.g., rule 1360). The priority represents a risk level of violating the stored behavior rules 1352. Thus, the rulebook 1352 is a formal framework to specify driving requirements that are expected to be enforced by traffic laws or culture and their relative priorities. The rulebook 1352 is a pre-ordered rule set with violation scores that embody the hierarchy of rule priorities. Thus, the rulebook 1352 enables AV behavior specification and evaluation in collision scenarios. Consider a situation in which a pedestrian 192 walks into a lane in which the AV 100 is driving. Referring to Figure 1 The pedestrian 192 is shown and described in more detail. A reasonable AV behavior would be to avoid collision with the pedestrian 192 and other vehicles 193 (high priority), despite the cost of violating lower-priority rules by reducing the speed to less than the minimum speed limit or by deviating from the lane.

[0156] In an embodiment, violating the behavior rule includes operating the AV 100 such that the AV 100 collides with the vehicle 193. Reference is made to Figure 1 , Figure 13A The vehicle 193 is shown and described in more detail. For example, the risk of collision between the AV 100 and the vehicle 193 in the case of violating the rule 1360 is greater than in the case of violating only the rule 1356. Thus, the rule 1360 has a higher priority than the rule 1356. Similarly, the rule 1372 has a higher priority than the rules 1368, 1364.

[0157] In an embodiment, violating the behavior rule includes operating the AV 100 such that the AV 100 exceeds a speed limit (e.g., 45 mph). For example, the rule 1356 indicates that the AV 100 should not violate the speed limit of the lane in which it is traveling. For example, in Figure 13A , the speed limit of the lane 1316 is 45 miles per hour. However, the rule 1356 is a lower priority rule; thus, the AV 100 can violate the rule 1356 to prevent a collision (e.g., with the vehicle 193) and act in accordance with the rule 1372. In an embodiment, violating the behavior rule includes operating the AV 100 such that the AV 100 stops before reaching the destination 199. Reference is made to Figure 1 , Figure 13A The destination 199 is shown and described in more detail. For example, the rule 1360 indicates that the AV 100 should stay in its own lane. For example, in Figure 13A , the AV 100 is traveling in the lane 1316. However, the priority of the rule 1360 is lower than the priority of the rule 1372. Thus, as shown and described in more detail with reference to Figure 13A , the AV 100 violates only the rule 1360 to avoid collisions with the objects 1304, 1308 and complies with the two higher priority rules 1368 (reach the destination 199) and 1372 (avoid a collision).

[0158] In an embodiment, violating the stored behavior rule 1352 of operation of the AV 100 includes operating the AV 100 such that the lateral separation between the AV 100 and the objects 1304, 1308 decreases to less than a threshold lateral distance. For example, the rule 1364 indicates that the AV 100 should maintain a threshold lateral distance (e.g., half a car length or 1 meter) from any other object (e.g., the objects 1304, 1308). However, the priority of the rule 1364 is lower than the priority of the rule 1368 (reach the destination 199). Thus, as shown and described in more detail with reference to Figure 13A , the AV 100 can violate the rule 1364 to comply with the higher priority rules 1368 (reach the destination 199) and 1372 (avoid a collision).

[0159] In this embodiment, alternative safety metrics are used to assess AV safety. Alternative safety metrics are used to more quickly evaluate road safety and integrate concepts into the overall theoretical framework. The priority of operational rules (e.g., Rule 1356) can be adjusted based on the frequency of violations. For example, empirical evidence from human driver data can be used to support this assessment. Figure 13B The application of stored behavior rule 1352 to road safety.

[0160] Figure 14 An example flowchart illustrating the operation of AV 100 using behavior rule checking according to one or more embodiments is shown. (See reference...) Figure 13A In the first embodiment described, by (referencing) Figure 1 (A more detailed description) The processor 146 of the AV 100 performs... Figure 14 The processing involves at least one processor 146 located within the planning circuitry 404 of the AV 100. During operation of the AV 100, at least one processor 146 receives first sensor data and second sensor data (AV behavior). Thus, acting as a trajectory checker or as the rule-based planning circuitry 404 or controller, the rule-based control method described herein (selecting AV behavior) is executed on the AV 100 for real-time evaluation. Furthermore, embodiments may include different and / or additional steps, or steps performed in a different order. Reference Figure 4 The planned circuit 404 is shown and described in more detail.

[0161] In (reference) Figure 13A In the second embodiment described, at least one processor (on server 136) is based on (reference) Figure 13A The second trajectory (described) is used to adjust the operation of the planning circuit 404 of AV 100. In the second embodiment, at least one processor is located on a computer device (server 136) external to AV 100. After the operation of AV 100, server 136 receives first sensor data (AV behavior) and second sensor data. For example, as Figure 14 As shown, the motion planning process of the planning circuit 404 is adjusted based on the frequency of violations of behavioral rules. For example, in the application (reference) Figure 13B (Showing and describing in more detail) A validated rulebook 1352 is used for the design and implementation of autonomous vehicle systems 120. In the case of machine drivers that typically have a system model, the rulebook can be used to evaluate AV driving performance (evaluate AV behavior).

[0162] In this embodiment, the risk level (interpretation of AV behavior) of the motion planning process of AV 100 is determined based on the frequency of violations of one or more of the stored behavior rules 1352. (See reference...) Figure 13BRules 1352 are shown and described in more detail. For example, as shown in FIG. 13, the performance of the design and planning circuit 404 of the AV system 120 models the impact on the planned trajectory. Referring to FIG. 14, the AV system 120 is shown and described in more detail. Referring to FIG. 15, the planning circuit 404 is shown and described in more detail. The planned trajectory is scored to measure overall driving performance, according to the design of the AV system 120 and the performance of the subsystem (planning circuit 404). The (sub)system requirements are derived from the behavioral norms (rules 1352), the performance is optimized, and the resources are prioritized. Figure 14 Figure 1 The AV system 120 is shown and described in more detail. Referring to FIG. 15, the planning circuit 404 is shown and described in more detail. The planned trajectory is scored to measure overall driving performance, according to the design of the AV system 120 and the performance of the subsystem (planning circuit 404). The (sub)system requirements are derived from the behavioral norms (rules 1352), the performance is optimized, and the resources are prioritized. Figure 4

[0163] In embodiments, the at least one processor uses at least one of a minimal violation planning, model predictive control (MPC), and machine learning to select the second trajectory from the plurality of alternative trajectories. Minimal violation planning refers to a method for path planning for the AV 100 that enables the use of multiple continuous objectives (e.g., finding the shortest path) with discrete constraints from logic (such as constraints resulting from the hierarchical ruleset 1352). MPC refers to a method for controlling the process (trajectory generation and selection) while satisfying a set of constraints (the hierarchical ruleset 1352). In embodiments, the MPC uses a dynamic model of the AV system 120 as a linear empirical model. Referring to FIG. 16, the AV system 120 is shown and described in more detail. Figure 1 The AV system 120 is shown and described in more detail. Machine learning refers to the use of a model that is automatically improved through experience to generate alternative trajectories. The AV system 120 or the server 136 builds a mathematical model based on sample data (referred to as “training data”) to make predictions or decisions without being explicitly programmed to do so. For example, the training data for selecting the second trajectory is the hierarchical ruleset 1352 and known outcomes of violating particular rules. Thus, as shown in FIG. 17, in the second embodiment described with reference to FIG. 16, candidate trajectories are selected a posteriori from a plurality of methods with information about the scenario and how the AV 100 performed. Figure 14 Figure 13A In the second embodiment described with reference to FIG. 16, candidate trajectories are selected a posteriori from a plurality of methods with information about the scenario and how the AV 100 performed.

[0164] For example, an online framework implements a receding horizon (model predictive control, MPC) optimization, where the reference trajectory tracking error is included in the cost. In the online framework, active rules at a given time (e.g., rules corresponding to detected instances or particular scenarios) add constraints to the optimization problem in the online situation. Rules are categorized as instance-dependent rules (such as spacing from pedestrians, spacing from parked cars, etc.) and instance-independent rules (such as speed limits and comfort, etc.). Instance-independent rules should always be considered. However, instance-dependent rules are only considered when the corresponding instance is within the local sensing range of the AV. The local sensing range generally refers to the range of sensor data available to the AV (such as data captured by sensors located on or associated with the AV, etc.).

[0165] ​​​In embodiments, at initialization or at time t = 0, instance-dependent rules are deactivated in the hierarchical rule set. As instances occur, the corresponding instance-dependent rules are activated. For each instance at the current time t, the deactivated rules (e.g., rules that do not apply to the current instance) are removed from the hierarchical rule set. Thus, in the online approach, the hierarchical rule set is iteratively modified as instances occur. In examples, the modification occurs periodically according to a predetermined time period. In examples, the activated rules are activated as long as the corresponding instance occurs.

[0166] Referring again to Figure 13B , a set of hierarchical rule sets 1352 is provided. Consider the example in which a roadblock and object (e.g., 1304, 1308 in Figure 13A blocks the travel lane of the AV, as described with respect to Figure 13A . In this example, the instance is the roadblock in the travel lane. As shown in the example of Figure 13A , there are no pedestrians within the local sensing range of the AV. In this example, the rules associated with pedestrians (e.g., no instance detected) are deactivated. As the AV navigates around the roadblock, the rules associated with the instance of the detected pedestrian are removed from the hierarchical rule set.

[0167] Figure 15 An example flowchart of vehicle operation using behavioral rule checking is shown in accordance with one or more embodiments. In a first embodiment (described with respect to Figure 13A ), the processing of Figure 1 is performed by the processor 146 of the AV 100, described in more detail with respect to Figure 15 . In a second embodiment (described with respect to Figure 13A ), the processing of Figure 15 is performed by at least one processor (on the server 136). Further, embodiments can include different and / or additional steps, or perform steps in a different order.

[0168] In step 1504, the processor determines whether the first trajectory (e.g., trajectory 198) violates any behavioral rules of the hierarchical rule set 1352 of the operation of the AV 100 based on the first sensor data and the second sensor data. The trajectory 198 is shown and described in more detail with respect to Figure 1 , Figure 13A The hierarchical rule set 1352 is shown and described in more detail with respect to Figure 13B . In step 1508, if the processor finds no violated rules, the processing moves to step 1512 and the planning circuit 404 and AV behavior pass the verification check. The planning circuit 404 is shown and described in more detail with respect to Figure 4 .

[0169] In step 1508, if the processor detects a rule violation, processing proceeds to step 1516. The violated rule is represented as a first behavior rule with a first priority. In step 1516, the processor determines whether there are alternative trajectories with fewer violations. For example, the processor generates multiple alternative trajectories for AV 100 based on first sensor data and second sensor data. (See reference...) Figure 13A In more detail, multiple alternative trajectories can be generated using CBF. The processor identifies whether there exists a second trajectory that violates only the second behavior rule in hierarchical rule set 1352, such that the second behavior rule has a second priority lower than the first priority.

[0170] If no other trajectory exists that violates only a second behavior rule with a lower priority than the first priority, the process moves to step 1520. Planning circuitry 404 and the AV behavior pass the verification check. In step 1516, if the processor determines that an alternative trajectory with fewer violations exists, planning circuitry 404 and the AV behavior causes the verification check to fail. Optionally, the processor may move to step 1528 and determine whether to stop optimization (further move to step 1532 and terminate) or move to step 1536. In step 1536, the processor examines each of the multiple alternative trajectories to identify the least violating trajectory, e.g., an alternative trajectory that does not violate a rule or violates a rule with the lowest priority of any violated rule. The least violating trajectory can be used (in reference...) Figure 13A In the first embodiment described, AV 100 is operated or (in reference) Figure 13A (In the second embodiment described) Adjustment planning circuit 404.

[0171] Figure 16 Examples of behavior rule checks performed on AV 100 according to one or more embodiments are shown. References Figure 1 , Figure 13A The AV 100 is shown and described in more detail. Figure 16 In this process, a graphical user interface is displayed on server 136, where multiple alternative trajectories are generated. The different trajectories are examined to identify the least violated trajectory based on data from the first and second sensors. (Referencing...) Figure 13A In the second embodiment described, when as Figure 16 The data generated on the graphical user interface during behavior rule checking of the AV 100, as shown, was used to adjust and improve trajectory generation via planning circuit 404. (Reference) Figure 4 The planned circuit 404 is shown and described in more detail.

[0172] Figure 16The implementation of the AV control strategy is shown to meet complex specifications designed according to traffic regulations (e.g., Rulebook 1352) and cultural expectations of reasonable driving behavior. These specifications are designated as rules by constructing a pre-ordered structure called Rulebook 1352 (see...). Figure 13B The disclosed embodiments present a recursive framework in which the satisfaction of rules in rulebook 1352 is iteratively relaxed based on rule priority. In the embodiments, convergence to the desired state is achieved using (CLF), and security is enforced by CBF. CLF can be used to stabilize the system to the desired state. CBF can be used to enforce forward invariance and improve the satisfaction of security requirements. The framework can be used for post-trajectory pass / fail evaluation (if the processing finds that the controller generates an alternative trajectory that causes a smaller violation of rulebook 1352, then the given trajectory 198 is rejected).

[0173] Figure 17 An example flowchart illustrating vehicle operation using behavior rules for inspection according to one or more embodiments is shown. (Refer to...) Figure 13A In the first embodiment described, by referring to Figure 1 The processor 146 of the AV 100 is described in more detail. Figure 17 The processing. (See reference) Figure 13A In the second embodiment described, at least one processor (on server 136) performs... Figure 17 The processing. Furthermore, embodiments may include different and / or additional steps, or steps performed in a different order.

[0174] In step 1704, the processor determines whether the trajectory of AV 100 (e.g., trajectory 198) is acceptable. (See reference) Figure 1 , Figure 13A Trajectory 198 and AV 100 are shown and described in more detail. For example, in step 1704, the processor determines, based on the first sensor data and the second sensor data, whether trajectory 198 violates any behavioral rules in the hierarchical rule set 1352 of the operation of AV 100. (See reference...) Figure 13B The hierarchical rule set 1352 is shown and described in more detail. In step 1704, if the processor finds no rule violation, processing proceeds to step 1708, and the planning circuit 404 and AV behavior pass the verification check. (Reference) Figure 4 The planned circuit 404 is shown and described in more detail.

[0175] At step 1704, if the processor finds a rule violation, processing moves to step 1712. The violated rule is denoted as a first behavioral rule having a first priority. Processing moves to step 1716. At step 1716, the processor determines whether there is an alternative less-violating trajectory. For example, the processor generates a plurality of alternative trajectories for the AV 100 based on the first sensor data and the second sensor data. The plurality of alternative trajectories can be generated using a control barrier function as described in more detail with reference to Figure 13A The plurality of alternative trajectories is generated using a control barrier function as described in more detail with reference to

[0176] If there is no other trajectory that only violates a second behavioral rule in the hierarchy of rules 1352 having a second priority less than the first priority, processing moves to step 1720. The planning circuit 404 and the AV behavior pass the verification check. If, at step 1716, the processor determines that there is an alternative less-violating trajectory, the planning circuit 404 and the AV behavior fail the verification check.

[0177] Figure 18 Example outputs for behavioral rule checking for a vehicle are shown in accordance with one or more embodiments. In a first embodiment (described with reference to Figure 13A In a first embodiment (described with reference to Figure 1 The processor 146 of the AV 100 uses the example outputs described in more detail with reference to Figure 18 In a second embodiment (described with reference to Figure 13A The outputs of the at least one processor (on the server 136) use the example outputs described in more detail with reference to Figure 18 The example output indicates that the candidate trajectory (e.g., trajectory 198) under verification violates rule R10 (minimum lateral spacing from other active vehicles on the road). For example, the trajectory 198 causes the AV 100 to operate closer than a minimum threshold distance to an active vehicle (e.g., vehicle 193). Reference is made to Figure 1 , Figure 13A The vehicle 193 is shown and described in more detail. The example output indicates that the second (alternative) trajectory complies with rule R10.

[0178] The example output indicates that the candidate trajectory 198 violates rule R8 (minimum lateral spacing from other non-active vehicles on the road). For example, the trajectory 198 causes the AV 100 to operate closer than a minimum threshold distance to a non-active vehicle (e.g., vehicle 1304). Reference is made to Figure 13A The vehicle 1304 is shown and described in more detail. The example output indicates that the alternative trajectory complies with rule R8. Rule R10 has a higher priority than rule R8, which means that the AV 100 should strive to satisfy rule R10 even if rule R8 has to be violated to do so.

[0179] The example output indicates that the candidate trajectory 198 complies with rule R4b (minimum speed limit on road). For example, the trajectory 198 causes the AV 100 to drive slower than the minimum speed limit. The example output indicates that the alternative trajectory violates rule R4b. Rules R8, R10 have a higher priority than rule R4b, which means that the AV 100 should strive to satisfy rules R8, R10 even if rule R4b must be violated to do so. However, the trajectory 198 causes the AV 100 to comply with rule R4b while violating rules R8, R10. The alternative trajectory causes the AV 100 to comply with rules R8, R10 while violating rule R4b. Thus, the trajectory check for trajectory 198 fails and the alternative trajectory is used.

[0180] Figure 19 An example flowchart of vehicle operation using behavior rule checking is shown in accordance with one or more embodiments. In a first embodiment (described with reference to Figure 13A In a second embodiment (described with reference to Figure 1 The processing by the processor 146 of the AV 100 (described in more detail) is performed in a first embodiment. In a second embodiment, the processing by the at least one processor (on the server 136) is performed. Figure 19 The processing by the processor 146 of the AV 100 (described in more detail) is performed in a first embodiment. In a second embodiment, the processing by the at least one processor (on the server 136) is performed. Figure 13A The processing by the processor 146 of the AV 100 (described in more detail) is performed in a first embodiment. In a second embodiment, the processing by the at least one processor (on the server 136) is performed. Figure 19 The processing by the processor 146 of the AV 100 (described in more detail) is performed in a first embodiment. In a second embodiment, the processing by the at least one processor (on the server 136) is performed.

[0181] In step 1904, the processor receives first sensor data from a first set of sensors 120 of the AV 100 and second sensor data from a second set of sensors 121 of the AV 100. The first sensor data represents operation of the AV 100 according to a first trajectory 198. The second sensor data represents at least one object 1304, 1308. The sensors 120, 121 are shown and described in more detail with reference to Figure 1 The trajectory 198 is shown and described in more detail with reference to Figure 1 , Figure 13A The trajectory 198 is shown and described in more detail with reference to Figure 13A The objects 1304, 1308 are shown and described in more detail with reference to

[0182] In step 1908, the processor determines that the first trajectory 198 violates a first behavior rule (e.g., rule 1360) of a set of hierarchical rules 1352 of operation of the AV 100 based on the first sensor data and the second sensor data. The rule 1360 and the set of hierarchical rules 1352 are shown and described in more detail with reference to Figure 13B The rule 1360 and the set of hierarchical rules 1352 are shown and described in more detail with reference to

[0183] In step 1912, the processor generates a plurality of alternative trajectories for the AV 100 based on the first sensor data and the second sensor data. The plurality of alternative trajectories is generated using the CBF. The processor iteratively relaxes the rules that need to be satisfied to determine whether there is a second trajectory with fewer violations. The processor uses the CLF and the CBF, which together guarantee that if there is a feasible lower violation trajectory, the algorithm will converge to that trajectory. Iteratively relaxing the rules can be used with other trajectory generation methods, including graph-based search, combined MPC, or machine learning-based planning methods.

[0184] In step 1916, the processor identifies a second trajectory from the plurality of alternative trajectories. The second trajectory violates a second behavioral rule (e.g., rule 1356) in the set of hierarchical rules 1352. Reference is made to Figure 13B The rule 1356 is shown and described in more detail. The second behavioral rule has a second priority that is less than the first priority. The requirement constraint is continuously differentiable, which makes the optimization problem quadratic. A continuously differentiable function is a function for which the derivative exists for each point in its domain. In other words, the graph of a continuously differentiable function has non-vertical tangent lines at every interior point in its domain. Approximating the rule with a conservative, differentiable function that is faster to evaluate than the more complex rule makes the rule appear non-differentiable. Since the optimization problem is quadratic, the computational complexity is reduced. For example, a nonlinear solver such as a Newton-Krylov solver, an Anderson solver, or a Broyden solver, etc. can be used to solve the optimization problem by modeling the AV system 120 as a nonlinear system. Thus, this approach is easier to implement on the embedded software of the AV 100 while meeting stringent vehicle safety requirements.

[0185] In step 1920, in response to identifying the second trajectory, the processor sends a message to the control circuit 406 of the AV 100 to operate the AV 100 based on the second trajectory. Reference is made to Figure 4 The control circuit 406 is shown and described in more detail. The embodiments disclosed herein extend beyond on-board planning and control by providing a scalable, objective way to pass or fail AV behavior in a post-hoc testing scenario. Post-hoc evaluation can help validate the driving choices made by the AV 100 in the real world by objectively proving that no more “reasonable” choices were available to the AV 100.

[0186] Figure 20 An example set of hierarchical rules for operation of the AV 100 using behavioral rule checking is shown in accordance with one or more embodiments. Reference is made to Figure 1 、 Figure 13AAV 100 is shown and described in more detail. Behavioral rules specify the desired behavior of AV 100 so that AV 100 complies with traffic laws, ethical norms, and local culture, e.g., “stay in lane,” “maintain distance from pedestrian 192,” “observe maximum speed limit,” “reach goal 199 within deadline,” etc. Reference is made to Figure 1 Pedestrian 192 is shown and described in more detail.

[0187] Rules are interpreted on a vehicle trajectory. Given a trajectory 198 and a rule, a violation score embodies how much the trajectory 198 violates the rule. Reference is made to Figure 1 、 Figure 13A Trajectory 198 is shown and described in more detail. For example, if AV 100 crosses double line 1312 and reaches lane 1320 along trajectory 198 at a distance of 1 m, then the violation score of trajectory 198 for the “stay in lane” rule is 1 m. Reference is made to Figure 13A Double line 1312 and lane 1320 are shown and described in more detail.

[0188] Rulebook 1352 defines the priority of rules and imposes a pre-ordering that can be used to rank AV trajectories. Reference is made to Figure 13B Rulebook 1352 is shown and described in more detail. Rulebook 1352 is a tuple <R,≤>, where R denotes a finite set of rules and ≤ denotes a pre-ordering on R. Rulebook 1352 can also be represented by a directed graph, where each node is a rule, and an edge between two rules means that the first rule has higher priority than the second rule. Formally, r1→r2 in the graph means that r1≤r2 (r2∈R has higher priority than r1∈R). Using the pre-ordering, two rules can be one of three relationships: comparable (one has higher priority than the other), incomparable, or equivalent (each has equal priority).

[0189] Figure 20 The rulebook shown in the middle includes six rules. In this example, rules r1 and r2 are incomparable, and both have higher priority than rules r3 and r4. Rules r3 and r4 are equivalent (r3≤r4 and r4≤r3), but are incomparable with rule r5. Rule r6 has the lowest priority among all rules.

[0190] In the foregoing description, embodiments of the application have been described with reference to numerous specific details that can vary from implementation to implementation. Accordingly, the specification and drawings are to be regarded in an illustrative rather than a restrictive sense. The sole and exclusive indicator of the scope of the application, and what is intended by the applicants to be the scope of the application, is the literal and equivalent scope of the claims issued after publication of this application and any subsequent correction. Any definitions of terms here in this detailed description are expressly incorporated by reference from the specific usage of such terms as set forth in the claims. In addition, when in this detailed description, the terms "comprises", "comprising", "has", "having", "includes", "including", "contains", "containing" or the negative of any of these terms are used, the meaning of these terms is that the mentioned step, feature, structure, or element is included, but not that with only the stated feature, step, structure, or element necessarily exists in the implementation. In addition, the term "comprising" is used herein to mean including whatever follows the term, but not to the exclusion of anything not mentioned. Further, the term "comprising" is used herein to refer to compositions, methods, and processes that include the recited elements, but not excluding others.

[0191] CROSS-REFERENCE TO RELATED APPLICATIONS

[0192] This application claims priority to U.S. provisional application 63 / 105,006, filed October 23, 2020, and U.S. provisional application 63 / 216,953, filed June 30, 2021, the entire contents of which are incorporated herein by reference.

Claims

1. A method for a vehicle, comprising: receiving, with at least one processor, first sensor data from a first set of sensors of the vehicle and second sensor data from a second set of sensors of the vehicle, the first sensor data representing operation of the vehicle according to a first trajectory, and the second sensor data representing at least one object; determining, with the at least one processor, a violation of a first behavioral rule of a plurality of hierarchical rules of operation of the vehicle for the first trajectory based on the first sensor data and the second sensor data, the first behavioral rule having a first priority; generating, with the at least one processor, a plurality of alternative trajectories of the vehicle based on the first sensor data and the second sensor data, the plurality of alternative trajectories generated using a control barrier function; identifying, with the at least one processor, a second trajectory from the plurality of alternative trajectories, wherein the second trajectory violates a second behavioral rule of the plurality of hierarchical rules, the second behavioral rule having a second priority less than the first priority; and in response to identifying the second trajectory, sending, with the at least one processor, a message to a control circuit of the vehicle to operate the vehicle based on the second trajectory.

2. The method of claim 1, wherein, the at least one processor is located within a planning circuit of the vehicle, and wherein the at least one processor receives the first sensor data and the second sensor data during operation of the vehicle.

3. The method of claim 1, further comprising: adjusting, with the at least one processor, operation of the planning circuit of the vehicle based on the second trajectory, wherein the at least one processor is located on a computer device external to the vehicle, and wherein the at least one processor receives the first sensor data and the second sensor data after operation of the vehicle.

4. The method of claim 1, wherein, the first set of sensors includes at least one of an accelerometer, a steering wheel angle sensor, a wheel sensor, and a brake sensor.

5. The method of claim 1, wherein, the first sensor data includes at least one of a velocity of the vehicle, an acceleration of the vehicle, a heading of the vehicle, an angular velocity of the vehicle, and a torque of the vehicle.

6. The method of claim 1, wherein, the second set of sensors includes at least one of a LiDAR, a radar, a camera, a microphone, an infrared sensor, and a sound navigation and ranging sensor (SONAR) sensor.

7. The method of claim 1, wherein, the second sensor data includes at least one of an image of the at least one object, a velocity of the at least one object, an acceleration of the at least one object, a lateral distance between the at least one object and the vehicle, and other kinematic data.

8. The method of claim 1, further comprising: selecting, with the at least one processor, the second trajectory from the plurality of alternative trajectories using at least one of a minimal violation planning, a model predictive control, and a machine learning, the selection based on the plurality of hierarchical rules.

9. The method of claim 1, wherein, Each behavioral rule of the plurality of hierarchical rules has a respective priority relative to each other behavioral rule of the plurality of hierarchical rules, the respective priority representing a risk level of violating the respective behavioral rule relative to the other behavioral rule.

10. The method of claim 1, wherein, Violating the first behavioral rule includes operating the vehicle such that a lateral distance between the vehicle and the at least one object decreases to be less than a threshold lateral distance.

11. The method of claim 1, wherein, Violating the first behavioral rule includes operating the vehicle such that the vehicle exceeds a speed limit.

12. The method of claim 1, wherein, Violating the first behavioral rule includes operating the vehicle such that the vehicle stops before reaching a destination.

13. The method of claim 1, wherein, Violating the first behavioral rule includes operating the vehicle such that the vehicle collides with the at least one object.

14. The method of claim 1, further comprising: Determining, with the at least one processor, a path of the at least one object based on the second sensor data, wherein determining that the first trajectory violates the first behavioral rule is further based on the path of the at least one object.

15. An autonomous vehicle, comprising: one or more processors; and one or more non-transitory storage media having stored instructions that, when executed by the one or more processors, cause performance of any of the methods of claims 1-14.

16. One or more non-transitory storage media storing instructions that, when executed by one or more computing devices, cause performance of any of the methods of claims 1-14.

17. A method for a vehicle, comprising performing operations involving machine execution of instructions that, when executed by one or more computing devices, cause performance of any of the methods of claims 1-14, wherein, the machine execution of the operations being at least one of sending the instructions, receiving the instructions, storing the instructions, and executing the instructions.

18. A computer program product comprising a program for causing a computer to perform any of the methods of claims 1-14.

Images