Methods, systems, and storage media for vehicles

Abstract

This disclosure relates to methods, systems, and storage media for vehicles. Methods for graphical exploration used in rulebook trajectory generation are provided. Some described methods include generating a set of next alternative trajectories for a vehicle from a next pose, the set of next alternative trajectories representing the vehicle's operation from the next pose, where the next pose is at the end of an identified trajectory. Next trajectories are iteratively identified from the corresponding set of next alternative trajectories, wherein the next trajectory violates a minimum behavior rule among a hierarchical set of rules, the minimum behavior rule having a lower priority than behavior rules associated with other trajectories in the corresponding set of next alternative trajectories, until a target pose is reached or a timeout occurs to generate a graph. Systems and computer program products are also provided.

Description

Technical Field

[0001] This disclosure relates to methods, systems, and storage media for vehicles, and particularly to graph exploration for rulebook trajectory generation. Background Technology

[0002] The operation of a vehicle from its initial location to its final destination typically requires the user or the vehicle's decision-making system to select a route through the road network. The route may involve fulfilling objectives, such as not exceeding maximum driving time. Furthermore, the vehicle may need to comply with complex regulations imposed by traffic laws and cultural expectations of driving behavior. Therefore, the operation of autonomous vehicles may require numerous decisions, making traditional autonomous driving algorithms impractical. Summary of the Invention

[0003] According to one aspect of the present invention, a method for a vehicle is provided, comprising: using at least one processor to generate an alternative trajectory set for the vehicle in a first posture, the alternative trajectory set representing the operation of the vehicle from the first posture; using the at least one processor to identify trajectories from the alternative trajectory set, wherein the trajectories violate a minimum behavior rule among a plurality of hierarchical rules, the minimum behavior rule having a lower priority than behavior rules associated with other trajectories in the alternative trajectory set; using the at least one processor, in response to the identification of the trajectory, generating a next alternative trajectory set for the vehicle from a next posture, the next alternative trajectory set representing the operation of the vehicle from the next posture, wherein the next posture is located at the end of the identified trajectory; using the at least one processor to iteratively identify next trajectories from a corresponding next alternative trajectory set until a target posture is reached or a timeout occurs to generate a graph, wherein the next trajectory violates a minimum behavior rule among the plurality of hierarchical rules, the minimum behavior rule having a lower priority than behavior rules associated with other trajectories in the corresponding next alternative trajectory set; and using the at least one processor to transmit a message to a control system of the vehicle to operate the vehicle based on the graph.

[0004] According to another aspect of the invention, the above method further includes using the at least one processor to prune trajectories from the set of alternative trajectories or the next set of alternative trajectories based on the probability of becoming the optimal trajectory.

[0005] According to another aspect of the invention, the method further includes using the at least one processor to trim trajectories from the alternative trajectory set or the next alternative trajectory set based on the remaining trajectories on the roadway.

[0006] According to another aspect of the present invention, a system for a vehicle is provided, comprising: at least one processor, and at least one non-transitory storage medium storing instructions that, when executed by the at least one processor, cause the at least one processor to perform the method described above.

[0007] According to another aspect of the invention, at least one non-transitory storage medium is provided, which stores instructions that, when executed by at least one processor, cause the at least one processor to perform the method described above. Attached Figure Description

[0008] Figure 1 It is an example environment that can realize a vehicle that includes one or more components of an autonomous system;

[0009] Figure 2 It is a diagram of one or more systems that include autonomous vehicles;

[0010] Figure 3 yes Figure 1 and Figure 2 A diagram of one or more devices and / or one or more system components;

[0011] Figure 4 It is a diagram of some components of an autonomous system;

[0012] Figure 5 This is a schematic diagram illustrating the implementation of a graphical exploration process used for generating trajectory data from a rulebook.

[0013] Figure 6 An example scenario is shown, illustrating the operation of an autonomous vehicle using graphical exploration with behavioral rule checks.

[0014] Figure 7 An example flowchart is shown for the process of using behavior rule checks to determine vehicle operations for a fixed set of trajectories;

[0015] Figure 8 It is a diagram of an iterative growing graph used to discover the best trajectory after the fact;

[0016] Figure 9 It is a diagram of the system that calculates scores according to the rule manual;

[0017] Figure 10 This is a flowchart for the graphical exploration process used in generating trajectory data from the rulebook. Detailed Implementation

[0018] In the following description, numerous specific details are set forth for purposes of explanation in order to provide a thorough understanding of this disclosure. However, it will be apparent that the embodiments described herein can be practiced without these specific details. In some instances, well-known constructions and apparatuses are illustrated in block diagram form to avoid unnecessarily obscuring aspects of this disclosure.

[0019] In the accompanying drawings, for ease of description, specific arrangements or orders of schematic elements (such as those representing systems, devices, modules, instruction blocks, and / or data elements) are illustrated. However, those skilled in the art will understand that, unless explicitly described, the specific order or arrangement of schematic elements in the drawings is not intended to imply a requirement for a particular processing order or sequence, or separation of processes. Furthermore, unless explicitly described, the inclusion of schematic elements in the drawings is not intended to imply that such elements are required in all embodiments, nor is it intended to imply that features represented by such elements cannot be included in some embodiments or cannot be combined with other elements in some embodiments.

[0020] Furthermore, in the accompanying drawings, connecting elements (such as solid or dashed lines or arrows) are used to illustrate connections, relationships, or associations between or among two or more other schematic elements. The absence of any such connecting element does not imply that connections, relationships, or associations cannot exist. In other words, some connections, relationships, or associations between elements are not illustrated in the drawings so as not to obscure the content of this disclosure. Additionally, for ease of illustration, a single connecting element may be used to represent multiple connections, relationships, or associations between elements. For example, if a connecting element represents communication of signals, data, or instructions (e.g., "software instructions"), those skilled in the art will understand that such an element may represent one or more signal paths (e.g., a bus) that may be necessary to influence the communication.

[0021] Although the terms "first," "second," and / or "third," etc., are used to describe various elements, these elements should not be limited by these terms. The terms "first," "second," and / or "third" are used only to distinguish one element from another. For example, without departing from the scope of the described embodiments, a first contact may be referred to as a second contact, and similarly, a second contact may be referred to as a first contact. Both the first contact and the second contact are contacts, but they are not the same contact.

[0022] The terminology used in the description of the various embodiments described herein is included for the purpose of describing particular embodiments only and is not intended to be limiting. As used in the description of the various embodiments described and the appended claims, the singular forms “a,” “an,” and “the” are also intended to include the plural forms and may be used interchangeably with “one or more” or “at least one” unless the context clearly indicates otherwise. It will also be understood that the term “and / or” as used herein refers to and includes any and all possible combinations of one or more of the associated listed items. It will also be understood that when the terms “comprising,” “including,” “possessing,” and / or “having” are used in this specification, they specifically indicate the presence of the stated features, integers, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0023] As used herein, the terms "communication" and "to communicate" refer to at least one of the following: receiving, receiving, transmitting, conveying, and / or providing information (or information represented by, for example, data, signals, messages, instructions, and / or commands). For a unit (e.g., an apparatus, system, component of an apparatus or system, and / or combinations thereof) that wants to communicate with another unit, this means that the unit is able to receive information directly or indirectly from the other unit and / or send (e.g., transmit) information to the other unit. This can refer to a direct or indirect connection that is essentially wired and / or wireless. Furthermore, two units can communicate with each other even if the transmitted information can be modified, processed, relayed, and / or routed between the first and second units. For example, the first unit can communicate with the second unit even if it passively receives information and does not actively transmit information to the second unit. As another example, the first unit can communicate with the second unit if at least one intermediary unit (e.g., a third unit located between the first and second units) processes information received from the first unit and transmits the processed information to the second unit. In some embodiments, a message may refer to a network packet that includes data (e.g., a data packet, etc.).

[0024] As used herein, depending on the context, the term "if" may optionally be interpreted as "when," "in," "in response to being determined," and / or "in response to being detected," etc. Similarly, depending on the context, the phrases "if determined" or "if [the stated condition or event] is detected" may optionally be interpreted as "in response to being determined," "in response to being determined," "or" "in response to being detected," and / or "in response to being detected," etc. Furthermore, as used herein, the terms "have," "possess," or "own," etc., are intended to be open-ended terms. Additionally, unless explicitly stated otherwise, the phrase "based on" is intended to mean "at least partially based on."

[0025] Reference will now be made in detail to embodiments, examples of which are illustrated in the accompanying drawings. Numerous specific details are set forth in the following detailed description in order to provide a thorough understanding of the various embodiments described. However, it will be apparent to those skilled in the art that the various embodiments described can be practiced without these specific details. In other instances, well-known methods, processes, components, circuits, and networks have not been described in detail so as not to unnecessarily obscure aspects of the embodiments.

[0026] General Overview

[0027] In some aspects and / or embodiments, the systems, methods, and computer program products described herein include and / or implement graph exploration for rulebook trajectory generation. In the example, a vehicle (such as an autonomous vehicle) navigates in an environment based on a trajectory. This technique enables the use of graph exploration to determine the optimal trajectory in a given scenario (e.g., a predetermined environment with a fixed set of trajectories). For a set of poses, this technique iteratively identifies the next trajectory from a corresponding set of alternative trajectories. The next trajectory is selected from a set of rules that violate a hierarchy, with the lowest priority of the behavior rule violated, until a target pose (e.g., a destination) is reached, where the lowest priority of the behavior rule is lower than the priority of the behavior rules associated with other trajectories in the corresponding set of next alternative trajectories. For ease of description, several rules with different hierarchical priorities are described herein. However, this technique is not limited to the specific rules, rule priorities, and rule hierarchies described herein. The specific rules are for illustrative purposes and should not be considered limiting.

[0028] By implementing the systems, methods, and computer program products described herein, these techniques enable graphical exploration for rulebook trajectory generation, which determines the optimal trajectory in complex, dynamic scenes. Some advantages of these techniques include determining the optimal trajectory in difficult-to-navigate scenes. This technique generates the optimal trajectory even without a baseline trajectory. As a result, the optimal trajectory is identified from a larger, more robust set of trajectories, and this optimal trajectory is unconstrained (e.g., not constrained by baseline trajectories). Furthermore, this technique reduces the computational resources required for graphically-based optimal trajectory determination.

[0029] Now for reference Figure 1 Example environment 100 is illustrated, in which vehicles including autonomous systems and vehicles not including autonomous systems operate. As illustrated, environment 100 includes vehicles 102a-102n, objects 104a-104n, routes 106a-106n, area 108, vehicle-to-infrastructure (V2I) device 110, network 112, remote autonomous vehicle (AV) system 114, queue management system 116, and V2I system 118. Vehicles 102a-102n, vehicle-to-infrastructure (V2I) device 110, network 112, autonomous vehicle (AV) system 114, queue management system 116, and V2I system 118 are interconnected via wired connections, wireless connections, or a combination of wired and wireless connections (e.g., establishing connections for communication, etc.). In some embodiments, objects 104a-104n are interconnected with at least one of vehicles 102a-102n, vehicle-to-infrastructure (V2I) devices 110, network 112, autonomous vehicle (AV) system 114, queue management system 116, and V2I system 118 via wired connection, wireless connection, or a combination of wired and wireless connection.

[0030] Vehicles 102a-102n (specifically referred to as vehicle 102 and collectively as vehicle 102) include at least one device configured to transport goods and / or people. In some embodiments, vehicle 102 is configured to communicate with V2I device 110, remote AV system 114, queue management system 116 and / or V2I system 118 via network 112. In some embodiments, vehicle 102 includes cars, buses, trucks and / or trains, etc. In some embodiments, vehicle 102 is associated with vehicle 200 described herein (see Figure 2The vehicles 102 are the same as or similar to autonomous vehicles 202. In some embodiments, vehicles 200 in a group of vehicles 200 are associated with an autonomous queue manager. In some embodiments, as described herein, vehicles 102 travel along corresponding routes 106a-106n (each individually referred to as route 106 and collectively as route 106). In some embodiments, one or more vehicles 102 include an autonomous system (e.g., an autonomous system that is the same as or similar to autonomous system 202).

[0031] Objects 104a-104n (each individually referred to as object 104 and collectively as object 104) include, for example, at least one vehicle, at least one pedestrian, at least one cyclist, and / or at least one structure (e.g., a building, a sign, a fire hydrant, etc.). Each object 104 (e.g., located at a fixed location and for a period of time) is either stationary or (e.g., having a speed and associated with at least one trajectory) moving. In some embodiments, object 104 is associated with a corresponding location in area 108.

[0032] Routes 106a-106n (each individually referred to as Route 106 and collectively as Route 106) are each associated with (e.g., defining) a series of actions (also referred to as trajectories) along which the connecting AV can navigate. Each Route 106 begins with an initial state (e.g., a state corresponding to a first spatiotemporal location and / or speed, etc.) and ends with a final target state (e.g., a state corresponding to a second spatiotemporal location different from the first spatiotemporal location) or a target area (e.g., a subspace of an acceptable state (e.g., a termination state)). In some embodiments, a first state includes a location where one or more individuals will board the AV, and a second state or area includes a location where one or more individuals boarding the AV will disembark. In some embodiments, Route 106 includes multiple acceptable state sequences (e.g., multiple spatiotemporal location sequences) associated with multiple trajectories (e.g., defining multiple trajectories). In the example, Route 106 includes only high-level actions or imprecise state locations, such as a series of connecting roads indicating a change of direction at a roadway intersection. Additionally or alternatively, route 106 may include more precise actions or states, such as, for example, specific target lanes or precise locations within a lane area and target rates at those locations. In the example, route 106 includes multiple precise state sequences along at least one high-level action with a finite look-ahead horizon leading to an intermediate target, wherein the cumulative combination of successive iterations of the finite horizon state sequences corresponds to multiple trajectories that collectively form a high-level route terminating at a final target state or region.

[0033] Region 108 includes a physical area (e.g., a geographic region) that the vehicle 102 can navigate. In the example, region 108 includes at least one state (e.g., a country, a province, a single state among multiple states included in a country, etc.), at least a portion of a state, at least one city, at least a portion of a city, etc. In some embodiments, region 108 includes at least one named arterial road (referred to herein as a "road"), such as a highway, interstate highway, park road, city street, etc. Additionally or alternatively, in some examples, region 108 includes at least one unnamed road, such as a driving lane, a section of a parking lot, a section of vacant land and / or undeveloped area, dirt road, etc. In some embodiments, a road includes at least one lane (e.g., a portion of the road that the vehicle 102 can traverse). In the example, a road includes at least one lane associated with at least one lane marking (e.g., identified based on at least one lane marking).

[0034] The Vehicle-to-Infrastructure (V2I) device 110 (sometimes referred to as a Vehicle-to-Everything (V2X) device) includes at least one device configured to communicate with vehicle 102 and / or V2I system 118. In some embodiments, V2I device 110 is configured to communicate with vehicle 102, remote AV system 114, queue management system 116, and / or V2I system 118 via network 112. In some embodiments, V2I device 110 includes radio frequency identification (RFID) devices, signs, cameras (e.g., two-dimensional (2D) and / or three-dimensional (3D) cameras), lane markings, streetlights, parking meters, etc. In some embodiments, V2I device 110 is configured to communicate directly with vehicle 102. Additionally or alternatively, in some embodiments, V2I device 110 is configured to communicate with vehicle 102, remote AV system 114, and / or queue management system 116 via V2I system 118. In some embodiments, V2I device 110 is configured to communicate with V2I system 118 via network 112.

[0035] Network 112 includes one or more wired and / or wireless networks. In the example, network 112 includes cellular networks (e.g., Long Term Evolution (LTE) networks, third-generation (3G) networks, fourth-generation (4G) networks, fifth-generation (5G) networks, Code Division Multiple Access (CDMA) networks, etc.), Public Land Mobile Networks (PLMNs), Local Area Networks (LANs), Wide Area Networks (WANs), Metropolitan Area Networks (MANs), telephone networks (e.g., Public Switched Telephone Networks (PSTN)), private networks, self-organizing networks, intranets, the Internet, fiber-based networks, cloud computing networks, etc., and / or combinations of some or all of these networks.

[0036] The remote AV system 114 includes at least one device configured to communicate with the vehicle 102, V2I device 110, network 112, queue management system 116, and / or V2I system 118 via network 112. In examples, the remote AV system 114 includes a server, server group, and / or other similar devices. In some embodiments, the remote AV system 114 is located in the same location as the queue management system 116. In some embodiments, the remote AV system 114 participates in the installation of some or all of the components of the vehicle, including autonomous systems, autonomous vehicle computing, and / or software implemented by autonomous vehicle computing. In some embodiments, the remote AV system 114 maintains (e.g., updates and / or replaces) these components and / or software during the lifespan of the vehicle.

[0037] The queue management system 116 includes at least one device configured to communicate with vehicle 102, V2I device 110, remote AV system 114, and / or V2I system 118. In examples, the queue management system 116 includes servers, server groups, and / or other similar devices. In some embodiments, the queue management system 116 is associated with a ride-sharing company (e.g., an organization for controlling the operation of multiple vehicles (e.g., vehicles including autonomous systems and / or vehicles not including autonomous systems)).

[0038] In some embodiments, the V2I system 118 includes at least one device configured to communicate with the vehicle 102, the V2I device 110, the remote AV system 114, and / or the queue management system 116 via a network 112. In some examples, the V2I system 118 is configured to communicate with the V2I device 110 via a connection different from the network 112. In some embodiments, the V2I system 118 includes a server, a server group, and / or other similar devices. In some embodiments, the V2I system 118 is associated with a municipality or private entity (e.g., a private entity maintaining the V2I device 110).

[0039] supply Figure 1 The number and arrangement of the elements are shown as examples. (and) Figure 1 Compared to the illustrated elements, there may be additional elements, fewer elements, different elements, and / or elements arranged differently. Additionally or alternatively, at least one element of environment 100 may be described as being composed of… Figure 1 One or more functions performed by at least one different element of environment 100. Additionally or alternatively, at least one group of elements of environment 100 may perform one or more functions described as performed by at least one different group of elements of environment 100.

[0040] Now for reference Figure 2 The vehicle 200 includes an autonomous system 202, a powertrain control system 204, a steering control system 206, and a braking system 208. In some embodiments, the vehicle 200 and the vehicle 102 (see...) Figure 1 The vehicle 200 is similar to or the same as the vehicle in question. In some embodiments, the vehicle 200 has autonomous capabilities (e.g., implementing at least one function, feature, and / or device that enables the vehicle 200 to operate partially or fully without human intervention, including but not limited to fully autonomous vehicles (e.g., vehicles that abandon human intervention) and / or highly autonomous vehicles (e.g., vehicles that abandon human intervention in certain situations)). For a detailed description of fully autonomous and highly autonomous vehicles, refer to SAE International's standard J3016: Taxonomy and Definitions for Terms Related to On-Road Motor Vehicle Automated Driving Systems, the entire contents of which are incorporated herein by reference. In some embodiments, the vehicle 200 is associated with an autonomous queue manager and / or a ride-sharing company.

[0041] Autonomous system 202 includes a sensor suite comprising one or more devices such as camera 202a, LiDAR sensor 202b, radar sensor 202c, and microphone 202d. In some embodiments, autonomous system 202 may include more or fewer devices and / or different devices (e.g., ultrasonic sensors, inertial sensors, GPS receivers (discussed below), and / or odometer sensors for generating data associated with an indication of the distance traveled by vehicle 200). In some embodiments, autonomous system 202 uses one or more devices included in autonomous system 202 to generate data associated with environment 100 as described herein. The data generated by one or more devices of autonomous system 202 may be used by one or more systems as described herein to observe the environment in which vehicle 200 is located (e.g., environment 100). In some embodiments, autonomous system 202 includes communication device 202e, autonomous vehicle computing 202f, and safety controller 202g.

[0042] Camera 202a includes components configured to communicate with communication device 202e, autonomous vehicle computing 202f, and / or safety controller 202g via a bus (e.g., with...). Figure 3At least one means of communicating with the same or similar bus as bus 302. Camera 202a includes at least one camera (e.g., a digital camera using a light sensor such as a charge-coupled device (CCD), a thermal camera, an infrared (IR) camera, and / or an event camera, etc.) for capturing images of physical objects (e.g., cars, buses, curbs, and / or people, etc.). In some embodiments, camera 202a generates camera data as output. In some examples, camera 202a generates camera data including image data associated with an image. In this example, the image data may specify at least one parameter corresponding to the image (e.g., image characteristics such as exposure, brightness, etc., and / or image timestamp, etc.). In such examples, the image may be in a format (e.g., RAW, JPEG, and / or PNG, etc.). In some embodiments, camera 202a includes multiple independent cameras configured (e.g., positioned on) a vehicle to capture images for stereoscopic imaging (stereoscopic vision). In some examples, camera 202a includes generating image data and transmitting the image data to an autonomous vehicle computing 202f and / or a queue management system (e.g., with...). Figure 1 The queue management system 116 (same as or similar to a queue management system) has multiple cameras. In such an example, the autonomous vehicle calculation 202f determines the depth of one or more objects in the fields of view of at least two of the multiple cameras based on image data from at least two cameras. In some embodiments, camera 202a is configured to capture images of objects within a distance relative to camera 202a (e.g., up to 100 meters and / or up to 1 kilometer, etc.). Therefore, camera 202a includes features such as sensors and lenses optimized for sensing objects at one or more distances relative to camera 202a.

[0043] In embodiments, camera 202a includes at least one camera configured to capture one or more images associated with one or more traffic lights, street signs, and / or other physical objects providing visual navigation information. In some embodiments, camera 202a generates traffic light data associated with one or more images. In some examples, camera 202a generates TLD data associated with one or more images, including formats such as RAW, JPEG, and / or PNG. In some embodiments, camera 202a, which generates TLD data, differs from other systems containing cameras described herein in that camera 202a may include one or more cameras with a wide field of view (e.g., a wide-angle lens, a fisheye lens, and / or a lens with an angle of view of about 120 degrees or greater) to generate images associated with as many physical objects as possible.

[0044] The laser detection and ranging (LiDAR) sensor 202b includes components configured to communicate with a communication device 202e, an autonomous vehicle computing unit 202f, and / or a safety controller 202g via a bus (e.g., with...). Figure 3 At least one device that communicates with the same or similar bus (bus 302). The LiDAR sensor 202b includes a system configured to emit light from a emitter (e.g., a laser emitter). The light emitted by the LiDAR sensor 202b includes light outside the visible spectrum (e.g., infrared light, etc.). In some embodiments, during operation, the light emitted by the LiDAR sensor 202b encounters a physical object (e.g., a vehicle) and is reflected back to the LiDAR sensor 202b. In some embodiments, the light emitted by the LiDAR sensor 202b does not penetrate the physical object it encounters. The LiDAR sensor 202b also includes at least one photosensor that detects the light after it has encountered a physical object. In some embodiments, at least one data processing system associated with the LiDAR sensor 202b generates an image (e.g., point cloud and / or combined point cloud, etc.) representing objects included in the field of view of the LiDAR sensor 202b. In some examples, at least one data processing system associated with the LiDAR sensor 202b generates an image representing the boundaries of a physical object and / or the surface of the physical object (e.g., the topology of the surface). In such examples, the image is used to determine the boundaries of the physical object within the field of view of the LiDAR sensor 202b.

[0045] The radio detection and ranging (radar) sensor 202c includes components configured to communicate with the communication device 202e, the autonomous vehicle computing 202f, and / or the safety controller 202g via a bus (e.g., with...). Figure 3 At least one device that communicates with the same or similar bus (bus 302). The radar sensor 202c includes a system configured to emit (pulsed or continuous) radio waves. The radio waves emitted by the radar sensor 202c include radio waves within a predetermined spectrum. In some embodiments, during operation, the radio waves emitted by the radar sensor 202c encounter a physical object and are reflected back to the radar sensor 202c. In some embodiments, the radio waves emitted by the radar sensor 202c are not reflected by some objects. In some embodiments, at least one data processing system associated with the radar sensor 202c generates a signal representing objects included in the field of view of the radar sensor 202c. For example, at least one data processing system associated with the radar sensor 202c generates an image representing the boundaries of physical objects and / or the surfaces of physical objects (e.g., surface topology). In some examples, this image is used to determine the boundaries of physical objects in the field of view of the radar sensor 202c.

[0046] Microphone 202d includes components configured to communicate with communication device 202e, autonomous vehicle computing 202f, and / or safety controller 202g via a bus (e.g., with...). Figure 3 At least one device that communicates with the same or similar bus as bus 302. Microphone 202d includes one or more microphones (e.g., array microphones and / or external microphones, etc.) that capture audio signals and generate data associated with (e.g., representing) the audio signals. In some examples, microphone 202d includes transducer devices and / or similar devices. In some embodiments, one or more systems described herein can receive data generated by microphone 202d and determine the position (e.g., distance, etc.) of an object relative to vehicle 200 based on the audio signal associated with the data.

[0047] The communication device 202e includes at least one device configured to communicate with a camera 202a, a LiDAR sensor 202b, a radar sensor 202c, a microphone 202d, an autonomous vehicle computing system 202f, a safety controller 202g, and / or a drive-by-wire (DBW) system 202h. For example, the communication device 202e may include communication with… Figure 3 The communication device 202e is the same as or similar to the communication interface 314. In some embodiments, the communication device 202e includes a vehicle-to-vehicle (V2V) communication device (e.g., a device for enabling wireless communication of data between vehicles).

[0048] The autonomous vehicle computing 202f includes at least one device configured to communicate with a camera 202a, a LiDAR sensor 202b, a radar sensor 202c, a microphone 202d, a communication device 202e, a security controller 202g, and / or a DBW system 202h. In some examples, the autonomous vehicle computing 202f includes devices such as client devices, mobile devices (e.g., cellular phones and / or tablets) and / or servers (e.g., computing devices including one or more central processing units and / or graphics processing units). In some embodiments, the autonomous vehicle computing 202f is the same as or similar to the autonomous vehicle computing 400 described herein. Additionally or alternatively, in some embodiments, the autonomous vehicle computing 202f is configured to communicate with an autonomous vehicle system (e.g., with...). Figure 1 Remote AV systems 114 are the same as or similar to autonomous vehicle systems), queue management systems (e.g., with...). Figure 1 The queue management system 116 is the same as or similar to the queue management system 116), and V2I devices (e.g., with Figure 1 V2I devices (same as or similar to V2I devices 110) and / or V2I systems (e.g., with V2I devices 110) Figure 1 The V2I system 118 communicates with the same or similar V2I system.

[0049] The safety controller 202g includes at least one device configured to communicate with a camera 202a, a LiDAR sensor 202b, a radar sensor 202c, a microphone 202d, a communication device 202e, an autonomous vehicle computing system 202f, and / or a DBW system 202h. In some examples, the safety controller 202g includes one or more controllers (electrical controllers and / or electromechanical controllers, etc.) configured to generate and / or transmit control signals to operate the vehicle 200 (e.g., powertrain control system 204, steering control system 206, and / or braking system 208, etc.). In some embodiments, the safety controller 202g is configured to generate control signals that take precedence over (e.g., override) the control signals generated and / or transmitted by the autonomous vehicle computing system 202f.

[0050] The DBW system 202h includes at least one device configured to communicate with the communication device 202e and / or the autonomous vehicle computing 202f. In some examples, the DBW system 202h includes one or more controllers (e.g., electrical controllers and / or electromechanical controllers, etc.) configured to generate and / or transmit control signals to operate the vehicle 200. Additionally or alternatively, one or more controllers of the DBW system 202h are configured to generate and / or transmit control signals to operate at least one different device (e.g., turn signals, headlights, door locks, and / or windshield wipers, etc.) of the vehicle 200.

[0051] The powertrain control system 204 includes at least one device configured to communicate with the DBW system 202h. In some examples, the powertrain control system 204 includes at least one controller and / or actuator, etc. In some embodiments, the powertrain control system 204 receives control signals from the DBW system 202h, and the powertrain control system 204 causes the vehicle 200 to start moving forward, stop moving forward, start moving backward, stop moving backward, accelerate in a certain direction, decelerate in a certain direction, make a left turn and / or make a right turn, etc. In examples, the powertrain control system 204 increases, keeps the same, or decreases the energy (e.g., fuel and / or electricity, etc.) supplied to the motor of the vehicle, thereby causing at least one wheel of the vehicle 200 to rotate or not rotate.

[0052] The steering control system 206 includes at least one device configured to rotate one or more wheels of the vehicle 200. In some examples, the steering control system 206 includes at least one controller and / or actuator, etc. In some embodiments, the steering control system 206 causes the two front wheels and / or the two rear wheels of the vehicle 200 to turn left or right, thereby causing the vehicle 200 to turn left or right.

[0053] The braking system 208 includes at least one device configured to actuate one or more brakes to decelerate and / or keep the vehicle 200 stationary. In some examples, the braking system 208 includes at least one controller and / or actuator configured to close one or more calipers associated with one or more wheels of the vehicle 200 on the respective rotor of the vehicle 200. Additionally or alternatively, in some examples, the braking system 208 includes an automatic emergency braking (AEB) system and / or a regenerative braking system, etc.

[0054] In some embodiments, the vehicle 200 includes at least one platform sensor (not explicitly illustrated) for measuring or inferring the nature of the state or conditions of the vehicle 200. In some examples, the vehicle 200 includes platform sensors such as a Global Positioning System (GPS) receiver, an Inertial Measurement Unit (IMU), a wheel rate sensor, a wheel brake pressure sensor, a wheel torque sensor, an engine torque sensor, and / or a steering angle sensor.

[0055] Now for reference Figure 3 A schematic diagram of device 300 is illustrated. As illustrated, device 300 includes a processor 304, a memory 306, a storage component 308, an input interface 310, an output interface 312, a communication interface 314, and a bus 302. In some embodiments, device 300 corresponds to: at least one device of vehicle 102 (e.g., at least one device of system of vehicle 102); and / or one or more devices of network 112 (e.g., one or more devices of system of network 112). In some embodiments, one or more devices of vehicle 102 (e.g., one or more devices of system of vehicle 102), and / or one or more devices of network 112 (e.g., one or more devices of system of network 112) include at least one device 300 and / or at least one component of device 300. Figure 3 As shown, the device 300 includes a bus 302, a processor 304, a memory 306, a storage component 308, an input interface 310, an output interface 312, and a communication interface 314.

[0056] Bus 302 includes components for communication between the components of the licensed device 300. In some embodiments, processor 304 is implemented in hardware, software, or a combination of hardware and software. In some examples, processor 304 includes a processor (e.g., a central processing unit (CPU), graphics processing unit (GPU), and / or accelerated processing unit (APU), a microphone, a digital signal processor (DSP), and / or any processing component that can be programmed to perform at least one function (e.g., a field-programmable gate array (FPGA) and / or application-specific integrated circuit (ASIC), etc.). Memory 306 includes random access memory (RAM), read-only memory (ROM), and / or another type of dynamic and / or static storage device (e.g., flash memory, magnetic memory, and / or optical memory, etc.) that stores data and / or instructions for use by processor 304.

[0057] Storage component 308 stores data and / or software related to the operation and use of device 300. In some examples, storage component 308 includes hard disks (e.g., magnetic disks, optical disks, magneto-optical disks, and / or solid-state disks), compact discs (CDs), digital versatile discs (DVDs), floppy disks, cassette tapes, magnetic tapes, CD-ROMs, RAM, PROMs, EPROMs, FLASH-EPROMs, NV-RAMs, and / or other types of computer-readable media, and corresponding drives.

[0058] Input interface 310 includes components that enable the device 300 to receive information, such as via user input (e.g., a touchscreen display, keyboard, keypad, mouse, buttons, switches, microphone, and / or camera). Additionally or alternatively, in some embodiments, input interface 310 includes sensors for sensing information (e.g., a Global Positioning System (GPS) receiver, accelerometer, gyroscope, and / or actuator). Output interface 312 includes components for providing output information from device 300 (e.g., a display, speaker, and / or one or more light-emitting diodes (LEDs)).

[0059] In some embodiments, the communication interface 314 includes transceiver-like components (e.g., a transceiver and / or separate receivers and transmitters) that enable the licensing device 300 to communicate with other devices via a wired connection, a wireless connection, or a combination of wired and wireless connections. In some examples, the communication interface 314 enables the licensing device 300 to receive information from and / or provide information to another device. In some examples, the communication interface 314 includes an Ethernet interface, an optical interface, a coaxial interface, an infrared interface, a radio frequency (RF) interface, a universal serial bus (USB) interface, a Wi-Fi interface, etc. Interfaces and / or cellular network interfaces, etc.

[0060] In some embodiments, device 300 performs one or more of the processes described herein. Device 300 performs these processes based on software instructions stored in a computer-readable medium, such as memory 306 and / or storage component 308, executed by processor 304. Computer-readable medium (e.g., non-transitory computer-readable medium) is defined herein as a non-transitory memory device. A non-transitory memory device includes storage space located within a single physical storage device or storage space distributed across multiple physical storage devices.

[0061] In some embodiments, software instructions are read from another computer-readable medium or from another device via communication interface 314 into memory 306 and / or storage component 308. When executed, the software instructions stored in memory 306 and / or storage component 308 cause processor 304 to perform one or more processes described herein. Additionally or alternatively, hard-wired circuitry may be used in place of or in combination with software instructions to perform one or more processes described herein. Therefore, unless explicitly stated otherwise, the embodiments described herein are not limited to any particular combination of hardware circuitry and software.

[0062] The memory 306 and / or storage component 308 include a data storage unit or at least one data structure (e.g., a database). The device 300 is capable of receiving information from the data storage unit or at least one data structure in the memory 306 or storage component 308, storing the information in the data storage unit or at least one data structure, communicating information to the data storage unit or at least one data structure, or searching for information stored in the data storage unit or at least one data structure. In some examples, the information includes network data, input data, output data, or any combination thereof.

[0063] In some embodiments, device 300 is configured to execute software instructions stored in the memory of memory 306 and / or another device (e.g., another device identical or similar to device 300). As used herein, the term "module" refers to at least one instruction stored in the memory of memory 306 and / or the other device, which, when executed by the processor of processor 304 and / or the processor of another device (e.g., another device identical or similar to device 300), causes device 300 (e.g., at least one component of device 300) to perform one or more processes as described herein. In some embodiments, modules are implemented in software, firmware, and / or hardware, etc.

[0064] supply Figure 3 The number and arrangement of components are illustrated as examples. In some embodiments, with Figure 3Compared to the illustrated components, device 300 may include additional components, fewer components, different components, or components arranged differently. Additionally or alternatively, a group of components of device 300 (e.g., one or more components) may perform one or more functions described as being performed by another component or another group of components of device 300.

[0065] Now for reference Figure 4 The diagram illustrates an example block diagram of an autonomous vehicle computing 400 (sometimes referred to as an "AV stack"). As illustrated, the autonomous vehicle computing 400 includes a perception system 402 (sometimes referred to as a perception module), a planning system 404 (sometimes referred to as a planning module), a positioning system 406 (sometimes referred to as a positioning module), a control system 408 (sometimes referred to as a control module), and a database 410. In some embodiments, the perception system 402, planning system 404, positioning system 406, control system 408, and database 410 are included in and / or implemented in the vehicle's automatic navigation system (e.g., the autonomous vehicle computing 202f of vehicle 200). Additionally or alternatively, in some embodiments, the perception system 402, planning system 404, positioning system 406, control system 408, and database 410 are included in one or more separate systems (e.g., one or more systems that are the same as or similar to the autonomous vehicle computing 400, etc.). In some examples, the perception system 402, planning system 404, positioning system 406, control system 408, and database 41 are included in one or more independent systems located within the vehicle and / or at least one remote system as described herein. In some embodiments, any and / or all of the systems included in the autonomous vehicle computing 400 are implemented in software (e.g., software instructions stored in memory), computer hardware (e.g., via microprocessors, microcontrollers, application-specific integrated circuits (ASICs), and / or field-programmable gate arrays (FPGAs), etc.), or a combination of computer software and computer hardware. It will also be understood that in some embodiments, the autonomous vehicle computing 400 is configured to communicate with remote systems (e.g., autonomous vehicle systems identical or similar to remote AV system 114, queue management systems identical or similar to queue management systems 116, and / or V2I systems identical or similar to V2I system 118, etc.).

[0066] In some embodiments, the perception system 402 receives data associated with at least one physical object in the environment (e.g., data used by the perception system 402 to detect at least one physical object) and classifies the at least one physical object. In some examples, the perception system 402 receives image data captured by at least one camera (e.g., camera 202a) that is associated with one or more physical objects within the field of view of the at least one camera (e.g., representing the one or more physical objects). In such examples, the perception system 402 classifies at least one physical object based on one or more groups of physical objects (e.g., bicycles, vehicles, traffic signs, and / or pedestrians, etc.). In some embodiments, based on the classification of physical objects by the perception system 402, the perception system 402 transmits data associated with the classification of the physical objects to the planning system 404.

[0067] In some embodiments, the planning system 404 receives data associated with a destination and generates data associated with at least one route (e.g., route 106) along which a vehicle (e.g., vehicle 102) can travel toward the destination. In some embodiments, the planning system 404 periodically or continuously receives data from the sensing system 402 (e.g., the data associated with the classification of physical objects described above), and the planning system 404 updates at least one trajectory or generates at least one different trajectory based on the data generated by the sensing system 402. In some embodiments, the planning system 404 receives data associated with the updated location of the vehicle (e.g., vehicle 102) from the positioning system 406, and the planning system 404 updates at least one trajectory or generates at least one different trajectory based on the data generated by the positioning system 406.

[0068] In some embodiments, positioning system 406 receives data associated with (e.g., representing) a location of a vehicle (e.g., vehicle 102) in an area. In some examples, positioning system 406 receives LiDAR data associated with at least one point cloud generated by at least one LiDAR sensor (e.g., LiDAR sensor 202b). In some examples, positioning system 406 receives data associated with at least one point cloud from multiple LiDAR sensors, and positioning system 406 generates a composite point cloud based on the individual point clouds. In these examples, positioning system 406 compares the at least one point cloud or composite point cloud with a two-dimensional (2D) and / or three-dimensional (3D) map of the area stored in database 410. Then, based on the comparison of the at least one point cloud or composite point cloud with the map, positioning system 406 determines the location of the vehicle in the area. In some embodiments, the map includes a composite point cloud of the area generated prior to navigation of the vehicle. In some embodiments, the map includes, but is not limited to, a high-precision map of the geometry of the roadway, a map describing the connectivity of the road network, a map describing the physical properties of the roadway (such as traffic speed, traffic flow, the number of vehicle and bicycle lanes, lane width, lane traffic direction, or the type and location of lane markings, or combinations thereof), and a map describing the spatial locations of road features (such as pedestrian crossings, traffic signs, or various types of other traffic lights). In some embodiments, the map is generated in real time based on data received by the sensing system.

[0069] In another example, positioning system 406 receives Global Navigation Satellite System (GNSS) data generated by a Global Positioning System (GPS) receiver. In some examples, positioning system 406 receives GNSS data associated with the location of a vehicle in an area, and positioning system 406 determines the latitude and longitude of the vehicle in the area. In such examples, positioning system 406 determines the location of the vehicle in the area based on the latitude and longitude of the vehicle. In some embodiments, positioning system 406 generates data associated with the location of the vehicle. In some examples, based on the location of the vehicle determined by positioning system 406, positioning system 406 generates data associated with the location of the vehicle. In such examples, the data associated with the location of the vehicle includes data associated with one or more semantic properties corresponding to the location of the vehicle.

[0070] In some embodiments, the control system 408 receives data associated with at least one trajectory from the planning system 404, and the control system 408 controls the operation of the vehicle. In some examples, the control system 408 receives data associated with at least one trajectory from the planning system 404, and the control system 408 controls the operation of the vehicle by generating and transmitting control signals to operate the powertrain control system (e.g., DBW system 202h and / or powertrain control system 204, etc.), the steering control system (e.g., steering control system 206), and / or the braking system (e.g., braking system 208). In an example, where the trajectory includes a left turn, the control system 408 transmits control signals to cause the steering control system 206 to adjust the steering angle of the vehicle 200, thereby causing the vehicle 200 to turn left. Additionally or alternatively, the control system 408 generates and transmits control signals to change the state of other devices of the vehicle 200 (e.g., headlights, turn signals, door locks, and / or windshield wipers, etc.).

[0071] In some embodiments, the perception system 402, planning system 404, positioning system 406, and / or control system 408 implement at least one machine learning model (e.g., at least one multilayer perceptron (MLP), at least one convolutional neural network (CNN), at least one recurrent neural network (RNN), at least one autoencoder, and / or at least one transformer, etc.). In some examples, the perception system 402, planning system 404, positioning system 406, and / or control system 408 implement at least one machine learning model individually or in combination with one or more of the aforementioned systems. In some examples, the perception system 402, planning system 404, positioning system 406, and / or control system 408 implement at least one machine learning model as part of a pipeline (e.g., a pipeline for identifying one or more objects located in the environment, etc.).

[0072] Database 410 stores data transmitted to, received from, and / or updated by the sensing system 402, planning system 404, positioning system 406, and / or control system 408. In some examples, database 410 includes storage components for storing operation-related data and / or software, and for computing 400 using autonomous vehicles (e.g., with...). Figure 3(The storage component 308 is the same as or similar to the storage component 308). In some embodiments, database 410 stores data associated with 2D and / or 3D maps of at least one area. In some examples, database 410 stores data associated with 2D and / or 3D maps of a part of a city, multiple parts of multiple cities, multiple cities, counties, states, and / or countries (e.g., countries). In such examples, a vehicle (e.g., the same as or similar to vehicle 102 and / or vehicle 200) can drive along one or more drivable areas (e.g., single-lane roads, multi-lane roads, highways, remote roads, and / or off-road roads, etc.) and causes at least one LiDAR sensor (e.g., the same as or similar to LiDAR sensor 202b) to generate data associated with images representing objects included in the field of view of the at least one LiDAR sensor.

[0073] In some embodiments, database 410 may be implemented across multiple devices. In some examples, database 410 includes a vehicle (e.g., a vehicle identical or similar to vehicle 102 and / or vehicle 200), an autonomous vehicle system (e.g., an autonomous vehicle system identical or similar to remote AV system 114), and a queue management system (e.g., with...). Figure 1 Queue management system 116 (same as or similar to queue management system) and / or V2I system (e.g., with Figure 1 Among the V2I systems (118 similar to or similar V2I systems), etc.

[0074] Now for reference Figure 5 A diagram illustrates an implementation 500 of a graphical exploration process for generating rulebook trajectories. In some embodiments, implementation 500 includes a planning system 504a. In some embodiments, planning system 504a and... Figure 4 The planning system 504a is the same as or similar to the planning system 404. Typically, the output of planning system 504a is a route from a starting point (e.g., source location or initial location) to an end point (e.g., destination or final location). Therefore, in Figure 5In the example, the route is transmitted to the control system 504b at reference numeral 516. During vehicle operation, the control system operates the vehicle to navigate the route. In an embodiment, the route and other AV computational data are stored for post-hoc evaluation of the route selected by the AV to navigate from the starting point to the destination. Typically, a route is defined by one or more road segments. For example, a road segment is the distance to be traveled over at least a portion of a street, road, highway, driveway, or other physical area suitable for vehicle travel. In some examples, for instance, if the AV is an off-road capable vehicle such as a four-wheel drive (4WD) or all-wheel drive (AWD) car, SUV, pickup truck, etc., the route includes “off-road” segments such as unpaved paths or open fields.

[0075] In addition to the route, the planning system 504a also outputs lane-level route planning data. This lane-level route planning data is used to traverse road segments based on conditions at specific times. In embodiments, lane-level route planning data is stored for post-event evaluation using graphical exploration as described herein. During operation, the lane-level route planning data is used to traverse road segments based on conditions at specific times. For example, if the route includes a multi-lane highway, the lane-level route planning data includes trajectory planning data, which the AV can use to select a lane from the multiple lanes based on factors such as whether an exit is approaching, whether another vehicle is present in one or more of the multiple lanes, or other factors that change over a few minutes or less during the vehicle's movement along the route. Similarly, in some implementations, the lane-level route planning data includes rate constraints specific to a particular road segment. For example, if the segment includes pedestrians or unexpected traffic, the rate constraint can limit the AV to a slower speed than expected, such as a speed based on the segment's speed limit data.

[0076] Figure 6 An example scenario of AV602 operation using graphical exploration with behavior rule checking according to one or more embodiments is shown. AV 602 can be, for example, reference... Figure 1 For a more detailed illustration and description of vehicle 102, or refer to [reference needed] Figure 2 The vehicle 200 is shown and described in more detail. The AV 602 operates in environment 600, which may be a reference. Figure 1 The environment 100 is shown and described in more detail. Figure 6In the example scenario shown, AV 602 operates in lane 606 near intersection 610. Similarly, another vehicle 604 operates in lane 608 near intersection 610. As indicated by the arrows, the traffic flow in lane 606 is opposite to the traffic flow in lane 608. There is a double lane 612 separating lane 606 from lane 608. However, there is no physical road divider or median strip separating lane 606 from lane 608. According to commonly understood road rules, traffic rules in environment 600 prohibit vehicles from crossing the double lane 612 or exceeding a predetermined speed limit (e.g., 45 miles per hour).

[0077] AV 602 operates in lane 606, reaching its destination outside intersection 610. As shown, pedestrian 614 is located in lane 606, obstructing lane 606. Other objects such as accidents obstructing traffic lanes, vehicle malfunctions, construction sites, and cyclists may obstruct the AV's planned trajectory. In this embodiment, AV 602 uses perception system 402 to identify objects such as pedestrian 614. (Refer to...) Figure 4 The perception system 402 is shown and described in more detail. Typically, the perception system 402 classifies objects into types such as cars, roadblocks, and traffic cones. These objects are then provided to the planning system 404. (See reference...) Figure 4 The planning system 404 is shown and described in more detail.

[0078] AV 602 determines that lane 606 is blocked by pedestrian 614. In the example, AV 602 is based on Figure 2 The characteristics of the data points (e.g., sensor data) detected by sensor 202 are used to detect the boundary of pedestrian 614. In order to reach the destination, the planning system 404 of AV602 ( Figure 4 A trajectory 616 is generated. Based on trajectory 616, AV 602 is operated to cause AV 602 to violate traffic rules and cross double lanes 612 to maneuver around pedestrian 614, allowing AV 602 to reach its destination. Some trajectories 616 cause AV 602 to cross double lanes 612 and enter lane 608 in the path of vehicle 604. AV 602 uses a hierarchical set of operating rules to provide feedback on the driving performance of AV 602. The hierarchical set of rules is sometimes referred to as a stored behavioral model or rulebook. In some embodiments, feedback is provided in a pass-fail manner. The embodiments disclosed herein detect AV 602 (e.g., Figure 4The planning system 404 determines when it generates a trajectory 616 that violates behavioral rules, and identifies alternative trajectories that AV 602 could have generated that would violate lower-priority behavioral rules. The occurrence of such a detection indicates a failure of the motion planning process. This technique uses graphical exploration to heuristically determine the optimal trajectory 616 for navigating through pedestrians 614 in lane 606 and reaching a destination (e.g., an objective). In an embodiment, the optimal trajectory is the one that starts in a starting posture and violates the lowest-priority behavioral rule compared to other trajectories.

[0079] In an embodiment, at least one processor receives sensor data after an AV operation. The sensor data represents the scene encountered by the AV while navigating through the environment. Hierarchical rules are applied to the scene simulated by the AV stack to modify and improve AV development post-hoc (e.g., after an AV operation in which sensor data is captured). In an example, the offline framework is configured to develop a transparent and reproducible rule-based pass / fail evaluation of AV trajectories in a test scenario. For example, in the offline framework, a given trajectory output by the planning system 404 is rejected if a trajectory is found that results in fewer violations of the rule priority structure. The planning system is modified and improved at least in part based on the rejected trajectories and the data associated with them. In an embodiment, the technique receives a post-generated fixed set of trajectories from a given scene and determines the optimal trajectory to evaluate whether the AV has passed a predetermined test or failed. The technique uses the fixed set of trajectories to create a graph. In an embodiment, the graph is an edge-weighted graph, where weights are assigned to edges corresponding to trajectories based on rule violations. Each trajectory is associated with one or more costs, each cost corresponding to a rule violation. Reference Figure 7 It describes the determination of a fixed set of trajectories.

[0080] Figure 7 An example flowchart is shown for a process 700 that uses behavior rule checks to determine vehicle operations based on a fixed set of trajectories. In an embodiment, Figure 7 The processing is handled by Figure 2 AV 200, Figure 3 Device 300 Figure 4 The AV calculation is performed by 400 or any combination thereof. In an embodiment, at least one processor located remotely from the vehicle performs the calculation. Figure 7 The processing 700. Similarly, embodiments may include different and / or additional steps, or perform these steps in a different order.

[0081] At box 704, the trajectory identified as AV 602 (e.g., trajectory 616) is acceptable. (See reference) Figure 6Trajectory 616 and AV 602 are shown and described in more detail. In the example, the trajectory is determined to be acceptable based on the behavioral rules in the hierarchical operation rule set that violate AV 602. If no rules are violated, processing moves to step 708, and the planning system 404 and AV behavior pass the validation check. Reference Figure 4 The planning system 404 is shown and described in more detail.

[0082] At box 704, if a rule is violated, processing moves to box 712. The violated rule is represented as a first behavior rule with a first priority. The processing moves to box 716. At box 716, the processor determines whether there are alternative trajectories with fewer violations. For example, the processor generates multiple alternative trajectories for AV 602 based on sensor data associated with the scene. In embodiments, sensor data characterizes information associated with the AV, information associated with objects, information associated with the environment, or any combination thereof. The processor identifies whether there are second trajectories that violate only a second behavior rule of a hierarchical rule set, such that the second behavior rule has a second priority lower than the first priority. In the example, if there are no other trajectories that violate only a second behavior rule with a lower priority than the first priority, processing moves to box 720. Planning system 404 and AV behavior pass the validation check. At box 716, alternative trajectories with fewer violations exist, and planning system 404 and AV behavior do not pass the validation check.

[0083] In the example, an AV can operate according to multiple hierarchical behavioral rules. Each behavioral rule has a priority relative to other rules. For example, a rulebook might include the following rules in ascending order of priority: 1: Maintain the predetermined speed limit; 2: Stay within the lane; 3: Maintain the predetermined distance; 4: Reach the destination; 5: Avoid a collision. In the example, priority indicates the level of risk of violating the behavioral rule. Therefore, a rulebook is a formal framework that defines driving requirements enforced by traffic law or cultural expectations and their relative priorities. A rulebook is a pre-ordered set of rules with violation scores that reflect the hierarchical structure of rule priorities. Therefore, a rulebook supports the regulation and assessment of AV behavior in conflict scenarios.

[0084] Refer again Figure 6Consider a scenario where pedestrian 614 enters the lane in which AV 602 is traveling. A reasonable AV behavior would be to avoid a collision with pedestrian 614 and other vehicles 604 (e.g., satisfying Rule 5: Avoid Collision, the highest priority in the exemplary rulebook), even at the cost of violating lower priority rules, such as reducing speed below the minimum speed limit (e.g., violating Rule 1: Maintain Predetermined Speed ​​Limit) or deviating from the lane (e.g., violating Rule 2: Stay Within Lane). The rulebook is generated by post-hoc prioritization of actions the AV should take based on complete information associated with the scenario (e.g., knowing predetermined values ​​or states). In an embodiment, a violation rule includes operating the AV such that it exceeds a predetermined speed limit (e.g., 45 mph). For example, Rule 1: Maintain Predetermined Speed ​​Limit means the AV should not violate the speed limit of the lane it is traveling in. However, Rule (1) is a lower priority rule; therefore, the AV might violate Rule (1) to prevent (e.g., a collision with another vehicle) and act according to Rule 5: Avoid Collision. In an embodiment, a violation rule includes operating the AV such that it stops before reaching its destination. In the example, rule 2: Stay in the lane means that the AV should stay in its own lane. Rule (2) has a lower priority than rule 5: Avoid collision. Therefore, the AV only violates rule 2: Stay in the lane to satisfy rule 5: Avoid collision, rule 4: Reach the destination, and rule 3: Maintain spacing.

[0085] In an embodiment, violating the stored AV operation rules includes manipulating the AV such that the lateral spacing between the AV and nearby objects decreases below a threshold lateral distance. For example, rule 3: Maintain a predetermined spacing means that the AV should maintain a threshold lateral distance (e.g., half a car length or 1 meter) relative to any other object (e.g., pedestrian 614). However, in this example, rule 3: Maintaining a predetermined spacing has lower priority than rule 4: Reaching the destination. Therefore, as referenced... Figure 6 As shown and described in more detail, the AV may violate rule (3) to comply with the higher priority rule 4: reach the target and rule 5: avoid collision.

[0086] In this embodiment, the alternative trajectory set is generated based on human driving behavior. In this embodiment, the trajectories are derived from a set of safe drivers or trained on a trajectory set obtained from human drivers. Trajectories exist in multiple sets and are strung together to generate a graphical representation of the trajectories. In this embodiment, the trajectory set represents all possible trajectories that the AV can take relative to its initial posture. Therefore, a trajectory is those routes that are possible considering a predetermined rate or heading (e.g., posture).

[0087] Figure 8This is an illustration of an iteratively growing graph 800 used to find the optimal trajectory after the fact. In an embodiment, the generated graphs 802, 804, and 806 are explored to find the optimal trajectory representing a reasonable path for the vehicle through the environment. The reasonable path can be used to compare the trajectories adopted by the AV in the same scene associated with the optimal trajectory according to the present technique. Graph generation makes it possible to evaluate the AV response by taking into account the rapidly generated optimal trajectory.

[0088] In embodiments, the optimal post-hoc trajectory is determined using known scenario information (e.g., using refinement information) and changes as the trajectory evolves. In other words, a trajectory optimal in the first pose may not remain optimal in subsequent poses. For example, during traversal of an environment, the AV may get stuck (e.g., unable to plan a path forward) or be forced to follow a path that results in spacing violations (e.g., violating the rulebook's hierarchical rules). In some cases, trajectories are generated without active reinforcement of the selected (e.g., traversed or navigated) trajectory as the AV travels. In conventional techniques, generated trajectories degrade over time. This technique evaluates candidate trajectories across a range of poses, such that a subset of the optimal trajectories across the range of poses is selected according to a rulebook. Trajectories are iteratively traversed to generate a graph of the optimal trajectory from the starting pose to the target pose. This technique generates the graph based on a fixed set of trajectories. In embodiments, the generated graph reflects the vehicle dynamics from the fixed set of trajectories using a range of poses.

[0089] exist Figure 8 In the example, the AV's first pose 810 is at the starting position. From the starting position, an alternative trajectory set 820 is generated for the vehicle under the first pose 810 (e.g., the root node of the corresponding graph), representing the vehicle's operation from the first pose 810. Within this alternative trajectory set, one or more optimal trajectories are determined. The optimal trajectory is used to determine the next pose, and alternative trajectory sets 822 and 824 are generated from the next pose. Specifically, the next pose 812 is evaluated to generate alternative trajectory set 822. The next pose 814 is evaluated to generate alternative trajectory set 824. In this embodiment, multiple alternative trajectory sets are generated iteratively until the target / destination 812 is reached.

[0090] like Figure 8 As shown, graphs 802, 804, and 806 are generated by computing an alternative trajectory set under the first pose 810 of the AV in a given scene. Based on the alternative trajectory set 820, a random subset of N1 trajectories is determined. The trajectories retained for the graph are those most likely to result in the optimal, best trajectory. In the example, the N2 trajectories with the highest scores at the current timestamp according to the rulebook are selected for the graph. For example, a pre-sorted list of scores based on rule violations is associated with each edge of the graph. This score is referenced below. Figure 9The following description is provided. Typically, values ​​for N1 and N2 are chosen for graph growth to balance graph quality with the computation speed. Larger values ​​for N1 and N2 result in an exponential increase in computation time, but also in improved graph quality.

[0091] In this embodiment, N2 trajectories are grown in addition to N1 more trajectories. For example, the next pose at the end of the N2 trajectories selected from the alternative trajectory set 820 (e.g., next poses 812, 814) is used to iteratively generate another random subset with N1 trajectories. Similarly, the trajectories retained are those most likely to produce the optimal trajectory (e.g., N2 trajectories). Graph growth continues until one or more trajectories leading to the target 812 are generated or a timeout occurs. The timeout can be a predetermined period of time before graph generation terminates. In this example, the timeout can be canceled or overridden to continue graph generation. The trajectories selected for the graph are those that have the lowest score according to the rulebook from the first pose to the target.

[0092] Figure 9 This is a diagram of System 900, which calculates scores according to the rulebook. Figure 9 In the example, rule manual 902 provides three exemplary hierarchical rules: R1 (highest priority), R2 (second highest priority), and R3 (lowest priority). Additionally, the fixed trajectory set 904 includes trajectory x, trajectory y, and trajectory z. This fixed trajectory set can be trajectory 616 ( Figure 6 )or Figure 8 The trajectories are 820, 822, or 824. In an embodiment, the fixed trajectory set represents the reasonable actions of the vehicle in most traffic situations. In the example, in response to simulations in a predetermined scenario, the planning system of the AV (e.g., Figure 4 The planning system 404 is used to generate a fixed trajectory set. In the example, as the AV travels from the initial pose, the AV calculates the inputs and outputs to represent the predetermined scene.

[0093] exist Figure 9 In the example, rule violations for each trajectory within a fixed set of trajectories are used to determine the rule violation score 906 for each trajectory. Specifically, each rule is evaluated to determine the rule violation score for each trajectory. At evaluation 908, rule R1 is evaluated to determine whether trajectory x, trajectory y, or trajectory z violates rule R1. Figure 9 In the example, trajectory z violates rule R1, while trajectories x and y do not. Trajectory z is assigned a score of 1 relative to rule R1. Trajectories x and y are assigned a score of 0 relative to rule R1. At evaluation 910, rule R2 is evaluated to determine whether trajectory x, trajectory y, or trajectory z violates rule R2. Figure 9In the example, no trajectory violates rule R2. Each trajectory is assigned a score of 0 relative to rule R2.

[0094] In some embodiments, the score represents the relative level of rule violation for all trajectories. Each individual rule is evaluated independently relative to each individual trajectory. The value of each score is at least partially based on the rule being evaluated. In other words, the score is calculated differently depending on the rule being evaluated. For example, for a rule maintaining spacing near a pedestrian, the score is the maximum number of instantaneous spacing violations associated with that pedestrian. In this example, a violation is entering the space near a pedestrian by exceeding a threshold distance to that pedestrian. The trajectories are sorted lexicographically based on the number of violations. Therefore, at evaluation 908, trajectory z is the only trajectory that violates rule R1, thus making trajectory z the worst trajectory. At evaluation 910, no trajectory violates rule R2, making no trajectory worse than the others relative to R2.

[0095] At evaluation point 912, rule R3 is evaluated to determine whether trajectory x, trajectory y, or trajectory z violates rule R3. Figure 9 In the example, trajectory z violates rule R3 more severely than trajectory y, and trajectory y violates rule R3 more severely than trajectory x. Trajectory z is assigned 10 points relative to rule R3, where 10 is the maximum number of violations of rule R3. Trajectory x is assigned a score of 1 relative to rule R3, and trajectory y is assigned a score of 2.

[0096] Typically, a random subset of N1 trajectories is determined from a fixed set of trajectories. Trajectories retained for the graph are those with scores above a predetermined threshold. In the example, N2 trajectories with scores above the predetermined threshold according to the rulebook are selected for the graph. In the embodiment, these N2 trajectories are grown along with N1 more trajectories. Generally, growing a graph refers to generating the next random set of trajectories from the ending poses of the N2 trajectories (e.g., those with scores above the predetermined threshold) from the previous fixed set of trajectories. The graph continues to grow until one or more trajectories reach the target pose. The trajectories selected for the graph are those with the lowest scores from the first pose to the target pose according to the rulebook. In this way, the graph is generated as a guided heuristic, using behavioral modeling and prediction datasets to create the graph. In the example, this technique does not converge to a single optimal trajectory. This technique obtains reasonably optimal trajectories, which is different from convergence to a single optimal trajectory.

[0097] Now for reference Figure 10 The diagram illustrates a flowchart of a process 1000 for graphical exploration of rulebook trajectory generation. In some embodiments, one or more steps described with respect to process 1000 are performed by... Figure 2 Autonomous vehicles 200 or Figure 4 The AV calculation 400 is performed (e.g., entirely and / or partially). Additionally or alternatively, in some embodiments, one or more steps described with respect to processing 1000 are performed by other means (such as...) that are separate from or include the autonomous system 202 relative to it. Figure 3 The device 300 (e.g., the device group) or the group of devices (e.g., completely and / or partially) are used.

[0098] At box 1002, an alternative trajectory set for the vehicle in the first pose is generated. In this embodiment, the alternative trajectories are a set of trajectories generated using behavior prediction. In this embodiment, the first pose is the root node of the corresponding graph. This alternative trajectory set represents the vehicle's operation starting from the first pose.

[0099] At box 1004, a trajectory from the alternative trajectory set is identified. In an embodiment, the trajectory violates the lowest behavior rule among a plurality of hierarchical rules, which has a lower priority than the behavior rules associated with other trajectories in the alternative trajectory set. Therefore, in an embodiment, this technique selects one or more trajectories that appear to be the optimal trajectory at the root node.

[0100] At box 1006, in response to the identified trajectory, a set of next alternative trajectories is generated from the next pose. The set of next alternative trajectories represents the vehicle's operation from the next pose. In the example, the next pose is located at the end of the identified trajectory. In this way, the graph grows iteratively based on the next pose generated from the identified trajectory. The set of next alternative trajectories for the vehicle can be generated from the next pose by applying the vehicle dynamics associated with the next pose to the possible trajectories associated with the location of the next pose. The vehicle dynamics include, for example, the rate and orientation associated with the trajectory in the next pose.

[0101] At box 1008, the next trajectory from the corresponding next alternative trajectory set is iteratively identified. In an embodiment, the next trajectory violates the lowest priority behavior rule among a plurality of hierarchical rules until a target pose is reached to generate a graph. This lowest priority behavior rule has a lower priority than the behavior rules associated with other trajectories in the corresponding next alternative trajectory set. In other words, in some embodiments, the technique iteratively repeats the step of identifying the best trajectory in a series of poses until the target pose is reached. In some embodiments, the best trajectory does not reach the target pose, and the technique iteratively repeats the step of identifying the best trajectory in a series of poses until a predetermined timeout occurs. In an example, the best trajectory is the trajectory that violates the lowest priority behavior rule compared to other trajectories, where trajectories are sorted according to rule violations. As described above, graph growth typically continues until a trajectory set is identified from the first pose. At box 1010, the vehicle is operated based on the graph. In an example, graph-based vehicle operation includes extracting the best trajectory from the graph and comparing the trajectory adopted by the vehicle with the best trajectory. In this way, the performance of the vehicle is evaluated with the best trajectory in mind. The best trajectory extracted from the graph is used to provide feedback on the performance of the vehicle.

[0102] In the preceding description, aspects and embodiments of this disclosure have been described with reference to numerous specific details, which may vary from implementation to implementation. Therefore, the specification and drawings should be considered illustrative rather than restrictive. The sole and exclusive indication of the scope of this invention, and what the applicant expects to be the scope of this invention, is the literal and equivalent scope of the claims published from this application in the specific form of the published claims, including any subsequent amendments. Any definitions of terms expressly set forth herein for inclusion in such claims should be taken as meaning as such terms are used in the claims. Furthermore, when the term “comprising” is used in the preceding specification or appended claims, what follows that phrase may be an additional step or entity, or a sub-step / sub-entity of a previously stated step or entity.

Claims

1. A method for a vehicle, comprising: generating, with at least one processor, a set of alternative trajectories for a vehicle in a first pose, the set of alternative trajectories representing operation of the vehicle from the first pose; identifying, with the at least one processor, a trajectory from the set of alternative trajectories, wherein the trajectory violates a lowest behavioral rule of a hierarchy of a plurality of rules, the lowest behavioral rule having a lower priority than behavioral rules associated with other trajectories in the set of alternative trajectories; generating, with the at least one processor, a next set of alternative trajectories for the vehicle from a next pose in response to identifying the trajectory, the next set of alternative trajectories representing operation of the vehicle from the next pose, wherein the next pose is at an end of the identified trajectory; iteratively identifying, with the at least one processor, a next trajectory from a corresponding next set of alternative trajectories until a target pose is reached or a timeout is reached to generate a graph, wherein a next trajectory violates the lowest behavioral rule of the hierarchy of the plurality of rules, the lowest behavioral rule having a lower priority than behavioral rules associated with other trajectories in the corresponding next set of alternative trajectories; and transmitting, with the at least one processor, a message to a control system of the vehicle to operate the vehicle based on the graph.

2. The method of claim 1, further comprising pruning, with the at least one processor, a trajectory from the set of alternative trajectories or the next set of alternative trajectories based on a likelihood of being an optimal trajectory.

3. The method of claim 1 or 2, further comprising pruning, with the at least one processor, a trajectory from the set of alternative trajectories or the next set of alternative trajectories based on a remaining trajectory on a roadway.

4. The method of claim 1 or 2, further comprising identifying, with the at least one processor, the trajectory or the next trajectory using at least one of a minimal violation planning, model predictive control, and machine learning, the identifying based on the hierarchy of the plurality of rules.

5. The method of claim 1 or 2, wherein, each behavioral rule of the hierarchy of the plurality of rules has a respective priority relative to each other behavioral rule of the hierarchy of the plurality of rules.

6. The method of claim 1 or 2, further comprising generating, with the at least one processor, a next set of alternative trajectories for the vehicle from the next pose by applying a vehicle dynamics associated with the next pose to possible trajectories associated with a location of the next pose.

7. The method of claim 1 or 2, further comprising assigning, with the at least one processor, a rule violation value to a trajectory of the set of alternative trajectories or the next set of alternative trajectories, wherein the rule violation value represents a weight associated with a trajectory in the graph.

8. A system for a vehicle, comprising: at least one processor, and at least one non-transitory storage medium storing instructions that, when executed by the at least one processor, cause the at least one processor to perform the method of any of claims 1-7.

9. At least one non-transitory storage medium having stored thereon instructions that, when executed by at least one processor, cause the at least one processor to perform the method of any of claims 1-7.

10. A computer program product comprising a program which, when executed by at least one processor, causes the at least one processor to carry out the method of any of claims 1-7.

Images