Homotopy generation

Abstract

Provided are methods, systems, and computer program products for homotopy generation. In an example, at least one free space trajectory is obtained. A spline baseline is created based on the at least one free space trajectory. Station constraints are computed in view of at least one mode of operation. A homotopy is generated based on, at least in part, the station constraints.

Description

Attorney Docket No.46154-0555WO1 P2023081 / US / PRV1 Homotopy Generation CROSS REFERENCE TO RELATED APPLICATION

[0001] This application claims priority to U.S. Provisional Application No. 63 / 693,964, entitled “Homotopy Generation” filed on September 12, 2024 the disclosure of which is incorporated herein by reference in its entirety. BACKGROUND

[0002] Autonomous vehicles navigate by computing trajectories within homotopies, also referred to as a trajectory realization. Model-predictive control (MPC) applies constraints within a homotropy associated with an autonomous vehicle to predict a trajectory for the autonomous vehicle. The homotopy constraints can limit the maneuvers available to the autonomous vehicle. BRIEF DESCRIPTION OF THE FIGURES

[0003] FIG. 1 is an example environment in which a vehicle including one or more components of an autonomous system can be implemented;

[0004] FIG. 2 is a diagram of one or more systems of a vehicle including an autonomous system;

[0005] FIG. 3 is a diagram of components of one or more devices and / or one or more systems of FIGS. 1 and 2;

[0006] FIG. 4A is a diagram of certain components of an autonomous system;

[0007] FIG. 4B is a diagram of an implementation of a neural network;

[0008] FIG. 4C and 4D are a diagram illustrating example operation of a CNN;

[0009] FIGS. 5A–5D show diagrams of implementations of a process for homotopy generation;

[0010] FIG. 6 shows a system architecture that enables homotopy generation;

[0011] FIG. 7A shows a maneuvers and constraints;

[0012] FIG. 7B shows conditions applied to the unstructured planner;Attorney Docket No.46154-0555WO1 P2023081 / US / PRV1

[0013] FIG. 7C shows various scenarios;

[0014] FIG. 7D shows output of unstructured planner.

[0015] FIG. 8A shows trajectory generation based on homotopies;

[0016] FIG. 8B shows a workflow for the generation of trajectories using spline homotopy;

[0017] FIG. 9 shows a step in a planning horizon;

[0018] FIG. 10A shows a first Mode A of operation;

[0019] FIG. 10B shows a second Mode B of operation;

[0020] FIG. 10C shows a third Mode C of operation;

[0021] FIG. 11 shows an architecture for homotopy generation;

[0022] FIG. 12 shows heuristic interactive prediction rollouts from prediction processing;

[0023] FIG. 13 shows prediction projection;

[0024] FIGs. 14A and 14B show station constraint computation;

[0025] FIG. 15 shows spline tube computation; and

[0026] FIG. 16 shows a flowchart of a process for homotopy generation. DETAILED DESCRIPTION

[0027] In the following description numerous specific details are set forth in order to provide a thorough understanding of the present disclosure for the purposes of explanation. It will be apparent, however, that the embodiments described by the present disclosure can be practiced without these specific details. In some instances, well-known structures and devices are illustrated in block diagram form in order to avoid unnecessarily obscuring aspects of the present disclosure.

[0028] Specific arrangements or orderings of schematic elements, such as those representing systems, devices, modules, instruction blocks, data elements, and / or the like are illustrated in the drawings for ease of description. However, it will be understood by those skilled in the art that the specific ordering or arrangement of the schematic elements in the drawings is not meant to imply that a particular order or sequence of processing, or separation of processes, is required unless explicitly described as such. Further, the inclusion of a schematic element in a drawing is not meant to imply that suchAttorney Docket No.46154-0555WO1 P2023081 / US / PRV1 element is required in all embodiments or that the features represented by such element may not be included in or combined with other elements in some embodiments unless explicitly described as such.

[0029] Further, where connecting elements such as solid or dashed lines or arrows are used in the drawings to illustrate a connection, relationship, or association between or among two or more other schematic elements, the absence of any such connecting elements is not meant to imply that no connection, relationship, or association can exist. In other words, some connections, relationships, or associations between elements are not illustrated in the drawings so as not to obscure the disclosure. In addition, for ease of illustration, a single connecting element can be used to represent multiple connections, relationships or associations between elements. For example, where a connecting element represents communication of signals, data, or instructions (e.g., “software instructions”), it should be understood by those skilled in the art that such element can represent one or multiple signal paths (e.g., a bus), as may be needed, to affect the communication.

[0030] Although the terms first, second, third, and / or the like are used to describe various elements, these elements should not be limited by these terms. The terms first, second, third, and / or the like are used only to distinguish one element from another. For example, a first contact could be termed a second contact and, similarly, a second contact could be termed a first contact without departing from the scope of the described embodiments. The first contact and the second contact are both contacts, but they are not the same contact.

[0031] The terminology used in the description of the various described embodiments herein is included for the purpose of describing particular embodiments only and is not intended to be limiting. As used in the description of the various described embodiments and the appended claims, the singular forms “a,” “an” and “the” are intended to include the plural forms as well and can be used interchangeably with “one or more” or “at least one,” unless the context clearly indicates otherwise. It will also be understood that the term “and / or” as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items. It will be further understood that the terms “includes,” “including,” “comprises,” and / or “comprising,” when used in thisAttorney Docket No.46154-0555WO1 P2023081 / US / PRV1 description specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0032] As used herein, the terms “communication” and “communicate” refer to at least one of the reception, receipt, transmission, transfer, provision, and / or the like of information (or information represented by, for example, data, signals, messages, instructions, commands, and / or the like). For one unit (e.g., a device, a system, a component of a device or system, combinations thereof, and / or the like) to be in communication with another unit means that the one unit is able to directly or indirectly receive information from and / or send (e.g., transmit) information to the other unit. This may refer to a direct or indirect connection that is wired and / or wireless in nature. Additionally, two units may be in communication with each other even though the information transmitted may be modified, processed, relayed, and / or routed between the first and second unit. For example, a first unit may be in communication with a second unit even though the first unit passively receives information and does not actively transmit information to the second unit. As another example, a first unit may be in communication with a second unit if at least one intermediary unit (e.g., a third unit located between the first unit and the second unit) processes information received from the first unit and transmits the processed information to the second unit. In some embodiments, a message may refer to a network packet (e.g., a data packet and / or the like) that includes data.

[0033] As used herein, the term “if” is, optionally, construed to mean “when”, “upon”, “in response to determining,” “in response to detecting,” and / or the like, depending on the context. Similarly, the phrase “if it is determined” or “if [a stated condition or event] is detected” is, optionally, construed to mean “upon determining,” “in response to determining,” “upon detecting [the stated condition or event],” “in response to detecting [the stated condition or event],” and / or the like, depending on the context. Also, as used herein, the terms “has”, “have”, “having”, or the like are intended to be open-ended terms. Further, the phrase “based on” is intended to mean “based at least partially on” unless explicitly stated otherwise.Attorney Docket No.46154-0555WO1 P2023081 / US / PRV1

[0034] Reference will now be made in detail to embodiments, examples of which are illustrated in the accompanying drawings. In the following detailed description, numerous specific details are set forth in order to provide a thorough understanding of the various described embodiments. However, it will be apparent to one of ordinary skill in the art that the various described embodiments can be practiced without these specific details. In other instances, well-known methods, procedures, components, circuits, and networks have not been described in detail so as not to unnecessarily obscure aspects of the embodiments.

[0035] General Overview

[0036] In some aspects and / or embodiments, systems, methods, and computer program products described herein include and / or implement homotopy generation. Traditional homotopy constraints are represented by a lane-centric, spatio-temporal tube with left and right bounds. As such, multiple types of maneuvers may be disabled or otherwise unavailable to an ego vehicle with such constraints. For example, trajectories realized using a traditional homotopy defined by lane structures comply with constraints established within the traditional homotopy. Paths that violate the lane structure are not possible within the traditional homotopy defined by lane structures, at least because assumptions are made on the vehicle state propagation through spatial sampling. This leads to a discrepancy between actual real-world homotopy constraints that represent the physical world (including obstacles) and the constraints which a traditional optimal control problem is solved for by a model predictive controller. Large discrepancies in these constraints occur in dynamic environments without lane / road markings. To accurately and quickly execute maneuvers or trajectories in a free, unstructured environment, at least one initial trajectory is obtained. A spline baseline is created based on the at least one initial trajectory. Station constraints are computed in view of at least one mode of operation and the spline baseline. A homotopy is generated based on, at least in part, the station constraints.

[0037] Referring now to FIG. 1, illustrated is example environment 100 in which vehicles that include autonomous systems, as well as vehicles that do not, are operated. As illustrated, environment 100 includes vehicles 102a–102n, objects 104a–104n, routes 106a–106n, area 108, vehicle-to-infrastructure (V2I) device 110, network 112, remoteAttorney Docket No.46154-0555WO1 P2023081 / US / PRV1 autonomous vehicle (AV) system 114, fleet management system 116, and V2I system 118. Vehicles 102a–102n, vehicle-to-infrastructure (V2I) device 110, network 112, autonomous vehicle (AV) system 114, fleet management system 116, and V2I system 118 interconnect (e.g., establish a connection to communicate and / or the like) via wired connections, wireless connections, or a combination of wired or wireless connections. In some embodiments, objects 104a–104n interconnect with at least one of vehicles 102a– 102n, vehicle-to-infrastructure (V2I) device 110, network 112, autonomous vehicle (AV) system 114, fleet management system 116, and V2I system 118 via wired connections, wireless connections, or a combination of wired or wireless connections.

[0038] Vehicles 102a–102n (referred to individually as vehicle 102 and collectively as vehicles 102) include at least one device configured to transport goods and / or people. In some embodiments, vehicles 102 are configured to be in communication with V2I device 110, remote AV system 114, fleet management system 116, and / or V2I system 118 via network 112. In some embodiments, vehicles 102 include cars, buses, trucks, trains, and / or the like. In some embodiments, vehicles 102 are the same as, or similar to, vehicles 200, described herein (see FIG.2). In some embodiments, a vehicle 200 of a set of vehicles 200 is associated with an autonomous fleet manager. In some embodiments, vehicles 102 travel along respective routes 106a–106n (referred to individually as route 106 and collectively as routes 106), as described herein. In some embodiments, one or more vehicles 102 include an autonomous system (e.g., an autonomous system that is the same as or similar to autonomous system 202).

[0039] Objects 104a–104n (referred to individually as object 104 and collectively as objects 104) include, for example, at least one vehicle, at least one pedestrian, at least one cyclist, at least one structure (e.g., a building, a sign, a fire hydrant, etc.), and / or the like. Each object 104 is stationary (e.g., located at a fixed location for a period of time) or mobile (e.g., having a velocity and associated with at least one trajectory). In some embodiments, objects 104 are associated with corresponding locations in area 108.

[0040] Routes 106a–106n (referred to individually as route 106 and collectively as routes 106) are each associated with (e.g., prescribe) a sequence of actions (also known as a trajectory) connecting states along which an AV can navigate. Each route 106 starts at an initial state (e.g., a state that corresponds to a first spatiotemporal location, velocity,Attorney Docket No.46154-0555WO1 P2023081 / US / PRV1 and / or the like) and ends at a final goal state (e.g., a state that corresponds to a second spatiotemporal location that is different from the first spatiotemporal location) or goal region (e.g. a subspace of acceptable states (e.g., terminal states)). In some embodiments, the first state includes a location at which an individual or individuals are to be picked-up by the AV and the second state or region includes a location or locations at which the individual or individuals picked-up by the AV are to be dropped-off. In some embodiments, routes 106 include a plurality of acceptable state sequences (e.g., a plurality of spatiotemporal location sequences), the plurality of state sequences associated with (e.g., defining) a plurality of trajectories. In an example, routes 106 include only high level actions or imprecise state locations, such as a series of connected roads dictating turning directions at roadway intersections. Additionally, or alternatively, routes 106 may include more precise actions or states such as, for example, specific target lanes or precise locations within the lane areas and targeted speed at those positions. In an example, routes 106 include a plurality of precise state sequences along the at least one high level action sequence with a limited lookahead horizon to reach intermediate goals, where the combination of successive iterations of limited horizon state sequences cumulatively correspond to a plurality of trajectories that collectively form the high level route to terminate at the final goal state or region.

[0041] Area 108 includes a physical area (e.g., a geographic region) within which vehicles 102 can navigate. In an example, area 108 includes at least one state (e.g., a country, a province, an individual state of a plurality of states included in a country, etc.), at least one portion of a state, at least one city, at least one portion of a city, etc. In some embodiments, area 108 includes at least one named thoroughfare (referred to herein as a “road”) such as a highway, an interstate highway, a parkway, a city street, etc. Additionally, or alternatively, in some examples area 108 includes at least one unnamed road such as a driveway, a section of a parking lot, a section of a vacant and / or undeveloped lot, a dirt path, etc. In some embodiments, a road includes at least one lane (e.g., a portion of the road that can be traversed by vehicles 102). In an example, a road includes at least one lane associated with (e.g., identified based on) at least one lane marking.Attorney Docket No.46154-0555WO1 P2023081 / US / PRV1

[0042] Vehicle-to-Infrastructure (V2I) device 110 (sometimes referred to as a Vehicle- to-Infrastructure or Vehicle-to-Everything (V2X) device) includes at least one device configured to be in communication with vehicles 102 and / or V2I infrastructure system 118. In some embodiments, V2I device 110 is configured to be in communication with vehicles 102, remote AV system 114, fleet management system 116, and / or V2I system 118 via network 112. In some embodiments, V2I device 110 includes a radio frequency identification (RFID) device, signage, cameras (e.g., two-dimensional (2D) and / or three- dimensional (3D) cameras), lane markers, streetlights, parking meters, etc. In some embodiments, V2I device 110 is configured to communicate directly with vehicles 102. Additionally, or alternatively, in some embodiments V2I device 110 is configured to communicate with vehicles 102, remote AV system 114, and / or fleet management system 116 via V2I system 118. In some embodiments, V2I device 110 is configured to communicate with V2I system 118 via network 112.

[0043] Network 112 includes one or more wired and / or wireless networks. In an example, network 112 includes a cellular network (e.g., a long term evolution (LTE) network, a third generation (3G) network, a fourth generation (4G) network, a fifth generation (5G) network, a code division multiple access (CDMA) network, etc.), a public land mobile network (PLMN), a local area network (LAN), a wide area network (WAN), a metropolitan area network (MAN), a telephone network (e.g., the public switched telephone network (PSTN), a private network, an ad hoc network, an intranet, the Internet, a fiber optic-based network, a cloud computing network, etc., a combination of some or all of these networks, and / or the like.

[0044] Remote AV system 114 includes at least one device configured to be in communication with vehicles 102, V2I device 110, network 112, fleet management system 116, and / or V2I system 118 via network 112. In an example, remote AV system 114 includes a server, a group of servers, and / or other like devices. In some embodiments, remote AV system 114 is co-located with the fleet management system 116. In some embodiments, remote AV system 114 is involved in the installation of some or all of the components of a vehicle, including an autonomous system, an autonomous vehicle compute, software implemented by an autonomous vehicle compute, and / or the like. InAttorney Docket No.46154-0555WO1 P2023081 / US / PRV1 some embodiments, remote AV system 114 maintains (e.g., updates and / or replaces) such components and / or software during the lifetime of the vehicle.

[0045] Fleet management system 116 includes at least one device configured to be in communication with vehicles 102, V2I device 110, remote AV system 114, and / or V2I infrastructure system 118. In an example, fleet management system 116 includes a server, a group of servers, and / or other like devices. In some embodiments, fleet management system 116 is associated with a ridesharing company (e.g., an organization that controls operation of multiple vehicles (e.g., vehicles that include autonomous systems and / or vehicles that do not include autonomous systems) and / or the like).

[0046] In some embodiments, V2I system 118 includes at least one device configured to be in communication with vehicles 102, V2I device 110, remote AV system 114, and / or fleet management system 116 via network 112. In some examples, V2I system 118 is configured to be in communication with V2I device 110 via a connection different from network 112. In some embodiments, V2I system 118 includes a server, a group of servers, and / or other like devices. In some embodiments, V2I system 118 is associated with a municipality or a private institution (e.g., a private institution that maintains V2I device 110 and / or the like).

[0047] The number and arrangement of elements illustrated in FIG. 1 are provided as an example. There can be additional elements, fewer elements, different elements, and / or differently arranged elements, than those illustrated in FIG.1. Additionally, or alternatively, at least one element of environment 100 can perform one or more functions described as being performed by at least one different element of FIG. 1. Additionally, or alternatively, at least one set of elements of environment 100 can perform one or more functions described as being performed by at least one different set of elements of environment 100.

[0048] Referring now to FIG. 2, vehicle 200 (which may be the same as, or similar to vehicles 102 of FIG.1) includes or is associated with autonomous system 202, powertrain control system 204, steering control system 206, and brake system 208. In some embodiments, vehicle 200 is the same as or similar to vehicle 102 (see FIG. 1). In some embodiments, autonomous system 202 is configured to confer vehicle 200 autonomous driving capability (e.g., implement at least one driving automation or maneuver-basedAttorney Docket No.46154-0555WO1 P2023081 / US / PRV1 function, feature, device, and / or the like that enable vehicle 200 to be partially or fully operated without human intervention including, without limitation, fully autonomous vehicles (e.g., vehicles that forego reliance on human intervention such as Level 5 ADS- operated vehicles), highly autonomous vehicles (e.g., vehicles that forego reliance on human intervention in certain situations such as Level 4 ADS-operated vehicles), conditional autonomous vehicles (e.g., vehicles that forego reliance on human intervention in limited situations such as Level 3 ADS-operated vehicles) and / or the like. In one embodiment, autonomous system 202 includes operational or tactical functionality required to operate vehicle 200 in on-road traffic and perform part or all of Dynamic Driving Task (DDT) on a sustained basis. In another embodiment, autonomous system 202 includes an Advanced Driver Assistance System (ADAS) that includes driver support features. Autonomous system 202 supports various levels of driving automation, ranging from no driving automation (e.g., Level 0) to full driving automation (e.g., Level 5). For a detailed description of fully autonomous vehicles and highly autonomous vehicles, reference may be made to SAE International's standard J3016: Taxonomy and Definitions for Terms Related to On-Road Motor Vehicle Automated Driving Systems, which is incorporated by reference in its entirety. In some embodiments, vehicle 200 is associated with an autonomous fleet manager and / or a ridesharing company.

[0049] Autonomous system 202 includes a sensor suite that includes one or more devices such as cameras 202a, LiDAR sensors 202b, radar sensors 202c, and microphones 202d. In some embodiments, autonomous system 202 can include more or fewer devices and / or different devices (e.g., ultrasonic sensors, inertial sensors, GPS receivers (discussed below), odometry sensors that generate data associated with an indication of a distance that vehicle 200 has traveled, and / or the like). In some embodiments, autonomous system 202 uses the one or more devices included in autonomous system 202 to generate data associated with environment 100, described herein. The data generated by the one or more devices of autonomous system 202 can be used by one or more systems described herein to observe the environment (e.g., environment 100) in which vehicle 200 is located. In some embodiments, autonomous system 202 includes communication device 202e, autonomous vehicle compute 202f, drive-by-wire (DBW) system 202h, and safety controller 202g.Attorney Docket No.46154-0555WO1 P2023081 / US / PRV1

[0050] Cameras 202a include at least one device configured to be in communication with communication device 202e, autonomous vehicle compute 202f, and / or safety controller 202g via a bus (e.g., a bus that is the same as or similar to bus 302 of FIG. 3). Cameras 202a include at least one camera (e.g., a digital camera using a light sensor such as a Charge-Coupled Device (CCD), a thermal camera, an infrared (IR) camera, an event camera, and / or the like) to capture images including physical objects (e.g., cars, buses, curbs, people, and / or the like). In some embodiments, camera 202a generates camera data as output. In some examples, camera 202a generates camera data that includes image data associated with an image. In this example, the image data may specify at least one parameter (e.g., image characteristics such as exposure, brightness, etc., an image timestamp, and / or the like) corresponding to the image. In such an example, the image may be in a format (e.g., RAW, JPEG, PNG, and / or the like). In some embodiments, camera 202a includes a plurality of independent cameras configured on (e.g., positioned on) a vehicle to capture images for the purpose of stereopsis (stereo vision). In some examples, camera 202a includes a plurality of cameras that generate image data and transmit the image data to autonomous vehicle compute 202f and / or a fleet management system (e.g., a fleet management system that is the same as or similar to fleet management system 116 of FIG. 1). In such an example, autonomous vehicle compute 202f determines depth to one or more objects in a field of view of at least two cameras of the plurality of cameras based on the image data from the at least two cameras. In some embodiments, cameras 202a is configured to capture images of objects within a distance from cameras 202a (e.g., up to 100 meters, up to a kilometer, and / or the like). Accordingly, cameras 202a include features such as sensors and lenses that are optimized for perceiving objects that are at one or more distances from cameras 202a.

[0051] In an embodiment, camera 202a includes at least one camera configured to capture one or more images associated with one or more traffic lights, street signs and / or other physical objects that provide visual navigation information. In some embodiments, camera 202a generates traffic light data associated with one or more images. In some examples, camera 202a generates TLD (Traffic Light Detection) data associated with one or more images that include a format (e.g., RAW, JPEG, PNG, and / or the like). In someAttorney Docket No.46154-0555WO1 P2023081 / US / PRV1 embodiments, camera 202a that generates TLD data differs from other systems described herein incorporating cameras in that camera 202a can include one or more cameras with a wide field of view (e.g., a wide-angle lens, a fish-eye lens, a lens having a viewing angle of approximately 120 degrees or more, and / or the like) to generate images about as many physical objects as possible.

[0052] Light Detection and Ranging (LiDAR) sensors 202b include at least one device configured to be in communication with communication device 202e, autonomous vehicle compute 202f, and / or safety controller 202g via a bus (e.g., a bus that is the same as or similar to bus 302 of FIG.3). LiDAR sensors 202b include a system configured to transmit light from a light emitter (e.g., a laser transmitter). Light emitted by LiDAR sensors 202b include light (e.g., infrared light and / or the like) that is outside of the visible spectrum. In some embodiments, during operation, light emitted by LiDAR sensors 202b encounters a physical object (e.g., a vehicle) and is reflected back to LiDAR sensors 202b. In some embodiments, the light emitted by LiDAR sensors 202b does not penetrate the physical objects that the light encounters. LiDAR sensors 202b also include at least one light detector which detects the light that was emitted from the light emitter after the light encounters a physical object. In some embodiments, at least one data processing system associated with LiDAR sensors 202b generates an image (e.g., a point cloud, a combined point cloud, and / or the like) representing the objects included in a field of view of LiDAR sensors 202b. In some examples, the at least one data processing system associated with LiDAR sensor 202b generates an image that represents the boundaries of a physical object, the surfaces (e.g., the topology of the surfaces) of the physical object, and / or the like. In such an example, the image is used to determine the boundaries of physical objects in the field of view of LiDAR sensors 202b.

[0053] Radio Detection and Ranging (radar) sensors 202c include at least one device configured to be in communication with communication device 202e, autonomous vehicle compute 202f, and / or safety controller 202g via a bus (e.g., a bus that is the same as or similar to bus 302 of FIG.3). Radar sensors 202c include a system configured to transmit radio waves (either pulsed or continuously). The radio waves transmitted by radar sensors 202c include radio waves that are within a predetermined spectrum. In some embodiments, during operation, radio waves transmitted by radar sensors 202cAttorney Docket No.46154-0555WO1 P2023081 / US / PRV1 encounter a physical object and are reflected back to radar sensors 202c. In some embodiments, the radio waves transmitted by radar sensors 202c are not reflected by some objects. In some embodiments, at least one data processing system associated with radar sensors 202c generates signals representing the objects included in a field of view of radar sensors 202c. For example, the at least one data processing system associated with radar sensor 202c generates an image that represents the boundaries of a physical object, the surfaces (e.g., the topology of the surfaces) of the physical object, and / or the like. In some examples, the image is used to determine the boundaries of physical objects in the field of view of radar sensors 202c.

[0054] Microphones 202d includes at least one device configured to be in communication with communication device 202e, autonomous vehicle compute 202f, and / or safety controller 202g via a bus (e.g., a bus that is the same as or similar to bus 302 of FIG. 3). Microphones 202d include one or more microphones (e.g., array microphones, external microphones, and / or the like) that capture audio signals and generate data associated with (e.g., representing) the audio signals. In some examples, microphones 202d include transducer devices and / or like devices. In some embodiments, one or more systems described herein can receive the data generated by microphones 202d and determine a position of an object relative to vehicle 200 (e.g., a distance and / or the like) based on the audio signals associated with the data.

[0055] Communication device 202e includes at least one device configured to be in communication with cameras 202a, LiDAR sensors 202b, radar sensors 202c, microphones 202d, autonomous vehicle compute 202f, safety controller 202g, and / or DBW (Drive-By-Wire) system 202h. For example, communication device 202e may include a device that is the same as or similar to communication interface 314 of FIG. 3. In some embodiments, communication device 202e includes a vehicle-to-vehicle (V2V) communication device (e.g., a device that enables wireless communication of data between vehicles).

[0056] Autonomous vehicle compute 202f include at least one device configured to be in communication with cameras 202a, LiDAR sensors 202b, radar sensors 202c, microphones 202d, communication device 202e, safety controller 202g, and / or DBW system 202h. In some examples, autonomous vehicle compute 202f includes a deviceAttorney Docket No.46154-0555WO1 P2023081 / US / PRV1 such as a client device, a mobile device (e.g., a cellular telephone, a tablet, and / or the like), a server (e.g., a computing device including one or more central processing units, graphical processing units, and / or the like), and / or the like. In some embodiments, autonomous vehicle compute 202f is the same as or similar to autonomous vehicle compute 400, described herein. Additionally, or alternatively, in some embodiments autonomous vehicle compute 202f is configured to be in communication with an autonomous vehicle system (e.g., an autonomous vehicle system that is the same as or similar to remote AV system 114 of FIG. 1), a fleet management system (e.g., a fleet management system that is the same as or similar to fleet management system 116 of FIG. 1), a V2I device (e.g., a V2I device that is the same as or similar to V2I device 110 of FIG. 1), and / or a V2I system (e.g., a V2I system that is the same as or similar to V2I system 118 of FIG. 1).

[0057] Safety controller 202g includes at least one device configured to be in communication with cameras 202a, LiDAR sensors 202b, radar sensors 202c, microphones 202d, communication device 202e, autonomous vehicle computer 202f, and / or DBW system 202h. In some examples, safety controller 202g includes one or more controllers (electrical controllers, electromechanical controllers, and / or the like) that are configured to generate and / or transmit control signals to operate one or more devices of vehicle 200 (e.g., powertrain control system 204, steering control system 206, brake system 208, and / or the like). In some embodiments, safety controller 202g is configured to generate control signals that take precedence over (e.g., overrides) control signals generated and / or transmitted by autonomous vehicle compute 202f.

[0058] DBW system 202h includes at least one device configured to be in communication with communication device 202e and / or autonomous vehicle compute 202f. In some examples, DBW system 202h includes one or more controllers (e.g., electrical controllers, electromechanical controllers, and / or the like) that are configured to generate and / or transmit control signals to operate one or more devices of vehicle 200 (e.g., powertrain control system 204, steering control system 206, brake system 208, and / or the like). Additionally, or alternatively, the one or more controllers of DBW system 202h are configured to generate and / or transmit control signals to operate at least oneAttorney Docket No.46154-0555WO1 P2023081 / US / PRV1 different device (e.g., a turn signal, headlights, door locks, windshield wipers, and / or the like) of vehicle 200.

[0059] Powertrain control system 204 includes at least one device configured to be in communication with DBW system 202h. In some examples, powertrain control system 204 includes at least one controller, actuator, and / or the like. In some embodiments, powertrain control system 204 receives control signals from DBW system 202h and powertrain control system 204 causes vehicle 200 to make longitudinal vehicle motion, such as start moving forward, stop moving forward, start moving backward, stop moving backward, accelerate in a direction, decelerate in a direction or to make lateral vehicle motion such as performing a left turn, performing a right turn, and / or the like. In an example, powertrain control system 204 causes the energy (e.g., fuel, electricity, and / or the like) provided to a motor of the vehicle to increase, remain the same, or decrease, thereby causing at least one wheel of vehicle 200 to rotate or not rotate.

[0060] Steering control system 206 includes at least one device configured to rotate one or more wheels of vehicle 200. In some examples, steering control system 206 includes at least one controller, actuator, and / or the like. In some embodiments, steering control system 206 causes the front two wheels and / or the rear two wheels of vehicle 200 to rotate to the left or right to cause vehicle 200 to turn to the left or right. In other words, steering control system 206 causes activities necessary for the regulation of the y-axis component of vehicle motion.

[0061] Brake system 208 includes at least one device configured to actuate one or more brakes to cause vehicle 200 to reduce speed and / or remain stationary. In some examples, brake system 208 includes at least one controller and / or actuator that is configured to cause one or more calipers associated with one or more wheels of vehicle 200 to close on a corresponding rotor of vehicle 200. Additionally, or alternatively, in some examples brake system 208 includes an automatic emergency braking (AEB) system, a regenerative braking system, and / or the like.

[0062] In some embodiments, vehicle 200 includes at least one platform sensor (not explicitly illustrated) that measures or infers properties of a state or a condition of vehicle 200. In some examples, vehicle 200 includes platform sensors such as a global positioning system (GPS) receiver, an inertial measurement unit (IMU), a wheel speedAttorney Docket No.46154-0555WO1 P2023081 / US / PRV1 sensor, a wheel brake pressure sensor, a wheel torque sensor, an engine torque sensor, a steering angle sensor, and / or the like. Although brake system 208 is illustrated to be located in the near side of vehicle 200 in FIG. 2, brake system 208 may be located anywhere in vehicle 200.

[0063] Referring now to FIG. 3, illustrated is a schematic diagram of a device 300. As illustrated, device 300 includes processor 304, memory 306, storage component 308, input interface 310, output interface 312, communication interface 314, and bus 302. In some embodiments, device 300 corresponds to at least one device of vehicles 102 (e.g., at least one device of a system of vehicles 102), and / or one or more devices of network 112 (e.g., one or more devices of a system of network 112). In some embodiments, one or more devices of vehicles 102 (e.g., one or more devices of a system of vehicles 102), and / or one or more devices of network 112 (e.g., one or more devices of a system of network 112) include at least one device 300 and / or at least one component of device 300. As shown in FIG. 3, device 300 includes bus 302, processor 304, memory 306, storage component 308, input interface 310, output interface 312, and communication interface 314.

[0064] Bus 302 includes a component that permits communication among the components of device 300. In some cases, processor 304 includes a processor (e.g., a central processing unit (CPU), a graphics processing unit (GPU), an accelerated processing unit (APU), and / or the like), a microphone, a digital signal processor (DSP), and / or any processing component (e.g., a field-programmable gate array (FPGA), an application specific integrated circuit (ASIC), and / or the like) that can be programmed to perform at least one function. Memory 306 includes random access memory (RAM), read- only memory (ROM), and / or another type of dynamic and / or static storage device (e.g., flash memory, magnetic memory, optical memory, and / or the like) that stores data and / or instructions for use by processor 304.

[0065] Storage component 308 stores data and / or software related to the operation and use of device 300. In some examples, storage component 308 includes a hard disk (e.g., a magnetic disk, an optical disk, a magneto-optic disk, a solid state disk, and / or the like), a compact disc (CD), a digital versatile disc (DVD), a floppy disk, a cartridge, aAttorney Docket No.46154-0555WO1 P2023081 / US / PRV1 magnetic tape, a CD-ROM, RAM, PROM, EPROM, FLASH-EPROM, NV-RAM, and / or another type of computer readable medium, along with a corresponding drive.

[0066] Input interface 310 includes a component that permits device 300 to receive information, such as via user input (e.g., a touchscreen display, a keyboard, a keypad, a mouse, a button, a switch, a microphone, a camera, and / or the like). Additionally or alternatively, in some embodiments input interface 310 includes a sensor that senses information (e.g., a global positioning system (GPS) receiver, an accelerometer, a gyroscope, an actuator, and / or the like). Output interface 312 includes a component that provides output information from device 300 (e.g., a display, a speaker, one or more light- emitting diodes (LEDs), and / or the like).

[0067] In some embodiments, communication interface 314 includes a transceiver-like component (e.g., a transceiver, a separate receiver and transmitter, and / or the like) that permits device 300 to communicate with other devices via a wired connection, a wireless connection, or a combination of wired and wireless connections. In some examples, communication interface 314 permits device 300 to receive information from another device and / or provide information to another device. In some examples, communication interface 314 includes an Ethernet interface, an optical interface, a coaxial interface, an infrared interface, a radio frequency (RF) interface, a universal serial bus (USB) interface, a Wi-Fi®interface, a cellular network interface, and / or the like.

[0068] In some embodiments, device 300 performs one or more processes described herein. Device 300 performs these processes based on processor 304 executing software instructions stored by a computer-readable medium, such as memory 305 and / or storage component 308. A computer-readable medium (e.g., a non-transitory computer readable medium) is defined herein as a non-transitory memory device. A non-transitory memory device includes memory space located inside a single physical storage device or memory space spread across multiple physical storage devices.

[0069] In some embodiments, software instructions are read into memory 306 and / or storage component 308 from another computer-readable medium or from another device via communication interface 314. When executed, software instructions stored in memory 306 and / or storage component 308 cause processor 304 to perform one or more processes described herein. Additionally or alternatively, hardwired circuitry is used inAttorney Docket No.46154-0555WO1 P2023081 / US / PRV1 place of or in combination with software instructions to perform one or more processes described herein. Thus, embodiments described herein are not limited to any specific combination of hardware circuitry and software unless explicitly stated otherwise.

[0070] Memory 306 and / or storage component 308 includes data storage or at least one data structure (e.g., a database and / or the like). Device 300 is capable of receiving information from, storing information in, communicating information to, or searching information stored in the data storage or the at least one data structure in memory 306 or storage component 308. In some examples, the information includes network data, input data, output data, or any combination thereof.

[0071] In some embodiments, device 300 is configured to execute software instructions that are either stored in memory 306 and / or in the memory of another device (e.g., another device that is the same as or similar to device 300). As used herein, the term “module” refers to at least one instruction stored in memory 306 and / or in the memory of another device that, when executed by processor 304 and / or by a processor of another device (e.g., another device that is the same as or similar to device 300) cause device 300 (e.g., at least one component of device 300) to perform one or more processes described herein. In some embodiments, a module is implemented in software, firmware, hardware, and / or the like.

[0072] The number and arrangement of components illustrated in FIG. 3 are provided as an example. In some embodiments, device 300 can include additional components, fewer components, different components, or differently arranged components than those illustrated in FIG. 3. Additionally or alternatively, a set of components (e.g., one or more components) of device 300 can perform one or more functions described as being performed by another component or another set of components of device 300.

[0073] Referring now to FIG. 4A, illustrated is an example block diagram of an autonomous vehicle compute 400 (sometimes referred to as an “AV stack”). As illustrated, autonomous vehicle compute 400 includes perception system 402 (sometimes referred to as a perception module), planning system 404 (sometimes referred to as a planning module), localization system 406 (sometimes referred to as a localization module), control system 408 (sometimes referred to as a control module), and database 410. In some embodiments, perception system 402, planning system 404, localization system 406,Attorney Docket No.46154-0555WO1 P2023081 / US / PRV1 control system 408, and database 410 are included and / or implemented in an autonomous navigation system of a vehicle (e.g., autonomous vehicle compute 202f of vehicle 200). Additionally, or alternatively, in some embodiments perception system 402, planning system 404, localization system 406, control system 408, and database 410 are included in one or more standalone systems (e.g., one or more systems that are the same as or similar to autonomous vehicle compute 400 and / or the like). In some examples, perception system 402, planning system 404, localization system 406, control system 408, and database 410 are included in one or more standalone systems that are located in a vehicle and / or at least one remote system as described herein. In some embodiments, any and / or all of the systems included in autonomous vehicle compute 400 are implemented in software (e.g., in software instructions stored in memory), computer hardware (e.g., by microprocessors, microcontrollers, application-specific integrated circuits (ASICs), Field Programmable Gate Arrays (FPGAs), and / or the like), or combinations of computer software and computer hardware. It will also be understood that, in some embodiments, autonomous vehicle compute 400 is configured to be in communication with a remote system (e.g., an autonomous vehicle system that is the same as or similar to remote AV system 114, a fleet management system 116 that is the same as or similar to fleet management system 116, a V2I system that is the same as or similar to V2I system 118, and / or the like).

[0074] In some embodiments, perception system 402 receives data associated with at least one physical object (e.g., data that is used by perception system 402 to detect the at least one physical object) in an environment and classifies the at least one physical object. In some examples, perception system 402 receives image data captured by at least one camera (e.g., cameras 202a), the image associated with (e.g., representing) one or more physical objects within a field of view of the at least one camera. In such an example, perception system 402 classifies at least one physical object based on one or more groupings of physical objects (e.g., bicycles, vehicles, traffic signs, pedestrians, and / or the like). In some embodiments, perception system 402 transmits data associated with the classification of the physical objects to planning system 404 based on perception system 402 classifying the physical objects.Attorney Docket No.46154-0555WO1 P2023081 / US / PRV1

[0075] In some embodiments, planning system 404 receives data associated with a destination and generates data associated with at least one route (e.g., routes 106) along which a vehicle (e.g., vehicles 102) can travel along toward a destination. In some embodiments, planning system 404 periodically or continuously receives data from perception system 402 (e.g., data associated with the classification of physical objects, described above) and planning system 404 updates the at least one trajectory or generates at least one different trajectory based on the data generated by perception system 402. In other words, planning system 404 may perform tactical function-related tasks that are required to operate vehicle 102 in on-road traffic. Tactical efforts involve maneuvering the vehicle in traffic during a trip, including but not limited to deciding whether and when to overtake another vehicle, change lanes, or selecting an appropriate speed, acceleration, deacceleration, etc. In some embodiments, planning system 404 receives data associated with an updated position of a vehicle (e.g., vehicles 102) from localization system 406 and planning system 404 updates the at least one trajectory or generates at least one different trajectory based on the data generated by localization system 406.

[0076] In some embodiments, localization system 406 receives data associated with (e.g., representing) a location of a vehicle (e.g., vehicles 102) in an area. In some examples, localization system 406 receives LiDAR data associated with at least one point cloud generated by at least one LiDAR sensor (e.g., LiDAR sensors 202b). In certain examples, localization system 406 receives data associated with at least one point cloud from multiple LiDAR sensors and localization system 406 generates a combined point cloud based on each of the point clouds. In these examples, localization system 406 compares the at least one point cloud or the combined point cloud to two-dimensional (2D) and / or a three-dimensional (3D) map of the area stored in database 410. Localization system 406 then determines the position of the vehicle in the area based on localization system 406 comparing the at least one point cloud or the combined point cloud to the map. In some embodiments, the map includes a combined point cloud of the area generated prior to navigation of the vehicle. In some embodiments, maps include, without limitation, high-precision maps of the roadway geometric properties, maps describing road network connectivity properties, maps describing roadway physicalAttorney Docket No.46154-0555WO1 P2023081 / US / PRV1 properties (such as traffic speed, traffic volume, the number of vehicular and cyclist traffic lanes, lane width, lane traffic directions, or lane marker types and locations, or combinations thereof), and maps describing the spatial locations of road features such as crosswalks, traffic signs or other travel signals of various types. In some embodiments, the map is generated in real-time based on the data received by the perception system.

[0077] In another example, localization system 406 receives Global Navigation Satellite System (GNSS) data generated by a global positioning system (GPS) receiver. In some examples, localization system 406 receives GNSS data associated with the location of the vehicle in the area and localization system 406 determines a latitude and longitude of the vehicle in the area. In such an example, localization system 406 determines the position of the vehicle in the area based on the latitude and longitude of the vehicle. In some embodiments, localization system 406 generates data associated with the position of the vehicle. In some examples, localization system 406 generates data associated with the position of the vehicle based on localization system 406 determining the position of the vehicle. In such an example, the data associated with the position of the vehicle includes data associated with one or more semantic properties corresponding to the position of the vehicle.

[0078] In some embodiments, control system 408 receives data associated with at least one trajectory from planning system 404 and control system 408 controls operation of the vehicle. In some examples, control system 408 receives data associated with at least one trajectory from planning system 404 and control system 408 controls operation of the vehicle by generating and transmitting control signals to cause a powertrain control system (e.g., DBW system 202h, powertrain control system 204, and / or the like), a steering control system (e.g., steering control system 206), and / or a brake system (e.g., brake system 208) to operate. For example, control system 408 is configured to perform operational functions such as a lateral vehicle motion control or a longitudinal vehicle motion control. The lateral vehicle motion control causes activities necessary for the regulation of the y-axis component of vehicle motion. The longitudinal vehicle motion control causes activities necessary for the regulation of the x-axis component of vehicle motion. In an example, where a trajectory includes a left turn, control system 408 transmits a control signal to cause steering control system 206 to adjust a steering angleAttorney Docket No.46154-0555WO1 P2023081 / US / PRV1 of vehicle 200, thereby causing vehicle 200 to turn left. Additionally, or alternatively, control system 408 generates and transmits control signals to cause other devices (e.g., headlights, turn signal, door locks, windshield wipers, and / or the like) of vehicle 200 to change states.

[0079] In some embodiments, perception system 402, planning system 404, localization system 406, and / or control system 408 implement at least one machine learning model (e.g., at least one multilayer perceptron (MLP), at least one convolutional neural network (CNN), at least one recurrent neural network (RNN), at least one autoencoder, at least one transformer, and / or the like). In some examples, perception system 402, planning system 404, localization system 406, and / or control system 408 implement at least one machine learning model alone or in combination with one or more of the above-noted systems. In some examples, perception system 402, planning system 404, localization system 406, and / or control system 408 implement at least one machine learning model as part of a pipeline (e.g., a pipeline for identifying one or more objects located in an environment and / or the like). An example of an implementation of a machine learning model is included below with respect to FIGS. 4B–4D.

[0080] Database 410 stores data that is transmitted to, received from, and / or updated by perception system 402, planning system 404, localization system 406 and / or control system 408. In some examples, database 410 includes a storage component (e.g., a storage component that is the same as or similar to storage component 308 of FIG. 3) that stores data and / or software related to the operation and uses at least one system of autonomous vehicle compute 400. In some embodiments, database 410 stores data associated with 2D and / or 3D maps of at least one area. In some examples, database 410 stores data associated with 2D and / or 3D maps of a portion of a city, multiple portions of multiple cities, multiple cities, a county, a state, a State (e.g., a country), and / or the like). In such an example, a vehicle (e.g., a vehicle that is the same as or similar to vehicles 102 and / or vehicle 200) can drive along one or more drivable regions (e.g., single-lane roads, multi-lane roads, highways, back roads, off road trails, and / or the like) and cause at least one LiDAR sensor (e.g., a LiDAR sensor that is the same as or similar to LiDAR sensors 202b) to generate data associated with an image representing the objects included in a field of view of the at least one LiDAR sensor.Attorney Docket No.46154-0555WO1 P2023081 / US / PRV1

[0081] In some embodiments, database 410 can be implemented across a plurality of devices. In some examples, database 410 is included in a vehicle (e.g., a vehicle that is the same as or similar to vehicles 102 and / or vehicle 200), an autonomous vehicle system (e.g., an autonomous vehicle system that is the same as or similar to remote AV system 114, a fleet management system (e.g., a fleet management system that is the same as or similar to fleet management system 116 of FIG. 1, a V2I system (e.g., a V2I system that is the same as or similar to V2I system 118 of FIG. 1) and / or the like.

[0082] Referring now to FIG. 4B, illustrated is a diagram of an implementation of a machine learning model. More specifically, illustrated is a diagram of an implementation of a convolutional neural network (CNN) 420. For purposes of illustration, the following description of CNN 420 will be with respect to an implementation of CNN 420 by perception system 402. However, it will be understood that in some examples CNN 420 (e.g., one or more components of CNN 420) is implemented by other systems different from, or in addition to, perception system 402 such as planning system 404, localization system 406, and / or control system 408. While CNN 420 includes certain features as described herein, these features are provided for the purpose of illustration and are not intended to limit the present disclosure.

[0083] CNN 420 includes a plurality of convolution layers including first convolution layer 422, second convolution layer 424, and convolution layer 426. In some embodiments, CNN 420 includes sub-sampling layer 428 (sometimes referred to as a pooling layer). In some embodiments, sub-sampling layer 428 and / or other subsampling layers have a dimension (i.e., an amount of nodes) that is less than a dimension of an upstream system. By virtue of sub-sampling layer 428 having a dimension that is less than a dimension of an upstream layer, CNN 420 consolidates the amount of data associated with the initial input and / or the output of an upstream layer to thereby decrease the amount of computations necessary for CNN 420 to perform downstream convolution operations. Additionally, or alternatively, by virtue of sub-sampling layer 428 being associated with (e.g., configured to perform) at least one subsampling function (as described below with respect to FIGS. 4C and 4D), CNN 420 consolidates the amount of data associated with the initial input.Attorney Docket No.46154-0555WO1 P2023081 / US / PRV1

[0084] Perception system 402 performs convolution operations based on perception system 402 providing respective inputs and / or outputs associated with each of first convolution layer 422, second convolution layer 424, and convolution layer 426 to generate respective outputs. In some examples, perception system 402 implements CNN 420 based on perception system 402 providing data as input to first convolution layer 422, second convolution layer 424, and convolution layer 426. In such an example, perception system 402 provides the data as input to first convolution layer 422, second convolution layer 424, and convolution layer 426 based on perception system 402 receiving data from one or more different systems (e.g., one or more systems of a vehicle that is the same as or similar to vehicle 102), a remote AV system that is the same as or similar to remote AV system 114, a fleet management system that is the same as or similar to fleet management system 116, a V2I system that is the same as or similar to V2I system 118, and / or the like). A detailed description of convolution operations is included below with respect to FIG. 4C.

[0085] In some embodiments, perception system 402 provides data associated with an input (referred to as an initial input) to first convolution layer 422 and perception system 402 generates data associated with an output using first convolution layer 422. In some embodiments, perception system 402 provides an output generated by a convolution layer as input to a different convolution layer. For example, perception system 402 provides the output of first convolution layer 422 as input to sub-sampling layer 428, second convolution layer 424, and / or convolution layer 426. In such an example, first convolution layer 422 is referred to as an upstream layer and sub-sampling layer 428, second convolution layer 424, and / or convolution layer 426 are referred to as downstream layers. Similarly, in some embodiments perception system 402 provides the output of sub-sampling layer 428 to second convolution layer 424 and / or convolution layer 426 and, in this example, sub-sampling layer 428 would be referred to as an upstream layer and second convolution layer 424 and / or convolution layer 426 would be referred to as downstream layers.

[0086] In some embodiments, perception system 402 processes the data associated with the input provided to CNN 420 before perception system 402 provides the input to CNN 420. For example, perception system 402 processes the data associated with theAttorney Docket No.46154-0555WO1 P2023081 / US / PRV1 input provided to CNN 420 based on perception system 402 normalizing sensor data (e.g., image data, LiDAR data, radar data, and / or the like).

[0087] In some embodiments, CNN 420 generates an output based on perception system 402 performing convolution operations associated with each convolution layer. In some examples, CNN 420 generates an output based on perception system 402 performing convolution operations associated with each convolution layer and an initial input. In some embodiments, perception system 402 generates the output and provides the output as fully connected layer 430. In some examples, perception system 402 provides the output of convolution layer 426 as fully connected layer 430, where fully connected layer 430 includes data associated with a plurality of feature values referred to as F1, F2 . . . FN. In this example, the output of convolution layer 426 includes data associated with a plurality of output feature values that represent a prediction.

[0088] In some embodiments, perception system 402 identifies a prediction from among a plurality of predictions based on perception system 402 identifying a feature value that is associated with the highest likelihood of being the correct prediction from among the plurality of predictions. For example, where fully connected layer 430 includes feature values F1, F2, . . . FN, and F1 is the greatest feature value, perception system 402 identifies the prediction associated with F1 as being the correct prediction from among the plurality of predictions. In some embodiments, perception system 402 trains CNN 420 to generate the prediction. In some examples, perception system 402 trains CNN 420 to generate the prediction based on perception system 402 providing training data associated with the prediction to CNN 420.

[0089] Referring now to FIGS. 4C and 4D, illustrated is a diagram of example operation of CNN 440 by perception system 402. In some embodiments, CNN 440 (e.g., one or more components of CNN 440) is the same as, or similar to, CNN 420 (e.g., one or more components of CNN 420) (see FIG. 4B).

[0090] At step 450, perception system 402 provides data associated with an image as input to CNN 440 (step 450). For example, as illustrated, perception system 402 provides the data associated with the image to CNN 440, where the image is a greyscale image represented as values stored in a two-dimensional (2D) array. In some embodiments, the data associated with the image may include data associated with a color image, the colorAttorney Docket No.46154-0555WO1 P2023081 / US / PRV1 image represented as values stored in a three-dimensional (3D) array. Additionally, or alternatively, the data associated with the image may include data associated with an infrared image, a radar image, and / or the like.

[0091] At step 455, CNN 440 performs a first convolution function. For example, CNN 440 performs the first convolution function based on CNN 440 providing the values representing the image as input to one or more neurons (not explicitly illustrated) included in first convolution layer 442. In this example, the values representing the image can correspond to values representing a region of the image (sometimes referred to as a receptive field). In some embodiments, each neuron is associated with a filter (not explicitly illustrated). A filter (sometimes referred to as a kernel) is representable as an array of values that corresponds in size to the values provided as input to the neuron. In one example, a filter may be configured to identify edges (e.g., horizontal lines, vertical lines, straight lines, and / or the like). In successive convolution layers, the filters associated with neurons may be configured to identify successively more complex patterns (e.g., arcs, objects, and / or the like).

[0092] In some embodiments, CNN 440 performs the first convolution function based on CNN 440 multiplying the values provided as input to each of the one or more neurons included in first convolution layer 442 with the values of the filter that corresponds to each of the one or more neurons. For example, CNN 440 can multiply the values provided as input to each of the one or more neurons included in first convolution layer 442 with the values of the filter that corresponds to each of the one or more neurons to generate a single value or an array of values as an output. In some embodiments, the collective output of the neurons of first convolution layer 442 is referred to as a convolved output. In some embodiments, where each neuron has the same filter, the convolved output is referred to as a feature map.

[0093] In some embodiments, CNN 440 provides the outputs of each neuron of first convolutional layer 442 to neurons of a downstream layer. For purposes of clarity, an upstream layer can be a layer that transmits data to a different layer (referred to as a downstream layer). For example, CNN 440 can provide the outputs of each neuron of first convolutional layer 442 to corresponding neurons of a subsampling layer. In an example, CNN 440 provides the outputs of each neuron of first convolutional layer 442 toAttorney Docket No.46154-0555WO1 P2023081 / US / PRV1 corresponding neurons of first subsampling layer 444. In some embodiments, CNN 440 adds a bias value to the aggregates of all the values provided to each neuron of the downstream layer. For example, CNN 440 adds a bias value to the aggregates of all the values provided to each neuron of first subsampling layer 444. In such an example, CNN 440 determines a final value to provide to each neuron of first subsampling layer 444 based on the aggregates of all the values provided to each neuron and an activation function associated with each neuron of first subsampling layer 444.

[0094] At step 460, CNN 440 performs a first subsampling function. For example, CNN 440 can perform a first subsampling function based on CNN 440 providing the values output by first convolution layer 442 to corresponding neurons of first subsampling layer 444. In some embodiments, CNN 440 performs the first subsampling function based on an aggregation function. In an example, CNN 440 performs the first subsampling function based on CNN 440 determining the maximum input among the values provided to a given neuron (referred to as a max pooling function). In another example, CNN 440 performs the first subsampling function based on CNN 440 determining the average input among the values provided to a given neuron (referred to as an average pooling function). In some embodiments, CNN 440 generates an output based on CNN 440 providing the values to each neuron of first subsampling layer 444, the output sometimes referred to as a subsampled convolved output.

[0095] At step 465, CNN 440 performs a second convolution function. In some embodiments, CNN 440 performs the second convolution function in a manner similar to how CNN 440 performed the first convolution function, described above. In some embodiments, CNN 440 performs the second convolution function based on CNN 440 providing the values output by first subsampling layer 444 as input to one or more neurons (not explicitly illustrated) included in second convolution layer 446. In some embodiments, each neuron of second convolution layer 446 is associated with a filter, as described above. The filter(s) associated with second convolution layer 446 may be configured to identify more complex patterns than the filter associated with first convolution layer 442, as described above.

[0096] In some embodiments, CNN 440 performs the second convolution function based on CNN 440 multiplying the values provided as input to each of the one or moreAttorney Docket No.46154-0555WO1 P2023081 / US / PRV1 neurons included in second convolution layer 446 with the values of the filter that corresponds to each of the one or more neurons. For example, CNN 440 can multiply the values provided as input to each of the one or more neurons included in second convolution layer 446 with the values of the filter that corresponds to each of the one or more neurons to generate a single value or an array of values as an output.

[0097] In some embodiments, CNN 440 provides the outputs of each neuron of second convolutional layer 446 to neurons of a downstream layer. For example, CNN 440 can provide the outputs of each neuron of first convolutional layer 442 to corresponding neurons of a subsampling layer. In an example, CNN 440 provides the outputs of each neuron of first convolutional layer 442 to corresponding neurons of second subsampling layer 448. In some embodiments, CNN 440 adds a bias value to the aggregates of all the values provided to each neuron of the downstream layer. For example, CNN 440 adds a bias value to the aggregates of all the values provided to each neuron of second subsampling layer 448. In such an example, CNN 440 determines a final value to provide to each neuron of second subsampling layer 448 based on the aggregates of all the values provided to each neuron and an activation function associated with each neuron of second subsampling layer 448.

[0098] At step 470, CNN 440 performs a second subsampling function. For example, CNN 440 can perform a second subsampling function based on CNN 440 providing the values output by second convolution layer 446 to corresponding neurons of second subsampling layer 448. In some embodiments, CNN 440 performs the second subsampling function based on CNN 440 using an aggregation function. In an example, CNN 440 performs the first subsampling function based on CNN 440 determining the maximum input or an average input among the values provided to a given neuron, as described above. In some embodiments, CNN 440 generates an output based on CNN 440 providing the values to each neuron of second subsampling layer 448.

[0099] At step 475, CNN 440 provides the output of each neuron of second subsampling layer 448 to fully connected layers 449. For example, CNN 440 provides the output of each neuron of second subsampling layer 448 to fully connected layers 449 to cause fully connected layers 449 to generate an output. In some embodiments, fully connected layers 449 are configured to generate an output associated with a predictionAttorney Docket No.46154-0555WO1 P2023081 / US / PRV1 (sometimes referred to as a classification). The prediction may include an indication that an object included in the image provided as input to CNN 440 includes an object, a set of objects, and / or the like. In some embodiments, perception system 402 performs one or more operations and / or provides the data associated with the prediction to a different system, described herein.

[0100] Referring now to FIGS. 5A–5D, illustrated are diagrams of implementations 500A-500D of a process for homotopy generation. In some embodiments, a homotopy is a space in which any path starting with an initial AV state and ending at a goal AV state can be continuously deformed. For example, within a homotopy, a first path can be transformed to a second path of the homotopy without breaks or discontinuities in the path. In examples, a homotopy describes a class of possible paths or trajectories, where two trajectories are considered to be in a same class if one trajectory can be continuously transformed into the other trajectory without intersecting objects in the environment.

[0101] In examples, a homotopy is generated based on, at least in part, a mode of operation associated with the vehicle. The mode of operation is, for example, predefined rules for navigation in unstructured environments (e.g., environments without lane structures or markings). Traditionally, the generation of homotopies based on existing lane structures results in limited maneuvers. For example, some maneuvers are not possible, such as maneuvers in pick up and drop off (PuDo) areas and construction areas, where the lane structure is violated by objects, including external actors and authorities.

[0102] In some embodiments, a separate, independent planning algorithm is implemented, such as a free space planner. In examples, free space planners generate candidate trajectories without fully modeling kinematic constraints or considering smoothness and comfort. In some embodiments, a homotopy is generated that is evaluated (e.g., taken into consideration) in parallel or simultaneously with the other traditionally generated lane centric homotopies. This enables the creation of unstructured maneuvers. As described herein, an unstructured maneuver refers to a particular trajectory or route in an unstructured environment. For example, an unstructured maneuver includes driving between two stanchion signs demarcating the PuDo entrance or driving around a parked vehicle to reach the drop off location or drive in between cones inside of a construction zone. Using the generated homotopies, a safe, smooth andAttorney Docket No.46154-0555WO1 P2023081 / US / PRV1 comfortable ego trajectory is realized using a same trajectory realizer as other structured homotopies.

[0103] Referring now to FIG. 5A, an implementation 500A of a process for homotopy generation is shown. The implementation 500A includes a user device 550A. The user device 550A is, for example, a device such as a client device, a mobile device (e.g., a cellular telephone, a tablet, and / or the like). The user device 550A transmits data 510A associated with a request for services associated with the vehicle, such as autonomous package delivery, robotaxi services, or any combinations thereof. In examples, the data 510A includes a date, time, starting location, ending, location, and user identification. In examples, the data 510A is transmitted to a V2I device, a remote AV system, a fleet management system, and / or V2I system 118 via a network. In examples, the V2I device, remote AV system, fleet management system, V2I system, and network are the same as or similar to the V2I device 110, remote AV system 114, fleet management system 116, V2I system 118, and network 112 shown in FIG. 1.

[0104] In examples, the data 510A is obtained by or transmitted to a vehicle 502A. In examples, the vehicle 502A is the same as or similar to vehicles 102 of FIG. 1 and / or vehicle 200 of FIG. 2. The vehicle 502A includes an AV compute 400A. In examples, the AV compute 400A is the same as or similar to the AV compute of FIG. 4. In some embodiments, the AV compute 400A includes a planning system (e.g., planning system 404 of FIG. 4) that performs tactical function-related tasks to operate vehicle on-road traffic and off-road traffic. Tactical efforts involve maneuvering the vehicle in traffic during a trip, including but not limited to deciding whether and when to overtake another vehicle, change lanes, or selecting an appropriate speed, acceleration, deacceleration, etc.

[0105] In order to generate smooth, comfortable and safe trajectories, an ego vehicle generates a trajectory from a set of constraints, called a homotopy. Traditionally, homotopies are dictated by existing lane structures, and baselines defined by them. In examples, homotopies enable high level maneuvers, such as “stay in a particular lane but stay behind the actor in front of the ego vehicle,” or “overtake the actor in front of the ego vehicle using the left lane.” There are many situations where the ego vehicle cannot rely on homotopy constraints based on lanes and other semantic road structures because they are not applicable, either permanently or temporarily. For example, in constructionAttorney Docket No.46154-0555WO1 P2023081 / US / PRV1 zones vehicles are diverted away from lanes. Also, in environments such PuDo areas, many vehicles are parked and stopped in various areas typically not occupied on public roads. Construction zones, PuDo areas, and other unstructured environments call for maneuvers in space and around vehicles and other roadblocks. In some embodiments, a planning system of the AV compute 400A receives data associated with an updated position of a vehicle (e.g., vehicles 502A) from a localization system (e.g., localization system 406 of FIG. 4) and the planning system updates the at least one trajectory or generates at least one different trajectory based on the data generated by localization system. In examples, a path source obtains a representation of a space based on, at least in part, data 510A from the user device 550A. In examples, homotopies are generated using a path source, such as a free space planning algorithm. The path source provides a baseline (e.g., initial trajectory) for a homotopy generator. The baseline guides the homotopy generation and imposes physical constraints based on features of the environment, such as objects (e.g., objects 104a–104n of FIG.1) including other vehicles, pedestrians and other generic objects. The homotopy generator imposes other road constraints, like valid drivable area and road direction, not considering lane structures. Homotopy generation is further described with respect to FIGS. 6A-16. The homotopy generation described herein improves functionality of the vehicle 502A. Additionally, the routes determined using homotopy generation as described herein enables more efficient navigation of the vehicle 502A.

[0106] Referring now to FIG. 5B, an implementation 500B of a process for homotopy generation is shown. The implementation 500B includes an AV compute 400B of a vehicle 502B. In examples, the AV compute 400B is the same as or similar to the AV compute 400 of FIG.4. A request for transport 512B is sent to a planning system 404B. In examples, the planning system 404B is the same as or similar to the planning system 404 of FIG. 4. In examples, the request for transport 512B is a request for services associated with the vehicle 502B, such as autonomous package delivery, robotaxi services, or any combinations thereof. In examples, the request for transport 512B includes a date, time, starting location, ending, location, and user identification. In examples, the request for transport 512B is obtained from a V2I device, a remote AV system, a fleet management system, and / or V2I system 118 via a network. In examples, the V2I device, remote AVAttorney Docket No.46154-0555WO1 P2023081 / US / PRV1 system, fleet management system, V2I system, and network are the same as or similar to the V2I device 110, remote AV system 114, fleet management system 116, V2I system 118, and network 112 shown in FIG. 1.

[0107] In examples, the request for transport 512B is obtained by or transmitted to a planning system. Similar to the planning system of the AV compute 400A of FIG. 5A, the planning system 404B performs tactical function-related tasks to operate vehicle on-road traffic and off-road traffic. The planning system 404B generates homotopies by relying on a path source, such as a free space planning algorithm with physical constraints imposed based features of the environment, such as objects (e.g., objects 104a–104n of FIG. 1) Similar to the vehicle 502A described in FIG. 5A, the homotopy generation described herein improves functionality of the vehicle 502B. Additionally, the routes determined using the homotopy generation described herein enables more efficient navigation of the vehicle 502B.

[0108] Referring now to FIG. 5C, an implementation 500C of a process for homotopy generation is shown. The implementation 500C includes an AV compute 400C of a vehicle 502C. In examples, the AV compute 400C is the same as or similar to the AV compute 400 of FIG. 4. A route is determined 514C by a planning system 404C. In examples, the planning system 404C is the same as or similar to the planning system 404 of FIG. 4. In examples, the route is determined 514C in response to a request for services associated with the vehicle 502C, such as autonomous package delivery, robotaxi services, or any combinations thereof. In examples, the route is the same as or similar to the routes 106a–106n of FIG. 1. In examples, the route is obtained from a V2I device, a remote AV system, a fleet management system, and / or V2I system 118 via a network. In examples, the V2I device, remote AV system, fleet management system, V2I system, and network are the same as or similar to the V2I device 110, remote AV system 114, fleet management system 116, V2I system 118, and network 112 shown in FIG. 1.

[0109] In examples, the route is generated by a planning system 404C. Similar to the planning systems of the AV compute 400A (FIG.5A) and the AV compute 400B (FIG.5B), the planning system 404C performs tactical function-related tasks to operate vehicle on- road traffic and off-road traffic. A route is transmitted 516C from the planning system to a control system 408C. In examples, the control system 408C is the same as or similar toAttorney Docket No.46154-0555WO1 P2023081 / US / PRV1 the control system 408 of FIG. 4. In examples, the control system 408C receives data associated with at least one route or trajectory from planning system 404C and control system 408C controls operation of the vehicle. The planning system 404C generates homotopies by relying on a path source, such as a free space planning algorithm with physical constraints imposed based on objects (e.g., objects 104a–104n of FIG. 1) in the environment. Similar to the vehicle 502A described in FIG. 5A, the homotopy generation described herein improves functionality of the vehicle 502C. Additionally, the routes determined using homotopy generation described herein enables more efficient navigation of the vehicle 502C.

[0110] Referring now to FIG. 5D, an implementation 500D of a process for homotopy generation is shown. The implementation 500D includes an AV compute 400D of a vehicle 502D. In examples, the AV compute 400D is the same as or similar to the AV compute 400 of FIG. 4. A control signal 518D is generated by a control system 408D. In examples, the control system 408D is the same as or similar to the control system 408 of FIG. 4. In examples, the control system 408D receives data associated with at least one route or trajectory a planning system, such as the planning system 404 of FIG. 4. In examples, the control signal 518D is generated in response to a request for services associated with the vehicle 502D, such as autonomous package delivery, robotaxi services, or any combinations thereof. In examples, control signal 520D is transmitted to a DBW system, such as the DBW system 202h of FIG. 2. In some examples, control system 408D receives data associated with at least one trajectory from a planning system and control system 408D controls operation of the vehicle 502D by generating and transmitting control signals to cause a powertrain control system (e.g., DBW system 202h, powertrain control system 204 of FIG. 2, and / or the like), a steering control system (e.g., steering control system 206 of FIG. 2), and / or a brake system (e.g., brake system 208 of FIG. 2) to operate.

[0111] In examples, the control signal 518D is generated by the control system 408D based on, at least in part, data from a planning system such as the planning systems 400, 400A, 400B, and 400C of FIGs. 4-5C. The control signal 518D is based on homotopies generated by relying on a path source, such as a free space planning algorithm with physical constraints imposed based on objects (e.g., objects 104a–104n of FIG. 1) in theAttorney Docket No.46154-0555WO1 P2023081 / US / PRV1 environment. Similar to the vehicle 502A described in FIG. 5A, the homotopy generation described herein improves functionality of the vehicle 502D. Additionally, the routes determined using homotopy generation described herein enables more efficient navigation of the vehicle 502D.

[0112] Traditional lane centric homotopies are limited to lane geometry, hand- engineered behavioral decisions, and the corresponding limited homotopies. Further, traditional lane centric homotopies do not straightforwardly accommodate machine learning based trajectories, or trajectories by any other source. Accordingly, it is challenging to resolve certain navigation scenarios using traditional lane centric homotopies, such as getting stuck in PuDo areas, aggressive lane changes, etc.

[0113] FIG. 6 shows a system architecture that enables homotopy generation. Data from the environment is captured and referred to as scene 602. Data from the environment is captured by one or more sensors, such as cameras 202a, LiDAR sensors 202b, radar sensors 202c, and microphones 202d of FIG. 2. A motion planner 604 provides an initial trajectory or trajectory sketch 606. In examples, the motion planner 604 is based on machine learning (e.g., behavioral cloning) or sampling (e.g., stochastic search). In examples, the motion planner 604 is a free space planner.

[0114] The initial trajectory or trajectory sketch 606 is provided to an unstructured planner 608. In examples, the unstructured planner 608 obtains the initial trajectory or trajectory sketch 606 and generates a homotopy defined by a set of constraints and generates at least one maneuver 610 that reflects a desired local intent, such as changing lanes or navigating around parked cars. The maneuver is input to an MPC controller 612, where constraints associated with the at least one maneuver are solved as a constraint optimization problem. Solving the constraint optimization problem results in a refined trajectory 614 that is smooth, comfortable, and safe to execute. In examples, the unstructured planner 608 is a versatile wrapper around motion planning algorithms, which can lack safety and comfort guarantees. The present techniques produce trajectories for challenging urban driving scenarios that would be difficult to capture through lane-based approaches. By decoupling global maneuver decisions from local trajectory optimization, the present techniques enable a wide range of planning approaches that create more complex behaviors than traditional approaches constrained by lane geometry. ForAttorney Docket No.46154-0555WO1 P2023081 / US / PRV1 example, the present techniques enable separation between high-level planning (e.g., an intent of the vehicle) and the low-level execution (e.g., how trajectories are generated). This separation improves modularity, scalability, and responsiveness of the vehicle.

[0115] FIG. 7A shows a scene 700A including constraints associated with at least one maneuver. A scene 700A is shown at a series of time stamps t=0s, t=1s, and t=8s. At the first time stamp 710 and ego vehicle is shown in a lane. At the second time stamp 720, a vehicle enters the lane in front of the ego vehicle. As the ego vehicle navigates along the lane, several constraints govern the trajectory traveled by the vehicle. In examples, station constraints refer to specific locations or waypoints that the ego vehicle must observe. In the example of FIG. 7A, a first station constraint directs the ego vehicle to stay ahead of a particular waypoint, and a second station constraint directs the ego vehicle to stay behind of a particular waypoint. A first lateral constraint directs the ego vehicle to stay on a drivable area, and a second lateral constraint directs the ego vehicle to avoid vehicles and other obstacles. A baseline constraint directs the ego vehicle to follow the baseline spatially. A tracking constraint directs the ego vehicle to follow a track temporally. At the third time stamp 730, the vehicle has entered the lane in front of the ego vehicle. The ego vehicle satisfies the constraints as it navigates a trajectory along the lane. As shown in the example of FIG. 7A, the constraints are dynamic and the ego vehicle executes a trajectory derived from, at least in part, the constraints.

[0116] FIG. 8A is a block diagram of motion planner 800A for an AV, according one or more embodiments. For purposes of illustration, the following description of motion planner 800A will be with respect to an implementation of motion planner 800A by a planning system 404. However, it will be understood that in some examples motion planner 800A (e.g., one or more components of motion planner 800A) is implemented by other systems different from, or in addition to, planning system 404 such as perception system 402, localization system 406, and / or control system 408. While motion planner 800A includes certain features as described herein, these features are provided for the purpose of illustration and are not intended to limit the present disclosure.

[0117] In examples, a homotopy is generated corresponding to an unstructured environment, such as a free space or area without traditional lane or road markings. For example, the free space or area is a parking zone, PuDo area, construction zone, or anyAttorney Docket No.46154-0555WO1 P2023081 / US / PRV1 combinations thereof. FIG. 8A shows the generation of trajectories using a trajectory proposal generator 802, trajectory scoring or rulebooks at block 816, and a trajectory tracker 820. Trajectory proposal generator 802 further includes a route planner 808 and an unstructured planner 834. In examples, the trajectory proposal generator 802 obtains as input a route destination 804 and information used for an initial guess source / higher level goals 806. The trajectory proposal generator 802 outputs trajectories 814A, 814B, ..., 814N, 814P (collectively referred to as trajectories 814), where N is the number of homotopies output by the maneuver extraction at block 810 and the trajectory 814P is based on at least in part, a free space planning algorithm. The trajectories 814 are input to trajectory scoring or rulebooks at block 816. In some embodiments, the trajectory scoring or rulebooks at block 816 is used to select one or more trajectories that satisfy a threshold, rule, or other metric. In examples, trajectory scoring compares each respective trajectory with rules in an iterative optimization problem that scores each respective trajectory based on a minimization of one or more cost functions (e.g., collisions, comfort, safety, etc.). In some embodiments, a best trajectory is selected based on comparison of the scores, where in some embodiments the candidate trajectory with the lowest cost score is selected as the best trajectory for the vehicle 822.

[0118] In some embodiments, a score is assigned to each trajectory such that trajectories with the least rule violations and / or less rule violations when compared with other trajectories receives a higher score. The highest scoring trajectory is transmitted to trajectory tracker 860. In some embodiments, a score can be assigned to each trajectory such that trajectories with more rule violations and / or more degree of rule violations receives a higher score, and the lowest scoring trajectory is transmitted to trajectory tracker 820. In examples, trajectory tracker 820 generates control signals for controlling vehicle 822 so that vehicle 822 follows the selected best trajectory. In examples, the vehicle is the same as or similar to the vehicle 102 of FIG. 1.

[0119] As shown in FIG. 8A, a route destination 804 is input to a route planner 808. In examples, the route planner generates a high level route or trajectory that starts at an initial state and ends at a final goal state or goal region. In examples the route planner 808 proposes discrete intents or behavioral decisions (e.g., lane change, overtake, pass, etc.) that are translated into maneuvers or homotopies (sets of constraints on car behavior,Attorney Docket No.46154-0555WO1 P2023081 / US / PRV1 e.g. how far it can deviate from the lane center, where it needs to be at a certain time, etc.) during maneuver extraction at block 810. In examples, maneuver extraction at block 810 is based on behaviors as output by the route planner 808, predictions at block 824, and a changed state and position as captured by the vehicle 822. In examples, predictions at block 824 are predicted positions for surrounding objects as generated by the perception system (e.g., perception 402 of FIG.4). The maneuver extraction at block 810 generates homotopies 811A, 811B, ..., 811N. The homotopies 811A, 811B, ..., 811N are passed to a realizer which converts the homotopies 811A, 811B, ..., 811N to trajectories 814A, 814B, ..., 814N. In the example of FIG.8A, the realizer is based on, at least in part, model predictive control (MPC). Control models 812A, 812B, ... 812N, and 812P are shown. In examples, a control model corresponding to each homotopy solves an optimization problem at each time step to find an optimal trajectory (e.g., a control action such as acceleration, deceleration, braking, steering angle) 814A, 814B, ..., 814N, 814P that drives vehicle 822 from a starting location to an ending location within the respective homotopy. In some embodiments, an alternative trajectory generator 826 generates an alternative trajectory 814X independent of the trajectory proposal generator 802 for backup operations.

[0120] In the example of FIG. 8A, information for an initial guess source / higher level goals (806) is input to an initial guess path / trajectory source at block 830. A local intent is obtained. In examples, local intent refers to the local intent loosely refers to an aim or goal of the vehicle at a high level (e.g. change lanes to the left, bias to the right, slowdown, turn left, overtake, etc.) at a particular point in time. In examples, local intent is implicit in an initial trajectory or a free space trajectory sketch output at block 830. An unstructured planner 834 generates a homotopy based on, at least in part, a free space or unstructured area. In examples, a homotopy (i.e., a set of constraints describing a maneuver) is created directly from the free space trajectory sketch. This homotopy generated from an unstructured environment can then be realized, scored, and evaluated in parallel, simultaneously, with other homotopies. In examples, the free space trajectory sketch is obtained from a variety of sources, such as a search-based planner (e.g., A*), a hand- engineered search-based planner, a machine learning planner (e.g., imitation learning,Attorney Docket No.46154-0555WO1 P2023081 / US / PRV1 Urban Driver), hybrid ML+search (e.g., MCTS), human (e.g., remote vehicle assistance (RVA)), etc.

[0121] As shown in FIG.8A, a homotopy generated from an unstructured environment is evaluated in parallel with homotopies generated by a planning system. The present techniques enable flexible solutions to navigating unstructured areas, areas lacking conventional markings, and areas with unconventional markings. The present techniques getting stuck in PuDo, excessive lane changes, etc. For example, when navigating in unstructured environments (e.g., in PuDo areas), with a traditional lane based planner, double or triple lane changes may be performed to overtake stationary vehicles. With the traditional lane based planner, navigation is uncomfortably limited to returning to lane follow modes after each lane change when overtaking stationary vehicles is unsuccessful. The present techniques enable gradual and safe integration of alternative trajectory sources / planners. For example, by using a trajectory sketch to generate a homotopy, trajectory execution tasks (e.g., from a machine learning (ML) planner at block 830, a trajectory proposing module, etc.) can execute without the burden of ensuring comfort and (to some extent) safety. The trajectory sketch is controller-friendly, in that the coarse trajectory sketch can be based on sparse data. In examples, the generation of homotopies described herein is used to avoid stuck conditions (e.g., unable to navigate) within PuDo areas. In examples, the generation of homotopies as described herein is used to incorporate trajectories from machine learning planners to address challenging scenarios. In examples, the generation of homotopies as described herein is used to generate multiple alternative homotopies (e.g., from multiple Monte Carlo Tree Search (MCTS) trajectories as part of a single search). In examples, the generation of homotopies as described herein is used for remote vehicle assistance.

[0122] As shown in FIG.8A, an initial trajectory or trajectory sketch is obtained at block 830, and a local intent is derived from the initial trajectory or trajectory sketch. In examples, the unstructured planner outputs a homotopy defined by spline constraints that can be used by an MPC model to generate high-quality trajectories. In some embodiments, the an initial trajectory or trajectory sketch is represented mathematically by candidate values for each of a position (e.g., x(t),y(t)), orientation (e.g., θ(t)), velocity (e.g., v(t)), andAttorney Docket No.46154-0555WO1 P2023081 / US / PRV1 acceleration (e.g., a(t)). Using splines, each of these components can be modeled as a piecewise polynomial function over time or arc length.

[0123] FIG. 8B shows the generation of trajectories using a spline homotopy in a workflow 800B. In examples, a spline is a piecewise polynomial function that is used to approximate smooth and feasible paths for an AV to navigate. Using splines, a trajectory is approximated as a series of smooth curves. Predictions 852, map data 854, and an initial trajectory / path source 856 are obtained by a spline constraint extractor 860. In examples, predictions 852 are predicted positions for surrounding objects as generated by the perception system (e.g., perception 402 of FIG. 4). In examples, map data 854 is obtained from a localization system, such as localization system 406 of FIG. 4. The map data 854 is, for example, a two-dimensional (2D) and / or a three-dimensional (3D) map of the environment as stored in database 410 of FIG. 4.

[0124] In examples, a rough trajectory / path 858 is obtained by the spline constraint extractor 860 (e.g., an initial trajectory or trajectory sketch). The spline constraint extractor 860 outputs a homotopy defined by spline constraints. In examples, the homotopy is parameterized over various domains in the form of spline constraints. For example, the spline constraints correspond to parameters from one or more predefined domains. A spline baseline created from the initial trajectory or trajectory sketch based on the spline constraints represents the center of the homotopy. In some embodiments, spline constraints include continuity constraints, station constraints, smoothness constraints, and geometric constraints. Continuity constraints ensure that the path is smooth and continuous. Station constraints define the start and end points of the path. For example, a trajectory starts from a current or previous position of the AV and ends at goal state or goal region, with specific conditions like speed and direction at these points. Smoothness constraints minimize abrupt changes in the trajectory, ensuring a comfortable ride. Geometric constraints ensure that the trajectory adheres to the physical constraints of the environment, such as avoiding obstacles and staying within road boundaries as derived from the initial guess path / trajectory. In examples, a path is planned to avoid collisions with obstacles while following the desired trajectory.

[0125] In the example of FIG. 8B, a spline homotopy 862 is obtained by an MPC realizer 864. In examples, the MPC realizer 864 is the same as or similar to the controlAttorney Docket No.46154-0555WO1 P2023081 / US / PRV1 models 812 of FIG. 8A. A precision trajectory 865 is output by the MPC realizer 864. The precision trajectory 865 is obtained by a trajectory scoring / selection block 868. In examples, trajectory scoring compares each respective trajectory with rules in an iterative optimization problem that scores each respective trajectory based on a minimization of one or more cost functions, and a highest scoring trajectory is selected. In examples, alternative conventional trajectories 867 (e.g., trajectory 814X of FIG. 8A) are also compared with each respective trajectory. The selected trajectory 870 is obtained by the DBW 872. The DBW 872 is configured to generate and / or transmit control signals to operate one or more devices of vehicle 874.

[0126] FIG. 9 shows a step in a planning horizon 900 for ego vehicle 920. For ease of description, station constraints are shown in the example of FIG.9. However, one or more constraints can be applied. Actors 922 and 924 are shown in the example of FIG.9. Actor 922 is associated with predictions 908A, and actor 924 is associated with predictions 908B. A conditional behavior heuristic 910 is shown. The spline tube 902 and the station constraints 904 are both time-varying. In examples, the spline tube is a spatial corridor or envelope around a spline-based trajectory that defines the safe and feasible region the vehicle can travel through. In some embodiments, the spline tube 902 is generated according to spline constraints. Homotopy generation is achieved based on, at least in part, a spline-homotopy. A spline constraint extractor, such as the spline constraint extractor 860 of FIG. 8B computes a spline-homotopy based on an initial guess 906, without using lane geometry. Accordingly, the spline-homotopy is computed using free space planning and semantic reasoning logic. In examples, a spline homotopy extractor has inputs that include prediction data (e.g., predictions 908 of FIG. 9), map data, and an initial guess. In examples, a spline homotopy extractor has outputs that include a homotopy that represent an intent of the initial guess. In examples, prediction data contains tracked objects and the predictions associated with the tracked objects. In examples, map-data enables or is accessed by an AV map application programming interface (API). Additionally, in examples, an initial guess is a path, spline, trajectory, or any combinations thereof. The initial guess is further described with respect to baseline creation in FIG. 11. In examples, the output of the spline constraint extractor is at leastAttorney Docket No.46154-0555WO1 P2023081 / US / PRV1 one unimodal corridor constraint representing the homotopy as convex shapes with smooth edges.

[0127] In examples, spline constraint extraction is performed in at least one mode of operation. In examples, there are three modes for constraint extraction. FIG. 10A shows a first Mode A of operation. A first Mode A comprises a conservative assumption - no prediction. In the first Mode A, worst-case assumptions are made about the behavior of other actors in the environment. Accordingly, assumptions made in the first Mode A are heuristic and very conservative. In examples, the first Mode A is based on a path or trajectory. In this case no predictions are used and conservative assumptions about the allowed proximity to other cars are used in the constraint generation (similar to speed control). Constraints are imposed with an always behind assumption, where the ego vehicle will not try to overtake other cars. This will lead to very slow driving, but it can be a backup solution to avoid being stuck in unstructured areas.

[0128] FIG. 10B shows a second Mode B of operation. A second Mode B comprises an always behind assumption – with prediction. In examples, the second Mode B is based on a path or trajectory. In this case predictions are used for the computation of constraints. However, the station constraint will be conservative in the sense that the ego vehicle will drive behind conflicting tracks.

[0129] FIG. 10C shows a third Mode C of operation. A third Mode C comprises station based extraction – with prediction. In examples, the third Mode C is based on a full trajectory. In this case predictions are used for the computation of spline constraints. The station in the trajectory will indicate whether the ego car will be in front of conflicting tracks or behind.

[0130] FIG. 11 shows an architecture for homotopy generation. In the example of FIG. 11, the homotopy generation includes baseline creation, prediction processing, projection, station constraint computation, lateral tube computation, and post processing. In the example of FIG. 11, the homotopy is a spline-homotopy, or a homotopy defined by spline constraints.

[0131] In baseline creation, inputs are based on, at least in part the mode of operation. For example, for Option 1 (Mode A / B) of baseline creation, an input is a discrete initial guess path. In examples, the format of the input is a vector in a Special Euclidean groupAttorney Docket No.46154-0555WO1 P2023081 / US / PRV1 in 2 dimensions (SE2) poses (heading is ignored in this step). In examples, a vector of SE(2) poses represents a series of positions and orientations in a two dimensional (2D) plane. SE(2) poses combine translations and rotations. Each pose in SE(2) is defined by three parameters: (x, y, θ), where (x) and (y) are the coordinates in the 2D plane, and (θ) is the orientation angle (rotation) around the z-axis. A vector of SE(2) poses is a list or array of these (x, y, θ) triplets. This vector can represent a sequence of positions and orientations that a robot follows over time. In examples, an AV uses the vector of SE(2) poses to define its planned route, so that the poses represent the path. The poses are sorted in respect to the increasing station. Spacing of the poses should be as equidistant as possible. The initial guess should not conflict with static objects and mapped obstacles (this is the responsibility of the initial guess provider).

[0132] In examples, for Option 2 (Mode A / B) of baseline creation, an input is a spline initial guess path. In examples, the format of the input is a TransitionAsSpline class. The input can be provided as a spline. In this case the task of the module is trivial. The initial guess should not conflict with static objects and mapped obstacles (this is the responsibility of the initial guess provider).

[0133] In examples, for Option 3 (Mode A / B / C) of baseline creation, an input is a time- discretized initial guess trajectory. In examples, the format of the input is a vector of SE2 poses. (heading is ignored in this step) The poses represent the trajectory sampled according to the MPC horizon time steps. The initial guess should not conflict with static objects and mapped obstacles (this is the responsibility of the initial guess provider).

[0134] In examples the output of baseline creation is a spline baseline that is a spline curve describing the center of the homotopy. In examples, the format of the output is a TransitionAsSpline class. The spline transition represents a baseline that spans the homotopy state space. The baseline can be in conflict with tracks. It should however not conflict with static objects and mapped obstacles (this is the responsibility of the initial guess provider). A curvilinear state space is used (states are: station along baseline, lateral deviation from baseline, heading relative to baseline).

[0135] The TransitionAsSpline models a spline path by using a spline path function p (x, y). Heading is inferred by looking at the derivatives.Attorney Docket No.46154-0555WO1 P2023081 / US / PRV1

[0136] The fitting method is the same for a path and the trajectory. Appropriate values for spline progress (p) are chosen and a least square spline fit is performed. A TransitionAsSpline object is created that represents the full length of the supplied initial guess path or trajectory. In examples, fitting is performed to create a smooth path from the initial guess, which is a sequence of waypoints. The smooth path is the spline baseline / spline transition / TransitionAsSpline.

[0137] Prediction processing is performed after baseline creation. FIG. 12 shows heuristic interactive prediction rollouts from prediction processing. For example, consider a third Mode C of operation. Inputs to prediction processing include a time-discretized initial guess trajectory and prediction data. In examples, a format of the time-discretized initial guess trajectory is a vector of SE2 poses (heading is ignored in this step). These poses represent the path. The poses are sorted with respect to the increasing station. Spacing of the poses should be as equidistant as possible. In examples, a format of prediction data is TrackedObjects objects. This contains the tracked objects and the predictions.

[0138] In examples, the output of prediction processing is heuristic interactive prediction rollouts. The predictions could be either heuristic or ML-based. A format of the output of prediction processing is a TrackedObjects object. A TrackedObjects object representing the modified rolled out that are trajectories of the subset of relevant tracked objects. Relevant in this case means that they are closer to the baseline than a threshold. A conditional heuristic is used for rollout and constraint generation in order to preserve the cut-off associated with of the upstream algorithm. The general idea is that the prediction (coming from the AV-stack) is directly rolled out as long as the track is not cut- off by the ego trajectory or another affected track. If the track is affected a constant deceleration profile is applied for the track to try to satisfy the no collision requirement. A track is considered to be cut-off if the impeding car is moving in front of it as shown in FIG. 12. This algorithm is performed on the tracks that come closer to the baseline than a certain threshold.

[0139] FIG. 13 shows prediction projection. Inputs to prediction processing include a spline baseline, prediction data, and realizer time steps. In examples, the format of the spline baseline is TransitionAsSpline class. The spline transition represents the baselineAttorney Docket No.46154-0555WO1 P2023081 / US / PRV1 that spans the homotopy state space. In examples, the format of the prediction data is TrackedObjects objects. This contains the tracked objects and the predictions. In examples, the format of the realizer time-steps is a vector of time-steps that indicate the solver horizon time steps.

[0140] In examples, the output of prediction projection is a time-series of sampled points along the convex hull of the predictions projected into spline space (x, y) (p, n). IN examples, the format of the output of prediction projection is std::vector<std::vector<std::vector<std:pair<double, double>>>>. In examples the output of prediction projection includes a semantic data structure. The semantic data structure is interpretable and includes multiple tracks and their possible future trajectories. The semantic data structure can be used to constrain the homotopy, so that conflicts are avoided with the multiple tracks and their possible future trajectories. In examples, the outer vector is the different tracks, the middle vector is the different timesteps, and the inner vector are the sampled points in (p, n) space.

[0141] As shown in FIG. 13, prediction projection goes through predicted tracks and performs two steps. First, prediction projection checks if one of the corners of the convex hull is closer to the baseline than the threshold (green). Second, if it is closer than the threshold a number of samples along the convex hull are translated into spline space: (x, y) (p, n) where p is the spline progress and n is the lateral deviation from the spline baseline.

[0142] FIG. 14A shows station constraint computation. For example, consider a second Mode B of operation. In a second Mode B of operation, an "Always Behind" Station Constraint Computation (Mode B) is shown. An input to the station constraint computation is a time-series of sampled points along the convex hull of the predictions projected into spline space (x, y) (p, n). The format of the input to the station constraint computation is std::vector<std::vector<std::vector<std:pair<double, double>>>>. In examples, the format of the input to the station constraint computation includes a semantic data structure. In examples, the outer vector is the different tracks, the middle vector is the different timesteps, and the inner vector is the sampled points in (p, n) space.

[0143] In examples, the output of the station constraint computation is N x 2-sided station limits, where N is Number of stages in MPC horizon. In examples, the format ofAttorney Docket No.46154-0555WO1 P2023081 / US / PRV1 the output of the station constraint computation is std::vector<Bounds<double>>. The spline progress bounds with respect to the homotopy baseline. The vector entries correspond to the steps in the MPC horizon. As shown in FIG.14A, in the "always behind" station constraint computation (Mode B) if any tracked object at a certain timestep comes closer to the baseline than a certain threshold an "always behind" station constraint will be imposed. The input is already projected into spline and lateral space. Therefore the computation can be done without performing any projections.

[0144] FIG. 14B shows station constraint computation. For example, consider a third Mode C of operation. In a third Mode C of operation, an input to the station constraint computation is a time-series of sampled points along the convex hull of the predictions projected into spline space (x, y) (p, n). In this module they represent the Heuristic interactive Prediction Rollouts. The format of the input to the station constraint computation is std::vector<std::vector<std:: vector<std:pair<double, double>>>>. In examples, the format of the input to the station constraint computation includes a semantic data structure. In examples, the outer vector is the different tracks, the middle vector is the different timesteps, and the inner vector is the sampled points in (p, n) space.

[0145] In examples, the output of the station constraint computation is N x 2-sided spline progress limits, where N is Number of stages in MPC horizon. In examples, the format of the output of the station constraint computation is std::vector<Bounds<double>>. The spline progress bounds with respect to the homotopy baseline. The vector entries correspond to the steps in the MPC horizon. As shown in FIG. 14B, in the Mode C station constraint computation if any tracked object at a certain timestep comes closer to the baseline than a certain threshold a station constraint will be imposed. This can be either a minimum or a maximum constraint, depending on whether the tracked object is further along the baseline or not. As shown in FIG.14B, the input is already projected into spline and lateral space. Therefore the computation can be done without performing projections.

[0146] FIG. 15 shows spline tube computation. Inputs to spline tube computation are a spline baseline and prediction data. In examples, the format of the spline baseline is TransitionAsSpline class. The spline transition represents the baseline that spans the homotopy state space. In examples, the format of the prediction data is TrackedObjects objects. This contains the tracked objects and the predictions.Attorney Docket No.46154-0555WO1 P2023081 / US / PRV1

[0147] In examples, the output of spline tube computation is N x spline tube hard / soft spline tube constraints. In examples, the format of the output of spline tube computation is std::vector<TubeAsSpline>. The hard and soft spline tube that represents the lateral constraints. The spline tube covers the whole length of the baseline. The vector entries correspond to the steps in the MPC horizon. The tracks that are to be imposed as the tube will be used to compute the tube spline. The relevant tracks are the ones that do not have been used to impose any station constraints. The spline tube is fitted with the quadratic programming (QP) fitter. In examples, QP fitting optimizes the placement of the spline’s control points to minimize a specific objective function related to the constraints. The method is very similar to the one employed here (Spline Tube Fit). However, the translation into av-stack contour is skipped. As shown in FIG. 15, the input is already projected into spline and lateral space. Therefore, the computation can be done without performing any projections.

[0148] In examples, post processing is used to generate a homotopy and plan description. Inputs to post processing are spline baseline; N x spline tube hard / soft station constraints, where N is Number of stages in MPC horizon; and N x 2-sided spline progress limits, where N is Number of stages in MPC horizon.

[0149] In examples, the format of the spline baseline is TransitionAsSpline class. The spline transition represents the baseline that spans the homotopy state space. In examples, the format of the N x spline tube hard / soft station constraints is an std::vector<TubeAsSpline>. The hard and soft spline tube that represents the lateral constraints. The vector entries correspond to the steps in the MPC horizon. In examples, the format of the N x 2-sided spline progress limits is std::vector<Bounds<double>>. The spline progress bounds with respect to the homotopy baseline. The vector entries correspond to the steps in the MPC horizon. The output of the post processing is a homotopy.

[0150] In examples, to safely deploy and test planning algorithms in real-world scenarios is extremely challenging. Typically, the procedure is to have a safety operator paying attention as a monitor and overriding any unsafe actions that the autonomous vehicle (AV) may take. In some embodiments, an optimization-based approach is used to ensure the safe deployment of any motion planning algorithm. First, conflict constraintsAttorney Docket No.46154-0555WO1 P2023081 / US / PRV1 are extracted from a path or trajectory. Then, a model predictive control generates control commands satisfying conflict and kino-dynamic constraints while tracking closely the generated trajectory.

[0151] Embodiments of the invention may take a rough trajectory sketch and generate a set of constraints that reflect the desired local intent, such as changing lanes or navigating around parked cars. Solving the resulting constraint optimization problem then results in a refined trajectory that is smooth, comfortable, and safe to execute. Embodiments of the invention serve as a wrapper around powerful motion planning algorithms that lack safety and comfort guarantees, such as planners relying on sampling (e.g., stochastic search) or machine learning (e.g., behavioral cloning). Results demonstrate the ability to produce trajectories for challenging urban driving scenarios that would be difficult to capture through lane-based approaches alone. By decoupling global maneuver decisions from local trajectory optimization, this approach unlocks the potential of a wide range of planning approaches that enable for more complex behaviors than traditional approaches constrained by lane geometry.

[0152] Autonomous driving systems commonly employ model predictive control (MPC) for trajectory planning and control. Within the MPC pipeline, maneuver or trajectory generation provides an initial guess that is optimized by the controller. Planning can be considered the high-level discrete decision making, while control handles low-level trajectory optimization to find a locally optimal solution.

[0153] The complexity of MPC approaches ranges from pure path tracking to more sophisticated contouring methods with full trajectory optimization. However, the interface between the global plan and local controller can be challenging. Some approaches pass constraints from the global plan so the controller can find a globally optimal solution, but this significantly increases planning computation. Typically, maneuver generation relies heavily on lane structures from map data. This enables pre-computation of constraints, and simplifies modeling of maneuvers like lane changes and overtaking. However, lane- based planning struggles with unstructured environments like parking lots or hotel lobbies.

[0154] To address these limitations, the present techniques enable computation of globally feasible constraints for trajectory optimization that is not based on lane structures. By removing reliance on map-defined lanes, more flexible maneuvers can be generatedAttorney Docket No.46154-0555WO1 P2023081 / US / PRV1 while still ensuring plan feasibility. This approach combines the efficiency of separated planning and control with the global optimality of integrated methods. Enabling maneuver generation without lane dependence has the potential to significantly improve autonomous driving robustness and adaptability.

[0155] Referring again to FIG.7A, a number of constraints are shown. The constraints are predominantly characterized by the baseline lateral and longitudinal constraints, which delineate the feasible space of the trajectory across the planning horizon.

[0156] Human-level autonomous driving is an ever shifting goalpost, with planning and decision making (the executive functions that determine driving behavior) posing the greatest challenge. Progress is stifled by the technical difficulty of safely deploying experimental motion planners in naturalistic settings. The present techniques enable an unstructured planner, an optimization-based wrapper that can take a trajectory sketch from an arbitrary motion planner and convert it to a safe and comfortable trajectory that the car can follow. The unstructured planner defines a set of interpretable spatiotemporal constraints a maneuver that preserves the intent of the trajectory sketch, while imposing additional requirements such as kinematic feasibility, staying on the road, and avoiding other vehicles. The resulting constraint optimization problem is solved by an MPC controller to produce a refined trajectory that is smooth, comfortable, and safe to execute. Versatility of an unstructured planner is demonstrated by using it to deploy a ML- based planner and a search-based planner on autonomous vehicles. The systems described herein handle complex urban driving scenarios, including adaptive cruise control, cut-ins, and dense unstructured environments such as pick-up / drop-off areas. This enables quickly deploying and evaluating candidate motion planners in realistic settings, ensuring rapid iteration and accelerating progress towards human-level autonomy.

[0157] Self-driving cars have achieved remarkable progress towards human-level autonomous driving. Elements of self-driving technology already enjoy widespread adoption in consumer vehicle, while robotaxi services in certain locales offer glimpses of a driverless future for all. Much of this success is owed to progress in ML-based perception and prediction, which can attain a human-like understanding of the scene around the vehicle. Yet planning and decision making – the executive cognitive functionsAttorney Docket No.46154-0555WO1 P2023081 / US / PRV1 of the robot that ultimately drive behavior – largely remain unsolved, creating a bottleneck that makes human-level driving an ever elusive goal.

[0158] Classical approaches to the planning problem involve the crafting of hand- engineered rules. While still popular, they are laborious to maintain and generalize poorly to novel environments. In recent years, ML-based approaches which learn the driving rules from data have shown promise in overcoming those challenges. The emergence of motion planning datasets, benchmarks, and simulators has fueled a surge in ML-based candidate solutions to the planning bottleneck, a trend further accelerated by recent advances in generative AI.

[0159] This proliferation of experimental planners holds promise but it also poses challenges. Despite advances in simulation technology, there is still a substantial gap between simulation performance and real-world performance: the sim-to-real gap. Yet deploying experimental planners on real vehicles requires safety and comfort guarantees, which most ML-based approaches – and even most classical approaches – simply lack out of the box. Even ensuring basic kinematic feasibility (that the planned trajectory can be tracked by the downstream controller) is hard to achieve with pure ML. Providing such guarantees requires substantial engineering investment, which may be hard to justify without strong signal that a particular planner is worth deploying. But the sim-to-real gap means that obtaining such signal requires deploying the planner in the first place, creating a circular dependency that slows down R&D. There is thus pressing need for technology that can break this vicious cycle and allow companies and labs to take any experimental planner and rapidly answer the question of: “How will this drive in real life”?

[0160] The unstructured planner is an optimization-based wrapper that can take a rough trajectory sketch from an experimental planner, transform it into a set of interpretable spatiotemporal constraints – a maneuver – which can be solved by a model- predictive controller (MPC) to obtain a safe, comfortable, kinematically feasible trajectory. The initial trajectory sketch captures the high-level behavioral intent of the planner, while the final optimized trajectory captures the low-level sequence of commands that the robot needs to execute. This division of labor mirrors hierarchical motor control in biological brains and enables for robust and safe deployment of a wide range of experimental planners. The unstructured planner is agnostic to the source of the trajectory sketch – itAttorney Docket No.46154-0555WO1 P2023081 / US / PRV1 could be a classical planner, a ML-based planner, a hybrid of the two, or even a hand- drawn trajectory from a human operator. The unstructured planner imposes very few requirements on the trajectory sketch, which can be highly irregular. The unstructured planner is also highly configurable: it can act as a smoother that simply tracks the underlying trajectory, it can impose constraints based on the map, and it can also impose constraints based on other vehicles.

[0161] To demonstrate its wide applicability, the unstructured planner is used to deploy a ML-based planner (UrbanDriver) and a search based classical planner (A* [ref]). Performance is evaluated quantitatively on 1000+ resimulated scenarios in dense urban traffic using a high-fidelity physics simulator to demonstrate how different features of the unstructured planner enhance safety and comfort across both types of planners. In examples, planners deployed on real-world autonomous vehicles enable evaluation of performance qualitatively on scenarios on closed course and on public roads. An unstructured planner + UrbanDriver can safely handle ACC and cut-in scenarios, while the unstructured planner + A* can safely navigate crowded, unstructured urban environments such as the pickup / drop off areas. The performance demonstrates that the unstructured planner improves the safe deployment of different experimental planners by obtaining early indicators of real-world performance. This can drastically shorten the timeline from incubating an idea in the lab to deploying it on the road, enabling rapid R&D and accelerating progress towards human-level autonomous driving.

[0162] The constraints that define the feasible space of trajectories over the planning horizon are described, focusing on both lateral and longitudinal dimensions.

[0163] Baseline Constraints

[0164] The baseline trajectory is modeled as a 2D quartic cardinal B-spline. The spline is defined by a set of control points and corresponding knot vectors. The knot vectors are placed at spline progress values from 0 to N, where N is the total number of control points. The spline basis functions are valid within the domain spanning from 1.5 to N−2.5, ensuring that the spline is smooth and well-defined across this range. This baseline spline serves as the foundational coordinate system for defining subsequent trajectory constraints.

[0165] Lateral ConstraintsAttorney Docket No.46154-0555WO1 P2023081 / US / PRV1

[0166] Lateral constraints are formulated as 1D spline functions extending along the baseline spline, representing the drivable space boundaries. These constraints are segmented into ’hard’ and ’soft’ constraints for both the left and right boundaries of the trajectory. The definitions of these constraints are encapsulated by the following functions,which are indexed by spline progress:^^^^^,ℎ^^^(^),^^^^^, ^^^^(^),^^^^ℎ^, ℎ^^^(^), ^^^^ℎ^, ^^^^(^) (1)

[0167] Each lateral constraint is time-series-based, extending over the discretized planning horizon.

[0168] Station Constraints

[0169] Longitudinal constraints, also referred to as station constraints, are defined as a time series that spans the entire planning horizon. These constraints specify the permissible bounds on the vehicle’s progress along the spline, ensuring that the trajectory stays within the predetermined limits over time.

[0170] Unlike spline progress, which represents the input to the 2D baseline spline, station constraints are based on the actual Cartesian path integral along the spline. This distinction enables more accurate control over the vehicle’s position along the trajectory. To bridge the difference between spline progress and station, lookup tables are employed, facilitating conversions between the two coordinate systems.

[0171] By applying these longitudinal (station) constraints, the trajectory planning process ensures that the autonomous vehicle (AV) adheres to the defined spatial limits at each time step, thereby maintaining safe and feasible motion throughout the planning horizon.

[0172] Maneuver Extraction

[0173] Baseline Creation Process

[0174] The process of creating a baseline involves generating a spline from a sequence of 2D trajectory samples. This process is conducted in two main steps to ensure accurate representation and smoothness of the resulting spline. The first step determines the spline progress. Initially, the progress value for each sample point along the spline, denoted ˆpk, is calculated based on the spacing between consecutive samples. This progress reflects the cumulative distance along the path:Attorney Docket No.46154-0555WO1 P2023081 / US / PRV1 0^^^ = 0^^ (^̂ + ௗ^^௧ ^^^ೖ,^^^ೖషభ)ି^ ௗ^ ^^ℎ^^^^^^ (2)

[0175] Here, ^^^^(^^^^,^^^^ି^) represents the Euclidean distance betweenconsecutive sample points, and ^௧is the target Euclidean distance between control points on the spline. The second step is control point optimization. The control points of the spline are computed using a least squares method. The objective function for this optimization includes a second-order regularization term to minimize the curvature, thereby smoothing the spline: ()é ^ ^^̂^(^^̂)ù

[0176] and−1 2 −1 0 ⋯^ 0 −1 2 1 ⋯

[0177] Thisconstraint that penalizes excessive curvature. The optimization is formulated as: ^^^é ^^^^^ù

[0178] where ^and ^^^^is the regularization coefficient. Solving this equation via least squares provides the control points needed for constructing the desired spline.

[0179] Projection of Cartesian Coordinates onto Spline BaselineAttorney Docket No.46154-0555WO1 P2023081 / US / PRV1

[0180] Trajectory predictions for actors, such as vehicles and Vulnerable Road Users (VRUs), are represented by convex hulls. To account for potential spatial uncertainties, these convex hulls are first augmented with predefined buffers to account for potential spatial uncertainties in the predictions. The resulting modified convex hulls are then sampled to generate a set of Cartesian coordinates, each represented as an(^,^)point.

[0181] These Cartesian coordinates are transformed into a coordinate system aligned with the spline representing the baseline trajectory, referred to herein as ”spline-space”.In this coordinate system, each point is expressed as (^,^) , where ^ denotes theprogress along the spline, and ^ represents the lateral deviation from the spline. This transformation facilitates a more precise analysis of the spatial relationship between the sampled points and the baseline trajectory.

[0182] Projection of Cartesian Coordinates into Spline-Space and Sampling Process

[0183] The perception and prediction modules provide time-series data for the predicted positions of surrounding entities such as vehicles and pedestrians. These entities are encapsulated within spatial hulls, which are then sampled at discrete points to facilitate further processing.

[0184] The sampled points are projected into a spline-based coordinate system termed ”spline-space,” which is delineated by coordinates(^,^). Here, ^ represents the progress along the spline, and ^ signifies the lateral deviation from the spline’s baseline.

[0185] The projection process is done iteratively by approximating the spline locally as a circle and the finding the closest point of the circle to the query point:Attorney Docket No.46154-0555WO1 P2023081 / US / PRV1

[0186] Spline Tube Fitting

[0187] Fitting the spline tubes involves categorizing the sampled points based on their relationship with the baseline, determining whether they should be constrained by station or tube parameters. The decision process employs the following criteria:

[0188] - If the closest point to the baseline of a tracked object exceeds a specified threshold distance, samples from that object are subjected to the lateral tube constraints.

[0189] - If the closest point is within the threshold, there are two options: (1) represent every sample of the object by station constraints; or (2) apply station constraints to samples within the threshold and tube constraints to those beyond. The option that enables the most space within the constraints is selected via a heuristic approach.

[0190] The fitting of the spline tubes to the samples is conducted using a QuadraticProgramming (QP) solver, which addresses the following constraints:^ ^^^^^,ℎ^^^(^^) < ^^ , ^^^^ = 1 …^,^^^^ℎ^,ℎ^^^(^^) < ^^ , ^^^^ = 1 …^.

[0191] Here, (^^,^^) are the projected samples along the spline. The corresponding constraint is similarly applied to the right side but with the inequality reversed.

[0192] The primary objective of the QP solver is to maximize the available space within the tube. This goal is facilitated by the linear relationship between the spline function and its control points. Additionally, the cost function incorporates a regularization matrix R, which governs the rate of change of the spline. This regularization is represented by: −1 1 0 0 ⋯⋯

[0193] This matrixa consistent gradient across the spline, aiding in the stabilization of the fitted trajectory. This approach ensures that the spline tubes accurately reflect the spatial constraints imposed by the surrounding environment. In general, a higher regularization weight leads to less curvature.

[0194] MPC Formulation

[0195] Consider the following state vector and decision variables:

[0196] State VariablesAttorney Docket No.46154-0555WO1 P2023081 / US / PRV1

[0197] p(t) Progress along the predefined path (spline

[0198] progress, not path integral)

[0199] n(t) Lateral error from the desired path

[0200] ω(t) Local heading relative to the baseline path

[0201] v(t) System velocity

[0202] a(t) System acceleration

[0203] β(t) Steering angle

[0204] Decision Variables

[0205] j(t) Jerk (rate of acceleration change)

[0206] Δβ(t) Change in steering angle

[0207] The MPC problem with a non-linear cost function can be formulated as follows: ேି^^ ^ ℓ ൫^(^ = ^),^(^ + ^)൯ + ℓ^(^(^ + ^))

[0208] subject to:^(^ + 1) = ^(^(^),^(^))^௫ ≤ 0^^^^ ≤ ^(^) ≤ ^^^௫

[0209] Where:

[0210] • ℓ(x(t+k), u(t+k)) is the non-linear stage cost function.

[0211] • ℓf (x(t + N)) is the non-linear terminal cost function.

[0212] • N is the prediction horizon.

[0213] • f(x(t), u(t)) represents the state transition model— in

[0214] this case, a kinematic bicycle model.

[0215] • C(x) represents the non-linear state constraints.

[0216] • umin, umax are the control constraints.

[0217] • x0 is the initial state.Attorney Docket No.46154-0555WO1 P2023081 / US / PRV1

[0218] The non-linear constraint function C(x) includes terms that ensure the footprint of the ego vehicle remains within the spline constraints throughout the planning horizon. é0.5 ⋅ ^ + ^^ ⋅ sin(^) − ^^^^^(^ + ^^ ⋅ cos(^))ê0.5 ⋅ ^ + ^^ ⋅ ^^^(^) + ^^^^ℎ^(^ + ^^ ⋅ ^^^(^))ùú (7)

[0219] are various objectives, such as maintaining comfort, ensuring the autonomous vehicle (AV) stays within its operational boundaries, and minimizing perceived risks.

[0220] Imposition of Station Constraints

[0221] The method of imposing station constraints depends on the desired behavior of the AV, specifically whether it should always yield or consider overtaking other actors. Desired behaviors or intents include, but are not limited to, the following:

[0222] Always Yielding: If the AV is required to always yield, the station constraints are selected so that the current position of the AV is always included within the station constraints. This approach ensures that the AV remains safely behind other actors.

[0223] Overtaking or Cutting Off: If the AV is considering overtaking or cutting off another actor, the station constraints are chosen to encompass the projected time- series trajectory of the AV. This enables the AV to potentially move ahead of other actors when it is safe and feasible to do so.

[0224] This approach to station constraints enables the AV to adapt its behavior according to the situation, either by yielding or by executing a more assertive maneuver when appropriate.

[0225] Experiments

[0226] To illustrate the improvements, utility, and applicability of the unstructured planner, performance is evaluated with both ML-based and search-based experimental motion planners. Evaluation scenarios, metrics, different unstructured planner ablations used in the comparisons, and results from simulations and real-world driving are described.Attorney Docket No.46154-0555WO1 P2023081 / US / PRV1

[0227] Motion Planners

[0228] Performance is evaluated unstructured planner together with two motion planners that cover two main types of approaches used in the literature and industry: ML and search based planners.

[0229] In examples, the ML based planner is a regression based ML planner. For example, Urban Driver is an a ML-based motion planner which uses regression to learn to imitate an expert driver using closed-loop rollouts. An open-source version of Urban Driver is trained on the nuPlan dataset, and is adapted by restricting the training set to lane follow and adaptive cruise control (ACC) scenarios. The input to the model is a vectorized map and a vectorized representation of the other actors in the scene (vehicles, pedestrians, bicyclists, etc.) relative to the ego. The output was a sequence of 16(^,^)-waypoints corresponding to an 8-s open-loop ego trajectory at 2 Hz. The model is trained using multistep training: for each scenario, and the model executes for 13 s at 2 Hz and supervised the output open-loop trajectory at each time step with the corresponding ground truth expert trajectory using an L2 loss.

[0230] In examples, the search-based Planner is A*: A* is a classical graph search algorithm which is adapted for freespace motion planning. Vertices are defined by adiscrete space of (^, ^,^) poses and edges were defined by feasible transitionsbetween poses, with weights representing the relative cost of motion. The heuristic is an (approximate) lower bound of the cost-to-go to the target. The input to the planner was a starting pose, a target pose, and an occupancy grid with the drivable area and thestatic actors. The output was a sequence of (^, ^,^) -waypoints connecting the startingpose to the target pose. The waypoints did not have corresponding timestamps, so unlike Urban Driver, the output corresponded to a purely spatial path with no temporal dimension. This planner is designed for freespace planning in unstructured environments (e.g., parking lots) and ignored dynamic actors, instead relying on an unstructured planner for active collision avoidance.

[0231] Perception, Prediction, and Mapping

[0232] For simulated scenarios, object-oriented representations of the scene from the simulator were fed to the motion planner and to a prediction module, which generated unimodal 8-s predictions for each actor in the scene at each iteration of theAttorney Docket No.46154-0555WO1 P2023081 / US / PRV1 simulation. These predictions are used by the unstructured planner to compute lateral and station constraints for each open-loop time step of the maneuver. For real-world driving, a perception system (e.g.,perception 402) is used to recreate an object-oriented representation of the scene from camera and lidar sensor data. Perception ran at 20 Hz, while prediction ran at 10 Hz, at the same rate as planning. High-definition maps are available as input.

[0233] Scenarios

[0234] The unstructured planner is evaluated using an Object Sim simulator (formerly Simian) on 1000+ closed loop resim scenarios based on real-world drive logs generated by AVs in Las Vegas. Each scenario consists of a 30-s snippet from the drive log down sampled to 10 Hz. The first 3 s of each scenario consists of a “warm-up” period during which the system was run in open loop, but its inputs and outputs were clamped to (i.e., replayed from) the log. For the remaining 27 s, the system was run in closed loop at 10 Hz. The open-loop trajectory from each iteration is tracked by a VDS controller, which was designed to mimic the dynamics of the real AV. Other actors are replayed from the log, i.e. they were nonreactive to any deviations of ego behavior resulting from the closed-loop simulation.

[0235] Given the complementary strengths of the two motion planners, the performance is evaluated on different sets of scenarios nominal scenarios and freespace scenarios. First, nominal scenarios as used to evaluate Urban Driver and an unstructured planner on 638 nominal lane follow and ACC scenarios on the Las Vegas Strip similar to those from the nuPlan training set. This illustrates how the unstructured planner can act as a rule-based safety wrapper and smoother for a pre-trained but ML planner. Second, freespace scenarios are used to evaluate A* and the unstructured planner on 413 scenarios from PuDo areas of multiple casinos off the Las Vegas Strip. The unstructured PuDo environments include, for example, irregularly parked cars, slow-moving traffic, pedestrians jaywalking and getting on and off of vehicles. This demonstrates how the unstructured planner can be used in conjunction with a purely spatial planner to enable progress while ensuring safety in unstructured environments.

[0236] Model Predictive Controller (MPC)Attorney Docket No.46154-0555WO1 P2023081 / US / PRV1

[0237] The maneuver generated by the unstructured planner is converted into a smooth, kinematically feasible trajectory by satisfying the constraints using an MPC.

[0238] Metrics

[0239] To illustrate how the unstructured planner impacts safety, comfort, and progress, the following metrics are calculated for each scenario:

[0240] • Front conflicts: instances when the ego vehicle has a conflict event with a vehicle in front of it.

[0241] • Conflict-preventative takeovers (CPTOs): instances when the ego vehicle produced unsafe behavior that would have triggered a takeover by the safety driver in order to avoid a conflict event. This includes collisions, drivable area violation, nearest actor violation (min. clearance < 0.15 m to any bicycle / vehicle / obstacle), or proximity speed violation (min. clearance < 0.25 m and ego max. speed > 5.0 m / s for vehicles or > 0.0 m / s for bicycles / pedestrians).

[0242] • Drivable area violations: instances when any point of the ego footprint was outside the drivable area.

[0243] Longitudinal acceleration violations: instances when the (absolute) longitudinal acceleration was above a certain threshold. When decelerating, the threshold was 2.5 m / s2 for speeds < 10 m / s and 1.5 m / s2 for speeds > 20 m / s, with linear interpolation for in-between speeds. When accelerating, the threshold is 2.0 m / s2 for speeds < 10 m / s and 1.0 m / s2 for speeds > 15 m / s, with linear interpolation for in- between speeds.

[0244] Distance traveled: total distance traveled by the ego for the entire duration of the scenario.

[0245] Metrics are calculated separately for each scenario based on the closed-loop trajectory of the ego and the replayed trajectories of the other actors along the entire duration of the scenario. Each metric is aggregated across scenarios by averaging for each condition. Metrics are compared between conditions using the Wilcoxon signed- rank test, a nonparametric test for matched samples (e.g., the scenarios). A nonparametric test is used over the standard paired t-test or repeated measures ANOVA due to the highly non-Gaussian distribution of the metrics. For plotting, 95% bootstrap confidence intervals with 1000 bootstraps are calculated.Attorney Docket No.46154-0555WO1 P2023081 / US / PRV1

[0246] Ablations

[0247] To demonstrate the relative importance of different unstructured planner components, different ablated versions (conditions) of the unstructured planner (Fig. 7B) are evaluated. They are presented here in order from simplest to most complex, with each subsequent condition enabling a superset of the components enabled in the previous condition:

[0248] • Baseline: The baseline spline is fitted to the initial trajectory sketch. No lateral or station constraints were imposed based on the map or other actors in the scene. No tracking costs were imposed, and the ego accelerates to the speed limit. In this condition, the system simply generates kinematically feasible trajectories that follow the spatial (but not the temporal) intent of the initial trajectory sketch.

[0249] • Tracking: In addition to fitting the baseline (as in the baseline condition), longitudinal tracking costs were imposed at waypoint projections onto the baseline. These encouraged the ego to reach the corresponding speed derived from the initial directory sketch at each corresponding waypoint. No lateral or stationed constraints were imposed. In this condition, the system tracks the intent of the initial trajectory sketch both spatially and temporally.

[0250] • Map: In addition to fitting the baseline and tracking the waypoints (as in the Tracking condition), constraints were imposed based on the map. Specifically, the lateral tube was restricted to the boundaries of the drivable area. If the width of the lateral tube shrunk below a certain threshold (2 m), indicating that the ego will surely violate the lateral bounds and drive off the road, a station constraint was imposed. In this condition, the system tracks the intent of the initial trajectory sketch, while remaining on the drivable area.

[0251] • Pass-after: In addition to fitting the baseline, tracking the waypoints, and staying on the drivable area (as in the Map condition), constraints were imposed based on the other actors in the scene. Specifically, lateral constraints were imposed on actors within 4 m of the baseline, to ensure the ego circumvents them. If an actor. was too close to the baseline (< 2 m), indicating that driving around them is unsafe or infeasible, a stay-behind station constraint was imposed. In this condition, the ego was constrainedAttorney Docket No.46154-0555WO1 P2023081 / US / PRV1 to always stay behind other actors crossing its path and pass after they have moved out of the way.

[0252] Full: The full unstructured planner included components from the pass-after condition, namely fitting the baseline, tracking the waypoints, staying on the drivable area, and constraints based on the other actors. However, the station constraints were computed based on where the ego was expected to be along the baseline, according to the waypoint projections. Specifically, if the ego was expected to be behind an actor at time t, a stay-behind station constraint was imposed to ensure that ego indeed remains behind at time t, as in the pass-after condition. However, if the ego was expected to be in front of the actor at time t, a stay-ahead station constraint was imposed instead, ensuring that ego passes the actor by time t. Thus, the ego could either pass after or pass before other actors crossing its path, enabling for a diverse set of behaviors, including lane merging and cut-ins.

[0253] Urban Driver is evaluated in multiple conditions. For A*, the Baseline and Pass-after conditions are used. Since the A* implementation produces a path rather than a trajectory, there are no timestamps associated with the waypoints, so tracking and stay-ahead constraints could not be computed. Additionally, since the occupancy grid a priori constrains the search to only consider poses on the drivable area, the map condition produced identical performance to the Baseline condition.

[0254] Simulation Results

[0255] Attorney Docket No.46154-0555WO1 P2023081 / US / PRV1

[0256] ominal scenarios are shown in FIG. 7C and Table I. In terms of safety, more complex versions of unstructured planner performed better, with the baseline condition showing the most CPTOs, front collisions, and drivable area violations, and unstructured planner showing the least. Planned pairwise comparisons revealed that additional unstructured planner components consistently produced significant safety improvements.

[0257] In terms of comfort, including tracking and the map improved longitudinal acceleration violations. However, adding station constraints in the pass-after and full conditions reversed this trend, likely due to extra braking for the other actors. In terms of progress, more complex versions of unstructured planner resulted in less total distance travelled. This was expected, as introducing more constraints makes it more likely that the ego will slow down, stop, and / or remain stationary. Several example scenarios are shown that illustrate the key differences between conditions in terms of maneuvers and closed-loop ego behavior.

[0258] Results from A* and unstructured planner on 413 PuDo scenarios are shown in FIG. 7D and Table II. As with Urban Driver, the more complex Pass-after condition significantly improved safety compared to the Baseline. Interestingly, comfort was unaffected, while progress was worse, as expected.

[0259] Real-world Driving

[0260] Referring now to FIG. 16, illustrated is a flowchart of a process 1600 for homotopy generation. In some embodiments, one or more of the steps described with respect to process 1600 are performed (e.g., completely, partially, and / or the like) by autonomous system 202 of FIG. 2. Additionally, or alternatively, in some embodiments one or more steps described with respect to process 1600 are performed (e.g., completely, partially, and / or the like) by another device or group of devices separateAttorney Docket No.46154-0555WO1 P2023081 / US / PRV1 from or including autonomous system 202 of FIG. 2, such as device 300 of FIG. 3. Additionally, or alternatively, in some embodiments, one or more of the steps described with respect to process 1600 are performed (e.g., completely, partially, and / or the like) by an autonomous vehicle compute 400 of FIG. 4A, an implementation 500A of FIG. 5A, an implementation 500B of FIG. 5B, an implementation 500C of FIG. 5C, or an implementation 500D of FIG. 5D. Additionally, or alternatively, in some embodiments, one or more of the steps described with respect to process 1600 are performed (e.g., completely, partially, and / or the like) by a trajectory proposal generator 802 of FIG. 8A or a workflow 800B of FIG. 8B.

[0261] At block 1602, at least one initial trajectory is obtained. In examples, the initial trajectory or free space trajectory sketch is obtained from a free space planner that generates candidate trajectories without fully modeling kinematic constraints or considering smoothness and comfort. In examples, the initial trajectory is obtained from a search-based planner (e.g., A*), a hand-engineered search-based planner, a machine learning planner (e.g., imitation learning, Urban Driver), hybrid ML+search (e.g., MCTS), human (e.g., remote vehicle assistance (RVA)), etc. as described at block 806 of FIG. 8A.

[0262] At block 1604, a spline baseline is created based on the at least one free space trajectory from one or more spline constraints.

[0263] At block 1606, station constraints are computed in view of at least one mode of operation.

[0264] At block 1608, a homotopy is generated based on, at least in part, the station constraints and the spline constraints.

[0265] Embodiments

[0266] According to some non-limiting embodiments or examples, provided is method. The method includes obtaining at an initial trajectory from a free-space planner; creating a spline baseline that corresponds to spline constraints extracted from the initial trajectory in at least one mode of operation, wherein station constraints are based on the at least one mode of operation; generating a homotopy based on, at least in part, the spline constraints; generating a plurality of trajectories based on respective homotopies that include the homotopy generated based on the spline constraints; andAttorney Docket No.46154-0555WO1 P2023081 / US / PRV1 navigating a trajectory selected from the plurality of trajectories according to at least one metric.

[0267] According to some non-limiting embodiments or examples, provided is a system. The system includes a trajectory source configured to generate at least one first trajectory in an unstructured environment, wherein at least one homotopy is generated from the at least one first trajectory; a motion planner configured to generate at least one route, wherein maneuvers are extracted from at least one second trajectory; a model predictive controller configured to generate trajectories from the extracted maneuvers and maneuvers generated from the at least one first trajectory; and a drive-by-wire system configured to operate a vehicle according to at least one trajectory output by the model predictive controller.

[0268] Further non-limiting aspects or embodiments are set forth in the following numbered embodiments:

[0269] Embodiment 1: A method, including: obtaining at an initial trajectory from a free- space planner; creating a spline baseline that corresponds to spline constraints extracted from the initial trajectory in at least one mode of operation, where station constraints are based on the at least one mode of operation; generating a homotopy based on, at least in part, the spline constraints; generating a plurality of trajectories based on respective homotopies that include the homotopy generated based on the spline constraints; and navigating a trajectory selected from the plurality of trajectories according to at least one metric.

[0270] Embodiment 2: The method of any preceding embodiment, where the spline constraints include continuity constraints, station constraints, smoothness constraints, geometric constraints, or any combinations thereof.

[0271] Embodiment 3: The method of any preceding embodiment, where the free space planner is a machine learning planner, search-based planner, or a hybrid planner.

[0272] Embodiment 4: The method of any preceding embodiment, where the spline constraints correspond to parameters from one or more predefined domains.

[0273] Embodiment 5: The method of any preceding embodiment, where a model predictive control (MPC) based controller generates the plurality of trajectories based on respective homotopies.Attorney Docket No.46154-0555WO1 P2023081 / US / PRV1

[0274] Embodiment 6: A system, including: a trajectory source configured to generate at least one first trajectory in an unstructured environment, where at least one homotopy is generated from the at least one first trajectory; a motion planner configured to generate at least one route, where maneuvers are extracted from at least one second trajectory; a model predictive controller configured to generate trajectories from the extracted maneuvers and maneuvers generated from the at least one first trajectory; and a drive- by-wire system configured to operate a vehicle according to at least one trajectory output by the model predictive controller.

[0275] Embodiment 7: The system of any preceding embodiment, where the maneuvers are generated from the at least one first trajectory using spline constraints that include continuity constraints, station constraints, smoothness constraints, geometric constraints, or any combinations thereof.

[0276] Embodiment 8: The system of any preceding embodiment, where the trajectory source is a machine learning based free space planner, search-based planner, or a hybrid planner.

[0277] In the foregoing description, aspects and embodiments of the present disclosure have been described with reference to numerous specific details that can vary from implementation to implementation. Accordingly, the description and drawings are to be regarded in an illustrative rather than a restrictive sense. The sole and exclusive indicator of the scope of the invention, and what is intended by the applicants to be the scope of the invention, is the literal and equivalent scope of the set of claims that issue from this application, in the specific form in which such claims issue, including any subsequent correction. Any definitions expressly set forth herein for terms contained in such claims shall govern the meaning of such terms as used in the claims. In addition, when we use the term “further comprising,” in the foregoing description or following claims, what follows this phrase can be an additional step or entity, or a sub- step / sub-entity of a previously recited step or entity.

Claims

Attorney Docket No.46154-0555WO1 P2023081 / US / PRV1 WHAT IS CLAIMED IS:

1. A method, comprising: obtaining at an initial trajectory from a free-space planner; creating a spline baseline that corresponds to spline constraints extracted from the initial trajectory in at least one mode of operation, wherein station constraints are based on the at least one mode of operation; generating a homotopy based on, at least in part, the spline constraints; generating a plurality of trajectories based on respective homotopies that include the homotopy generated based on the spline constraints; and navigating a trajectory selected from the plurality of trajectories according to at least one metric.

2. The method of claim 1, wherein the spline constraints include continuity constraints, station constraints, smoothness constraints, geometric constraints, or any combinations thereof.

3. The method of claim 1, wherein the free space planner is a machine learning planner, search-based planner, or a hybrid planner.

4. The method of claim 1, wherein the spline constraints correspond to parameters from one or more predefined domains.

5. The method of claim 1, wherein a model predictive control (MPC) based controller generates the plurality of trajectories based on respective homotopies.

6. A system, comprising: a trajectory source configured to generate at least one first trajectory in an unstructured environment, wherein at least one homotopy is generated from the at least one first trajectory;Attorney Docket No.46154-0555WO1 P2023081 / US / PRV1 a motion planner configured to generate at least one route, wherein maneuvers are extracted from at least one second trajectory; a model predictive controller configured to generate trajectories from the extracted maneuvers and maneuvers generated from the at least one first trajectory; and a drive-by-wire system configured to operate a vehicle according to at least one trajectory output by the model predictive controller.

7. The system of claim 6, wherein the maneuvers are generated from the at least one first trajectory using spline constraints that include continuity constraints, station constraints, smoothness constraints, geometric constraints, or any combinations thereof.

8. The system of claim 6, wherein the trajectory source is a machine learning based free space planner, search-based planner, or a hybrid planner.

Images