Method and system for obstacle representation

Abstract

A method for obstacle representation is provided, which may include: obtaining sensor data; determining a dynamic state associated with the agent; generating obstacle data; and generating constraints based on the obstacle data. Some described methods also include providing data to cause operation of the autonomous vehicle. A system and a computer program product are also provided.

Description

[0001] This application claims the benefit of priority to U.S. Patent Provisional Application No. 63 / 396,233, filed on August 9, 2022, entitled METHODS AND SYSTEMS FOR OBSTACLEREPRESENTATION, which is incorporated herein by reference in its entirety. BRIEF DESCRIPTION OF THE DRAWINGS

[0002] Figure 1 is an example environment in which a vehicle including one or more components of an autonomous system may be implemented;

[0003] Figure 2 is a diagram of one or more example systems of a vehicle including an autonomous system;

[0004] Figure 3 yes Figure 1 and Figure 2 diagrams of components of one or more example devices and / or one or more example systems;

[0005] Figure 4 is a diagram of certain components of an example autonomous system;

[0006] Figure 5 is a diagram of an example implementation of a process for obstacle representation;

[0007] FIG. 6A to FIG. 6B is a diagram of an example implementation of a process for obstacle representation;

[0008] Figure 7 is a diagram illustrating example projections of obstacles according to one or more embodiments of the present disclosure;

[0009] FIG. 8A to FIG. 8C is a diagram illustrating an example generation of a station constraint according to one or more embodiments of the present disclosure;

[0010] Fig. 9 is a diagram illustrating example generation of lateral constraints according to one or more embodiments of the present disclosure;

[0011] Fig.10 is a flow chart of an example process for obstacle representation; and

[0012] Fig.11 An example station-lateral constraint and station-time constraint analysis based on sensor data associated with an agent moving toward a road along which an AV is moving is illustrated. DETAILED DESCRIPTION

[0013] In the following description, for the purpose of explanation, many specific details are set forth in order to provide a thorough understanding of the present disclosure. However, it will be apparent that the embodiments described in the present disclosure can be implemented without these specific details. In some instances, well-known configurations and devices are illustrated in block diagram form to avoid unnecessarily obscuring aspects of the present disclosure.

[0014] In the accompanying drawings, for ease of description, the specific arrangement or order of schematic elements (such as those representing systems, devices, modules, instruction blocks and / or data elements, etc.) is illustrated. However, those skilled in the art will understand that, unless explicitly described, the specific order or arrangement of schematic elements in the accompanying drawings is not intended to mean that a specific processing order or sequence, or separation of processing is required. In addition, unless explicitly described, the inclusion of schematic elements in the accompanying drawings is not intended to mean that such elements are required in all embodiments, nor is it intended to mean that the features represented by such elements cannot be included in some embodiments or cannot be combined with other elements in some embodiments.

[0015] In addition, in the accompanying drawings, connecting elements (such as solid or dotted lines or arrows, etc.) are used to illustrate the connection, relationship or association between or among two or more other schematic elements, and the absence of any such connecting elements is not intended to mean that there can be no connection, relationship or association. In other words, some connections, relationships or associations between elements are not illustrated in the accompanying drawings so as not to obscure the present disclosure. In addition, for ease of illustration, a single connecting element can be used to represent multiple connections, relationships or associations between elements. For example, if the connecting element represents the communication of a signal, data or instruction (e.g., "software instruction"), it will be understood by those skilled in the art that such an element can represent one or more than one signal path (e.g., bus) that may be needed to affect the communication.

[0016] Although the terms "first", "second" and / or "third", etc. are used to describe various elements, these elements should not be limited by these terms. The terms "first", "second" and / or "third" are used to distinguish one element from another. For example, a first contact may be referred to as a second contact, and similarly, a second contact may be referred to as a first contact without departing from the scope of the described embodiments. Both the first contact and the second contact are contacts, but they are not the same contacts.

[0017] The terms used in the specification of the various embodiments described herein are included only for the purpose of describing specific embodiments and are not intended to be limiting. As used in the specification of the various embodiments described and the appended claims, the singular forms "a", "an" and "the" are also intended to include plural forms and can be used interchangeably with "one or more than one" or "at least one", unless the context clearly indicates otherwise. It will also be understood that the term "and / or" as used herein refers to and includes any and all possible combinations of one or more than one of the associated listed items. It will also be understood that when the terms "include", "comprise", "have" and / or "have" are used in this specification, the stated features, integers, steps, operations, elements and / or components are specifically stated, but the presence or addition of one or more than one other features, integers, steps, operations, elements, components and / or groups thereof are not excluded.

[0018] As used herein, the terms "communication" and "communicating" refer to at least one of receiving, receiving, transmitting, transmitting and / or providing information (or information represented by, for example, data, signals, messages, instructions and / or commands, etc.). For a unit (e.g., a device, a system, a component of a device or system, and / or a combination thereof) to communicate with another unit, this means that the unit is able to directly or indirectly receive information from the other unit and / or send (e.g., transmit) information to the other unit. This can refer to a direct or indirect connection that is wired and / or wireless in nature. In addition, even if the transmitted information can be modified, processed, relayed and / or routed between the first unit and the second unit, the two units can communicate with each other. For example, even if the first unit passively receives information and does not actively transmit information to the second unit, the first unit can communicate with the second unit. As another example, if at least one intermediary unit (e.g., a third unit located between the first unit and the second unit) processes the information received from the first unit and transmits the processed information to the second unit, the first unit can communicate with the second unit. In some embodiments, a message may refer to a network packet (eg, a data packet, etc.) that includes data.

[0019] As used herein, the term "if" is optionally interpreted to mean "when," "at," "in response to being determined to be," and / or "in response to being detected," etc., depending on the context. Similarly, the phrases "if it is determined" or "if [the stated condition or event] is detected" are optionally interpreted to mean "when determining," "in response to being determined to be" or "when [the stated condition or event] is detected," and / or "in response to being detected," etc., depending on the context. In addition, as used herein, the terms "have," "have," or "possess," etc. are intended to be open-ended terms. Furthermore, unless expressly stated otherwise, the phrase "based on" is intended to mean "based at least in part on."

[0020] “At least one” and “one or more than one” include a function being performed by one element, a function being performed by more than one element, such as in a distributed manner, several functions being performed by one element, several functions being performed by several elements, or any combination of the above.

[0021] Some embodiments of the present disclosure are described herein in conjunction with thresholds. As described herein, satisfying (such as meeting, etc.) a threshold may refer to a value being greater than a threshold, more than a threshold, above a threshold, greater than or equal to a threshold, less than a threshold, less than a threshold, below a threshold, less than or equal to a threshold, and / or equal to a threshold, etc.

[0022] Reference will now be made in detail to embodiments, examples of which are illustrated in the accompanying drawings. In the following detailed description, numerous specific details are set forth in order to provide a thorough understanding of the various embodiments described. However, it will be apparent to one of ordinary skill in the art that the various embodiments described may be implemented without these specific details. In other cases, well-known methods, processes, components, circuits, and networks have not yet been described in detail in order not to unnecessarily obscure aspects of the embodiments.

[0023] General Overview

[0024] A vehicle may encounter various types of obstacles (eg, other vehicles, pedestrians, infrastructure). Planning the routes taken by an autonomous vehicle is a resource-intensive task that requires some optimization to provide real-time planning.

[0025] In some aspects and / or embodiments, the systems, methods, and computer program products described herein include and / or implement: obtaining, using at least one processor, sensor data indicating an agent in an environment in which an autonomous vehicle is configured to operate along a first trajectory; determining, using at least one processor, a dynamic track associated with the agent; generating, using at least one processor, obstacle data associated with the agent based on the dynamic track, wherein the obstacle data indicates that the agent is an obstacle along a first trajectory (e.g., a baseline) of the autonomous vehicle; determining, using at least one processor, station constraints and lateral constraints to apply to the trajectory based on the obstacle data; generating, using at least one processor, a second trajectory for the autonomous vehicle based on the station constraints and the lateral constraints; and providing, using at least one processor, data associated with the second trajectory, wherein the data associated with the second trajectory is configured to cause the autonomous vehicle to operate along the second trajectory.

[0026] With the implementation of the systems, methods, and computer program products described herein, techniques for obstacle representation advantageously provide more efficient generation of constraints for dynamic trajectories in route planning for autonomous vehicles while providing safety. The disclosed techniques can simplify the generation of station lateral and time (SLT) constraints, thereby improving computational efficiency while improving accuracy. The disclosed techniques further allow the use of SLT constraints to model the longitudinal behavior (e.g., rate and station) and lateral behavior (e.g., steering) of autonomous vehicles. With the implementation, these techniques can provide faster provision of trajectories when encountering an agent.

[0027] Reference now Figure 1, illustrates an example environment 100 in which vehicles including autonomous systems and vehicles not including autonomous systems operate. As illustrated, the environment 100 includes vehicles 102a-102n, objects 104a-104n, routes 106a-106n, areas 108, vehicle-to-infrastructure (V2I) devices 110, a network 112, a remote autonomous vehicle (AV) system 114, a fleet management system 116, and a V2I system 118. The vehicles 102a-102n, the vehicle-to-infrastructure (V2I) devices 110, the network 112, the autonomous vehicle (AV) system 114, the fleet management system 116, and the V2I system 118 are interconnected (e.g., establish connections for communication, etc.) via wired connections, wireless connections, or a combination of wired or wireless connections. In some embodiments, objects 104a-104n are interconnected with at least one of vehicles 102a-102n, vehicle-to-infrastructure (V2I) devices 110, networks 112, autonomous vehicle (AV) systems 114, fleet management systems 116, and V2I systems 118 via wired connections, wireless connections, or a combination of wired or wireless connections.

[0028] Vehicles 102a-102n (individually referred to as vehicles 102 and collectively referred to as vehicles 102) include at least one device configured to transport goods and / or people. In some embodiments, vehicles 102 are configured to communicate with V2I devices 110, remote AV systems 114, fleet management systems 116, and / or V2I systems 118 via network 112. In some embodiments, vehicles 102 include cars, buses, trucks, and / or trains, etc. In some embodiments, vehicles 102 are similar to vehicles 200 described herein (see Figure 2 ). In some embodiments, vehicles 200 in the set of vehicles 200 are associated with an autonomous queue manager. In some embodiments, vehicles 102 travel along respective routes 106a-106n (individually referred to as routes 106 and collectively referred to as routes 106) as described herein. In some embodiments, one or more vehicles 102 include an autonomous system (e.g., an autonomous system that is the same as or similar to autonomous system 202).

[0029] Objects 104a-104n (individually referred to as objects 104 and collectively referred to as objects 104) include, for example, at least one vehicle, at least one pedestrian, at least one cyclist, and / or at least one structure (e.g., a building, a sign, a fire hydrant, etc.), etc. Each object 104 is stationary (e.g., located at a fixed location and over a period of time) or moves (e.g., has a speed and is associated with at least one trajectory). In some embodiments, objects 104 are associated with corresponding locations in area 108.

[0030] Routes 106a-106n (individually referred to as routes 106 and collectively referred to as routes 106) are each associated with (e.g., specify a series of actions (also referred to as trajectories) connecting states along which the AV can navigate. Each route 106 begins at an initial state (e.g., a state corresponding to a first spatiotemporal location and / or speed, etc.) and ends at a final target state (e.g., a state corresponding to a second spatiotemporal location different from the first spatiotemporal location) or a target zone (e.g., a subspace of acceptable states (e.g., terminal states)). In some embodiments, the first state includes a location where one or more individuals will board the AV, and the second state or zone includes one or more locations where one or more individuals boarding the AV will disembark. In some embodiments, routes 106 include multiple acceptable state sequences (e.g., multiple spatiotemporal location sequences) that are associated with (e.g., define multiple trajectories). In an example, routes 106 include only high-level actions or imprecise state locations, such as a series of connecting roads indicating a change of direction at a roadway intersection, etc. Additionally or alternatively, the route 106 may include more precise actions or states, such as, for example, specific target lanes or precise locations within lane regions and target speeds at those locations, etc. In an example, the route 106 includes a plurality of precise state sequences along at least one high-level action with a limited look-ahead horizon to an intermediate target, wherein a combination of consecutive iterations of the limited horizon state sequences cumulatively correspond to a plurality of trajectories that collectively form a high-level route terminating at a final target state or region.

[0031] The area 108 includes a physical area (e.g., a geographic region) in which the vehicle 102 can navigate. In an example, the area 108 includes at least one state (e.g., a country, a province, a separate state of a plurality of states included in a country, etc.), at least a portion of a state, at least one city, at least a portion of a city, etc. In some embodiments, the area 108 includes at least one named thoroughfare (referred to herein as a "road"), such as a highway, an interstate highway, a parkway, a city street, etc. Additionally or alternatively, in some examples, the area 108 includes at least one unnamed road, such as a driveway, a section of a parking lot, a section of an open space and / or undeveloped area, a dirt road, etc. In some embodiments, the road includes at least one lane (e.g., a portion of the road that the vehicle 102 can traverse). In an example, the road includes at least one lane associated with (e.g., identified based on) at least one lane marking line.

[0032] The vehicle-to-infrastructure (V2I) device 110 (sometimes referred to as a vehicle-to-infrastructure or vehicle-to-everything (V2X) device) includes at least one device configured to communicate with the vehicle 102 and / or the V2I system 118. In some embodiments, the V2I device 110 is configured to communicate with the vehicle 102, the remote AV system 114, the fleet management system 116, and / or the V2I system 118 via the network 112. In some embodiments, the V2I device 110 includes a radio frequency identification (RFID) device, a sign, a camera (e.g., a two-dimensional (2D) and / or three-dimensional (3D) camera), lane markings, street lights, parking meters, etc. In some embodiments, the V2I device 110 is configured to communicate directly with the vehicle 102. Additionally or alternatively, in some embodiments, the V2I device 110 is configured to communicate with the vehicle 102, the remote AV system 114, and / or the fleet management system 116 via the V2I system 118. In some embodiments, the V2I device 110 is configured to communicate with the V2I system 118 via the network 112.

[0033] The network 112 includes one or more wired and / or wireless networks. In an example, the network 112 includes a cellular network (e.g., a long-term evolution (LTE) network, a third generation (3G) network, a fourth generation (4G) network, a fifth generation (5G) network, a code division multiple access (CDMA) network, etc.), a public land mobile network (PLMN), a local area network (LAN), a wide area network (WAN), a metropolitan area network (MAN), a telephone network (e.g., a public switched telephone network (PSTN)), a private network, an ad hoc network, an intranet, the Internet, a fiber-based network, a cloud computing network, etc., and / or a combination of some or all of these networks, etc.

[0034] The remote AV system 114 includes at least one device configured to communicate with the vehicle 102, the V2I device 110, the network 112, the fleet management system 116, and / or the V2I system 118 via the network 112. In an example, the remote AV system 114 includes a server, a server group, and / or other similar devices. In some embodiments, the remote AV system 114 is co-located with the fleet management system 116. In some embodiments, the remote AV system 114 participates in the installation of some or all of the components of the vehicle (including autonomous systems, autonomous vehicle computing, and / or software implemented by autonomous vehicle computing, etc.). In some embodiments, the remote AV system 114 maintains (e.g., updates and / or replaces) these components and / or software during the life of the vehicle.

[0035] The queue management system 116 includes at least one device configured to communicate with the vehicles 102, the V2I devices 110, the remote AV system 114, and / or the V2I system 118. In an example, the queue management system 116 includes a server, a server group, and / or other similar devices. In some embodiments, the queue management system 116 is associated with a ridesharing company (e.g., an organization for controlling the operation of multiple vehicles (e.g., vehicles including autonomous systems and / or vehicles not including autonomous systems)).

[0036] In some embodiments, the V2I system 118 includes at least one device configured to communicate with the vehicle 102, the V2I device 110, the remote AV system 114, and / or the fleet management system 116 via the network 112. In some examples, the V2I system 118 is configured to communicate with the V2I device 110 via a connection other than the network 112. In some embodiments, the V2I system 118 includes a server, a server group, and / or other similar devices. In some embodiments, the V2I system 118 is associated with a municipality or a private agency (e.g., a private agency for maintaining the V2I device 110, etc.).

[0037] In some embodiments, Fig.10 As illustrated, the apparatus 300 is configured as software instructions to perform one or more steps of the disclosed method.

[0038] supply Figure 1 The number and arrangement of elements illustrated are examples. Figure 1 There may be additional elements, fewer elements, different elements, and / or differently arranged elements than those illustrated. Additionally or alternatively, at least one element of environment 100 may be described as being Figure 1Additionally or alternatively, at least one set of elements of environment 100 may perform one or more functions described as being performed by at least one different set of elements of environment 100.

[0039] Reference now Figure 2 , vehicle 200 (which can be Figure 1 102) includes, or is associated with, autonomous system 202, powertrain control system 204, steering control system 206, and braking system 208. In some embodiments, vehicle 200 is similar to vehicle 102 (see Figure 1 ) is the same or similar. In some embodiments, the autonomous system 202 is configured to give the vehicle 200 autonomous driving capabilities (e.g., implementing at least one of the following driving automatic or maneuver-based functions, features and / or devices, etc., which enable the vehicle 200 to operate partially or completely without human intervention, including but not limited to fully autonomous vehicles (e.g., vehicles that abandon reliance on human intervention, such as Level 5 ADS-operated vehicles, etc.), highly autonomous vehicles (e.g., vehicles that abandon reliance on human intervention in certain situations, such as Level 4 ADS-operated vehicles, etc.), and / or conditionally autonomous vehicles (e.g., vehicles that abandon reliance on human intervention in limited situations, such as Level 3 ADS-operated vehicles, etc.), etc.). In one embodiment, the autonomous system 202 includes the operational or tactical functionality required to enable the vehicle 200 to operate in traffic on the road and continuously perform part or all of the dynamic driving task (DDT). In another embodiment, the autonomous system 202 includes an advanced driver assistance system (ADAS) including driver support features. The autonomous system 202 supports various levels of driving automation ranging from no driving automation (e.g., level 0) to full driving automation (e.g., level 5). For a detailed description of fully autonomous vehicles and highly autonomous vehicles, reference may be made to SAE International's standard J3016: Taxonomy and Definitions for Terms Related to On-Road Motor Vehicle Automated Driving Systems, the entire contents of which are incorporated by reference. In some embodiments, the vehicle 200 is associated with an autonomous queue manager and / or a ride-sharing company.

[0040] Autonomous system 202 includes a sensor suite that includes one or more devices such as camera 202a, LiDAR sensor 202b, Radar sensor 202c, and microphone 202d. In some embodiments, autonomous system 202 may include more or fewer devices and / or different devices (e.g., ultrasonic sensors, inertial sensors, GPS receivers (discussed below), and / or odometer sensors for generating data associated with an indication of the distance that vehicle 200 has traveled, etc.). In some embodiments, autonomous system 202 uses one or more devices included in autonomous system 202 to generate data associated with environment 100 described herein. Data generated by one or more devices of autonomous system 202 can be used by one or more systems described herein to observe the environment (e.g., environment 100) in which vehicle 200 is located. In some embodiments, autonomous system 202 includes communication device 202e, autonomous vehicle computing 202f, drive-by-wire (DBW) system 202h, and safety controller 202g.

[0041] The camera 202a includes a communication device 202e, an autonomous vehicle computer 202f, and / or a safety controller 202g configured to communicate with the communication device 202e via a bus (e.g., Figure 3 The camera 202a includes at least one device for communicating with the autonomous vehicle computing 202f (e.g., an image processing unit 202a, a bus ... Figure 1In some embodiments, the autonomous vehicle computing 202f determines a depth to one or more objects in a field of view of at least two of the plurality of cameras based on image data from the at least two cameras. In some embodiments, the camera 202a is configured to capture images of objects within a distance relative to the camera 202a (e.g., up to 100 meters and / or up to 1 kilometer, etc.). Thus, the camera 202a includes features such as sensors and lenses that are optimized for sensing objects at one or more distances relative to the camera 202a.

[0042] In an embodiment, the camera 202a includes at least one camera configured to capture one or more images associated with one or more traffic lights, street signs, and / or other physical objects that provide visual navigation information. In some embodiments, the camera 202a generates traffic light data associated with the one or more images. In some examples, the camera 202a generates TLD (traffic light detection) data associated with one or more images including a format (e.g., RAW, JPEG, and / or PNG, etc.). In some embodiments, the camera 202a that generates TLD data differs from other systems incorporating cameras described herein in that the camera 202a may include one or more cameras with a wide field of view (e.g., a wide-angle lens, a fisheye lens, and / or a lens with a viewing angle of approximately 120 degrees or greater, etc.) to generate images related to as many physical objects as possible.

[0043] The light detection and ranging (LiDAR) sensor 202b includes a communication device 202e, an autonomous vehicle computing device 202f, and / or a safety controller 202g via a bus (e.g., Figure 3The LiDAR sensor 202b includes at least one device that communicates with a bus (the same or similar bus as the bus 302 of the embodiment of the present invention). The LiDAR sensor 202b includes a system configured to emit light from a light emitter (e.g., a laser emitter). The light emitted by the LiDAR sensor 202b includes light outside the visible spectrum (e.g., infrared light, etc.). In some embodiments, during operation, the light emitted by the LiDAR sensor 202b encounters a physical object (e.g., a vehicle) and is reflected back to the LiDAR sensor 202b. In some embodiments, the light emitted by the LiDAR sensor 202b does not penetrate the physical object encountered by the light. The LiDAR sensor 202b also includes at least one light detector that detects the light emitted from the light emitter after encountering the physical object. In some embodiments, at least one data processing system associated with the LiDAR sensor 202b generates an image (e.g., a point cloud and / or a combined point cloud, etc.) representing objects included in the field of view of the LiDAR sensor 202b. In some examples, at least one data processing system associated with the LiDAR sensor 202b generates an image representing the boundaries of the physical object and / or the surface of the physical object (e.g., the topology of the surface), etc. In such examples, the image is used to determine the boundaries of the physical object in the field of view of the LiDAR sensor 202b.

[0044] The radio detection and ranging (Radar) sensor 202c includes a sensor configured to communicate with the communication device 202e, the autonomous vehicle computing 202f, and / or the safety controller 202g via a bus (e.g., Figure 3 At least one device for communicating with a bus (same or similar bus as bus 302 of the embodiment of the present invention). Radar sensor 202c includes a system configured to transmit (pulsed or continuous) radio waves. The radio waves transmitted by Radar sensor 202c include radio waves within a predetermined spectrum. In some embodiments, during operation, the radio waves transmitted by Radar sensor 202c encounter physical objects and are reflected back to Radar sensor 202c. In some embodiments, the radio waves transmitted by Radar sensor 202c are not reflected by some objects. In some embodiments, at least one data processing system associated with Radar sensor 202c generates a signal representing an object included in the field of view of Radar sensor 202c. For example, at least one data processing system associated with Radar sensor 202c generates an image representing the boundary of a physical object and / or the surface of a physical object (e.g., the topology of the surface), etc. In some examples, the image is used to determine the boundary of a physical object in the field of view of Radar sensor 202c.

[0045] The microphone 202d includes a microphone configured to communicate with the communication device 202e, the autonomous vehicle computing device 202f, and / or the safety controller 202g via a bus (e.g., Figure 3 At least one device for communicating with the vehicle 200 (the same or similar bus as bus 302 of FIG. 1 ). Microphone 202d includes one or more microphones (e.g., an array microphone and / or an external microphone, etc.) that capture an audio signal and generate data associated with (e.g., representing) the audio signal. In some examples, microphone 202d includes a transducer device and / or the like. In some embodiments, one or more systems described herein can receive the data generated by microphone 202d and determine the position (e.g., distance, etc.) of an object relative to vehicle 200 based on an audio signal associated with the data.

[0046] The communication device 202e includes at least one device configured to communicate with the camera 202a, the LiDAR sensor 202b, the Radar sensor 202c, the microphone 202d, the autonomous vehicle computing 202f, the safety controller 202g, and / or the DBW (drive-by-wire) system 202h. For example, the communication device 202e may include at least one device configured to communicate with the camera 202a, the LiDAR sensor 202b, the Radar sensor 202c, the microphone 202d, the autonomous vehicle computing 202f, the safety controller 202g, and / or the DBW (drive-by-wire) system 202h. Figure 3 The communication device 202e may be a device that is the same as or similar to the communication interface 314 of the vehicle. In some embodiments, the communication device 202e includes a vehicle-to-vehicle (V2V) communication device (eg, a device for enabling wireless communication of data between vehicles).

[0047] Autonomous vehicle computing 202f includes at least one device configured to communicate with camera 202a, LiDAR sensor 202b, Radar sensor 202c, microphone 202d, communication device 202e, safety controller 202g, and / or DBW system 202h. In some examples, autonomous vehicle computing 202f includes devices such as client devices, mobile devices (e.g., cellular phones and / or tablet computers, etc.), and / or servers (e.g., computing devices including one or more central processing units and / or graphics processing units, etc.). In some embodiments, autonomous vehicle computing 202f is the same or similar to autonomous vehicle computing 400 described herein. Additionally or alternatively, in some embodiments, autonomous vehicle computing 202f is configured to communicate with an autonomous vehicle system (e.g., with Figure 1 remote AV system 114 of the same or similar autonomous vehicle system), a fleet management system (e.g., Figure 1 of the same or similar queue management system as the queue management system 116), V2I devices (e.g., Figure 1 V2I device 110 that is the same as or similar to V2I device 110) and / or V2I system (e.g., Figure 1The V2I system 118 may communicate with the same or similar V2I system.

[0048] Safety controller 202g includes at least one device configured to communicate with camera 202a, LiDAR sensor 202b, Radar sensor 202c, microphone 202d, communication device 202e, autonomous vehicle computing 202f, and / or DBW system 202h. In some examples, safety controller 202g includes one or more controllers (electrical controllers and / or electromechanical controllers, etc.) configured to generate and / or transmit control signals to operate one or more devices of vehicle 200 (e.g., powertrain control system 204, steering control system 206, and / or braking system 208, etc.). In some embodiments, safety controller 202g is configured to generate control signals that take precedence over (e.g., override) control signals generated and / or transmitted by autonomous vehicle computing 202f.

[0049] The DBW system 202h includes at least one device configured to communicate with the communication device 202e and / or the autonomous vehicle computing 202f. In some examples, the DBW system 202h includes one or more controllers (e.g., electrical controllers and / or electromechanical controllers, etc.) configured to generate and / or transmit control signals to operate one or more devices of the vehicle 200 (e.g., powertrain control system 204, steering control system 206, and / or braking system 208, etc.). Additionally or alternatively, one or more controllers of the DBW system 202h are configured to generate and / or transmit control signals to operate at least one different device of the vehicle 200 (e.g., turn signal lights, headlights, door locks, and / or windshield wipers, etc.).

[0050] The powertrain control system 204 includes at least one device configured to communicate with the DBW system 202h. In some examples, the powertrain control system 204 includes at least one controller and / or actuator, etc. In some embodiments, the powertrain control system 204 receives a control signal from the DBW system 202h, and the powertrain control system 204 causes the vehicle 200 to perform longitudinal vehicle motion (such as starting to move forward, stopping to move forward, starting to move backward, stopping to move backward, accelerating in a certain direction, decelerating in a certain direction, etc.), or to perform lateral vehicle motion (such as turning left and / or turning right, etc.). In an example, the powertrain control system 204 increases, maintains the same, or decreases the energy (e.g., fuel and / or electricity, etc.) provided to the motor of the vehicle, thereby rotating or not rotating at least one wheel of the vehicle 200. In other words, the steering control system 206 causes the activity required to regulate the y-axis component of the vehicle motion.

[0051] The steering control system 206 includes at least one device configured to rotate one or more wheels of the vehicle 200. In some examples, the steering control system 206 includes at least one controller and / or actuator, etc. In some embodiments, the steering control system 206 rotates the two front wheels and / or the two rear wheels of the vehicle 200 to the left or right to turn the vehicle 200 left or right.

[0052] Braking system 208 includes at least one device configured to actuate one or more brakes to decelerate and / or hold vehicle 200 stationary. In some examples, braking system 208 includes at least one controller and / or actuator configured to cause one or more calipers associated with one or more wheels of vehicle 200 to close on respective rotors of vehicle 200. Additionally or alternatively, in some examples, braking system 208 includes an automatic emergency braking (AEB) system and / or a regenerative braking system, etc.

[0053] In some embodiments, vehicle 200 includes at least one platform sensor (not explicitly illustrated) for measuring or inferring a property of a state or condition of vehicle 200. In some examples, vehicle 200 includes platform sensors such as a global positioning system (GPS) receiver, an inertial measurement unit (IMU), wheel rate sensors, wheel brake pressure sensors, wheel torque sensors, engine torque sensors, and / or steering angle sensors. Although brake system 208 is illustrated as being located at Figure 2 The braking system 208 is located on the proximal side of the vehicle 200 , but the braking system 208 can be located anywhere in the vehicle 200 .

[0054] Reference now Figure 3, a schematic diagram of an exemplary device 300. As illustrated, the device 300 includes a processor 304, a memory 306, a storage component 308, an input interface 310, an output interface 312, a communication interface 314, and a bus 302. In some embodiments, the device 300 corresponds to: at least one device of the vehicle 102 (e.g., at least one device of a system of the vehicle 102); at least one device of the remote AV system 114, the fleet management system 116, the V2I system 118; and / or one or more devices of the network 112 (e.g., one or more devices of a system of the network 112). In some embodiments, one or more devices of vehicle 102 (e.g., one or more devices of a system of vehicle 102 (such as remote AV system 114, fleet management system 116, and at least one device of V2I system 118), and / or one or more devices of network 112 (e.g., one or more devices of a system of network 112) include at least one device 300 and / or at least one component of device 300. Figure 3 As shown, apparatus 300 includes a bus 302 , a processor 304 , a memory 306 , a storage component 308 , an input interface 310 , an output interface 312 , and a communication interface 314 .

[0055] The bus 302 includes components that permit communication between components of the device 300. In some cases, the processor 304 includes a processor (e.g., a central processing unit (CPU), a graphics processing unit (GPU), and / or an accelerated processing unit (APU), etc.), a microphone, a digital signal processor (DSP), and / or any processing component that can be programmed to perform at least one function (e.g., a field programmable gate array (FPGA) and / or an application specific integrated circuit (ASIC), etc.). The memory 306 includes a random access memory (RAM), a read-only memory (ROM), and / or another type of dynamic and / or static storage device (e.g., flash memory, magnetic memory, and / or optical memory, etc.) that stores data and / or instructions for use by the processor 304.

[0056] Storage component 308 stores data and / or software related to the operation and use of device 300. In some examples, storage component 308 includes a hard disk (e.g., a magnetic disk, an optical disk, a magneto-optical disk, and / or a solid-state disk, etc.), a compact disk (CD), a digital versatile disk (DVD), a floppy disk, a cassette, a tape, a CD-ROM, a RAM, a PROM, an EPROM, a FLASH-EPROM, an NV-RAM, and / or another type of computer-readable medium, and a corresponding drive.

[0057] The input interface 310 includes components that permit the device 300 to receive information, such as via user input (e.g., a touch screen display, a keyboard, a keypad, a mouse, a button, a switch, a microphone, and / or a camera, etc.). Additionally or alternatively, in some embodiments, the input interface 310 includes a sensor for sensing information (e.g., a global positioning system (GPS) receiver, an accelerometer, a gyroscope, and / or an actuator, etc.). The output interface 312 includes components for providing output information from the device 300 (e.g., a display, a speaker, and / or one or more light emitting diodes (LEDs), etc.).

[0058] In some embodiments, communication interface 314 includes a transceiver-like component (e.g., a transceiver and / or a separate receiver and transmitter, etc.) that permits device 300 to communicate with other devices via a wired connection, a wireless connection, or a combination of a wired connection and a wireless connection. In some examples, communication interface 314 permits device 300 to receive information from another device and / or provide information to another device. In some examples, communication interface 314 includes an Ethernet interface, an optical interface, a coaxial interface, an infrared interface, a radio frequency (RF) interface, a universal serial bus (USB) interface, interface and / or cellular network interface, etc.

[0059] In some embodiments, the device 300 performs one or more processes described herein. The device 300 performs these processes based on the processor 304 executing software instructions stored by a computer-readable medium such as a memory 306 and / or a storage component 308. Computer-readable media (e.g., non-transitory computer-readable media) are defined herein as non-transitory memory devices. Non-transitory memory devices include storage space located within a single physical storage device or storage space distributed across multiple physical storage devices.

[0060] In some embodiments, the software instructions are read into the memory 306 and / or storage component 308 from another computer-readable medium or from another device via the communication interface 314. The software instructions stored in the memory 306 and / or storage component 308, when executed, cause the processor 304 to perform one or more processes described herein. Additionally or alternatively, hardwired circuitry is used in place of or in combination with the software instructions to perform one or more processes described herein. Therefore, unless expressly stated otherwise, the embodiments described herein are not limited to any specific combination of hardware circuitry and software.

[0061] The memory 306 and / or the storage component 308 include a data storage unit or at least one data structure (e.g., a database, etc.). The device 300 can receive information from the data storage unit or at least one data structure in the memory 306 or the storage component 308, store information in the data storage unit or at least one data structure, communicate information to the data storage unit or at least one data structure, or search for information stored in the data storage unit or at least one data structure. In some examples, the information includes network data, input data, output data, or any combination thereof.

[0062] In some embodiments, the device 300 is configured to execute software instructions stored in the memory 306 and / or the memory of another device (e.g., another device that is the same as or similar to the device 300). As used herein, the term "module" refers to at least one instruction stored in the memory 306 and / or the memory of another device, which, when executed by the processor 304 and / or the processor of another device (e.g., another device that is the same as or similar to the device 300), causes the device 300 (e.g., at least one component of the device 300) to perform one or more processes described herein. In some embodiments, the module is implemented in software, firmware, and / or hardware, etc.

[0063] supply Figure 3 The number and arrangement of components illustrated are examples. Figure 3 The device 300 may include additional components, fewer components, different components, or differently arranged components than those illustrated. Additionally or alternatively, a set of components (e.g., one or more components) of the device 300 may perform one or more functions described as being performed by another component or set of components of the device 300.

[0064] Reference now Figure 4, illustrates an example block diagram of an autonomous vehicle computing 400 (sometimes referred to as an "AV stack"). As illustrated, the autonomous vehicle computing 400 includes a perception system 402 (sometimes referred to as a perception module), a planning system 404 (sometimes referred to as a planning module), a positioning system 406 (sometimes referred to as a positioning module), a control system 408 (sometimes referred to as a control module), and a database 410. In some embodiments, the perception system 402, the planning system 404, the positioning system 406, the control system 408, and the database 410 are included in and / or implemented in an automatic navigation system of a vehicle (e.g., the autonomous vehicle computing 202f of the vehicle 200). Additionally or alternatively, in some embodiments, the perception system 402, the planning system 404, the positioning system 406, the control system 408, and the database 410 are included in one or more independent systems (e.g., one or more systems that are the same or similar to the autonomous vehicle computing 400, etc.). In some examples, the perception system 402, planning system 404, positioning system 406, control system 408, and database 410 are included in one or more independent systems located in the vehicle and / or at least one remote system as described herein. In some embodiments, any and / or all of the systems included in the autonomous vehicle computing 400 are implemented in software (e.g., software instructions stored in a memory), computer hardware (e.g., by a microprocessor, microcontroller, application specific integrated circuit (ASIC) and / or field programmable gate array (FPGA), etc.), or a combination of computer software and computer hardware. It will also be understood that in some embodiments, the autonomous vehicle computing 400 is configured to communicate with a remote system (e.g., an autonomous vehicle system that is the same or similar to the remote AV system 114, a fleet management system 116 that is the same or similar to the fleet management system 116, and / or a V2I system that is the same or similar to the V2I system 118, etc.).

[0065] In some embodiments, the perception system 402 receives data associated with at least one physical object in the environment (e.g., data used by the perception system 402 to detect at least one physical object) and classifies the at least one physical object. In some examples, the perception system 402 receives image data captured by at least one camera (e.g., camera 202a), the image being associated with (e.g., representing) one or more physical objects within the field of view of the at least one camera. In such examples, the perception system 402 classifies the at least one physical object based on one or more groups of physical objects (e.g., bicycles, vehicles, traffic signs, and / or pedestrians, etc.). In some embodiments, based on the classification of the physical object by the perception system 402, the perception system 402 transmits data associated with the classification of the physical object to the planning system 404.

[0066] In some embodiments, the planning system 404 receives data associated with a destination and generates data associated with at least one route (e.g., route 106) along which a vehicle (e.g., vehicle 102) can travel toward the destination. In some embodiments, the planning system 404 periodically or continuously receives data (e.g., data associated with the classification of physical objects described above) from the perception system 402, and the planning system 404 updates at least one trajectory or generates at least one different trajectory based on the data generated by the perception system 402. In other words, the planning system 404 can perform tasks related to tactical functions required to operate the vehicle 102 in traffic on the road. Tactical efforts involve maneuvering the vehicle in traffic during the journey, which includes but is not limited to deciding whether and when to overtake another vehicle, change lanes, or select an appropriate rate, acceleration, deceleration, etc. In some embodiments, the planning system 404 receives data associated with an updated position of the vehicle (e.g., vehicle 102) from the positioning system 406, and the planning system 404 updates at least one trajectory or generates at least one different trajectory based on the data generated by the positioning system 406.

[0067] In some embodiments, the positioning system 406 receives data associated with (e.g., representing) a location of a vehicle (e.g., vehicle 102) in an area. In some examples, the positioning system 406 receives LiDAR data associated with at least one point cloud generated by at least one LiDAR sensor (e.g., LiDAR sensor 202b). In some examples, the positioning system 406 receives data associated with at least one point cloud from multiple LiDAR sensors, and the positioning system 406 generates a combined point cloud based on each point cloud. In these examples, the positioning system 406 compares the at least one point cloud or the combined point cloud with a two-dimensional (2D) and / or three-dimensional (3D) map of the area stored in the database 410. Then, based on the positioning system 406 comparing the at least one point cloud or the combined point cloud with the map, the positioning system 406 determines the position of the vehicle in the area. In some embodiments, the map includes a combined point cloud of the area generated before the navigation of the vehicle. In some embodiments, the map includes, but is not limited to, a high-precision map of roadway geometry, a map describing the connectivity of the road network, a map describing the physical properties of the roadway (such as traffic speed, traffic volume, the number of vehicle and bicycle traffic lanes, lane width, lane traffic direction, or the type and location of lane markings, or a combination thereof), and a map describing the spatial location of road features (such as crosswalks, traffic signs, or various types of other driving signals, etc.) In some embodiments, the map is generated in real time based on data received by the perception system.

[0068] In another example, positioning system 406 receives global navigation satellite system (GNSS) data generated by a global positioning system (GPS) receiver. In some examples, positioning system 406 receives GNSS data associated with a location of a vehicle in an area, and positioning system 406 determines the latitude and longitude of the vehicle in the area. In such an example, positioning system 406 determines the position of the vehicle in the area based on the latitude and longitude of the vehicle. In some embodiments, positioning system 406 generates data associated with the position of the vehicle. In some examples, based on the location of the vehicle determined by positioning system 406, positioning system 406 generates data associated with the position of the vehicle. In such an example, the data associated with the position of the vehicle include data associated with one or more semantic properties corresponding to the position of the vehicle.

[0069] In some embodiments, the control system 408 receives data associated with at least one trajectory from the planning system 404, and the control system 408 controls the operation of the vehicle. In some examples, the control system 408 receives data associated with at least one trajectory from the planning system 404, and the control system 408 controls the operation of the vehicle by generating and transmitting control signals to operate the powertrain control system (e.g., DBW system 202h and / or powertrain control system 204, etc.), the steering control system (e.g., steering control system 206) and / or the braking system (e.g., braking system 208). For example, the control system 408 is configured to perform operational functions such as lateral vehicle motion control or longitudinal vehicle motion control. Lateral vehicle motion control causes the activity required to regulate the y-axis component of the vehicle motion. Longitudinal vehicle motion control causes the activity required to regulate the x-axis component of the vehicle motion. In the example, in the case where the trajectory includes a left turn, the control system 408 transmits a control signal to cause the steering control system 206 to adjust the steering angle of the vehicle 200, thereby causing the vehicle 200 to turn left. Additionally or alternatively, the control system 408 generates and transmits control signals to cause other devices of the vehicle 200 (eg, headlights, turn signal lights, door locks, and / or windshield wipers, etc.) to change states.

[0070] In some embodiments, perception system 402, planning system 404, positioning system 406, and / or control system 408 implement at least one machine learning model (e.g., at least one multi-layer perceptron (MLP), at least one convolutional neural network (CNN), at least one recurrent neural network (RNN), at least one autoencoder, and / or at least one transformer, etc.). In some examples, perception system 402, planning system 404, positioning system 406, and / or control system 408 implement at least one machine learning model alone or in combination with one or more of the above systems. In some examples, perception system 402, planning system 404, positioning system 406, and / or control system 408 implement at least one machine learning model as part of a pipeline (e.g., a pipeline for identifying one or more objects located in an environment, etc.).

[0071] Database 410 stores data transmitted to, received from, and / or updated by perception system 402, planning system 404, positioning system 406, and / or control system 408. In some examples, database 410 includes a storage component (e.g., a storage component) for storing data and / or software related to operations and using at least one system of autonomous vehicle computing 400. Figure 3In some embodiments, database 410 stores data associated with a 2D and / or 3D map of at least one area. In some examples, database 410 stores data associated with a 2D and / or 3D map of a portion of a city, portions of multiple cities, multiple cities, counties, states, and / or countries (State) (e.g., a country), etc. In such an example, a vehicle (e.g., a vehicle that is the same or similar to vehicle 102 and / or vehicle 200) can be driven along one or more drivable areas (e.g., single-lane roads, multi-lane roads, highways, remote roads, and / or off-road roads, etc.) and cause at least one LiDAR sensor (e.g., a LiDAR sensor that is the same or similar to LiDAR sensor 202b) to generate data associated with an image representing an object included in the field of view of the at least one LiDAR sensor.

[0072] In some embodiments, database 410 can be implemented across multiple devices. In some examples, database 410 includes a vehicle (e.g., a vehicle that is the same or similar to vehicle 102 and / or vehicle 200), an autonomous vehicle system (e.g., an autonomous vehicle system that is the same or similar to remote AV system 114), a fleet management system (e.g., a vehicle that is the same or similar to remote AV system 114), and a fleet management system (e.g., a vehicle that is the same or similar to remote AV system 114). Figure 1 The same or similar queue management system as the queue management system 116 of FIG. 1 and / or the V2I system (e.g., Figure 1 The V2I system 118 is the same as or similar to the V2I system 118).

[0073] The present disclosure provides techniques for determining dynamic traces of obstacles (such as vehicles or pedestrians, etc.) as a function of time for a route within a time range. In some examples, dynamic traces associated with corresponding predictions are provided. The dynamic traces of obstacles provide data that enables, for example, tracking obstacles (e.g., agents) per time unit and predicting where the obstacles may be based on previous time frames. For example, the dynamic trace represents one or more obstacle projections in 2D space (for station constraints, station over time, and for space constraints, lateral clearance with station). The dynamic trace is used, for example, to determine where an AV (e.g., based on station constraints) can be along the route plan, and how much space the AV has relative to its sides at a specific time within the time range (e.g., based on lateral constraints).

[0074] The present disclosure relates to systems, methods and computer program products that provide for determining a dynamic track (with prediction) of an agent (such as a vehicle or pedestrian, etc.) as a function of time. The dynamic track can then be used to generate obstacle data indicating that the agent is an obstacle along the trajectory of the autonomous vehicle. The obstacle data is then used to determine the station lateral and time (SLT) constraints for the autonomous vehicle route planning. The SLT constraints can be used to generate the trajectory used by the autonomous vehicle by determining homotopy based on the SLT constraints. The SLT constraints can be considered as obstacle projections in 2D space. The autonomous vehicle can use obstacles to generate constraints that can be queried, which represent where the autonomous vehicle can be along the route planning (station) and how much space the autonomous vehicle has relative to its side (laterally) at a specific time within a time range.

[0075] Reference now Figure 5 , a diagram illustrating a system 500 for obstacle representation. In some embodiments, the system 500 is coupled to a vehicle (e.g., Figure 2 In one or more embodiments or examples, the system 500 is connected to and / or incorporated into an AV (e.g., an autonomous vehicle such as a vehicle 200 that is the same or similar to the vehicle 200). Figure 2 The illustrated autonomous system 202, Figure 3 300, etc.), AV system, AV computing 540 (such as Figure 2 AV Computing 202F and / or Figure 4 AV computing 400, etc.), remote AV systems (such as Figure 1 remote AV system 114, etc.), queue management systems (such as Figure 1 queue management system 116, etc.) and V2I systems (such as Figure 1 The system 500 may be used to operate a vehicle. In one or more examples, the system 500 is used to operate an autonomous vehicle.

[0076] In one or more embodiments or examples, the system 500 communicates with one or more of the following: a device (such as Figure 3 In one or more embodiments or examples, the system 500 includes one or more of the following: a planning system 504 (e.g., Figure 4 6 ), the perception system 502 (e.g., Figure 4 perception system 402), prediction system 504, and control system 508 (e.g., Figure 46 and the control system 604b of FIG. 6). In one or more embodiments or examples, the system 500 includes a constraint generation system 508, a trajectory generation system 510, and optionally a trajectory selector system 512.

[0077] A system 500 is disclosed herein. In one or more examples or embodiments, the system 500 includes at least one processor. In one or more examples or embodiments, the system 500 includes at least one memory storing instructions that, when executed by the at least one processor, cause the at least one processor to perform operations. In one or more examples or embodiments, the operations include obtaining sensor data 506 indicating an agent in an environment in which the autonomous vehicle is configured to operate along a first trajectory. In one or more examples or embodiments, the operations include determining a dynamic trace associated with the agent. In one or more examples or embodiments, the operations include generating obstacle data associated with the agent based on the dynamic trace. In one or more examples or embodiments, the obstacle data indicates that the agent is an obstacle along the first trajectory of the autonomous vehicle. In one or more examples or embodiments, the operations include determining a station constraint and a lateral constraint to apply to the trajectory based on the obstacle data. In one or more examples or embodiments, the operations include generating a second trajectory of the autonomous vehicle based on the station constraint and the lateral constraint. In one or more examples or embodiments, operations include providing data associated with the second trajectory, the data associated with the second trajectory configured to cause the autonomous vehicle to operate along the second trajectory.

[0078] In other words, the system 500 is configured to use the sensor data 506 to detect the agents around the autonomous vehicle and predict the path that the agents will take (e.g., a dynamic trajectory). Using the predicted dynamic path, the system 500 can provide a trajectory configured to cause control operations of the autonomous vehicle, including: by using constraints to avoid harmful interactions with the agents. The disclosed system is configured to greatly improve the computational efficiency for avoiding agents in the scene (especially enabling both the agents and / or the autonomous vehicle to potentially take multiple paths).

[0079] In some of the conventional methods, when determining the path used by the AV, the system samples data at fixed distances along the path and imposes rate constraints and spatial constraints on some or all obstacles encountered. In some cases, this method can be complex processing and can be used for a single path. There is a need for a more efficient method for generating constraints to avoid collisions or interactions with dynamic agents near a baseline path. The disclosed systems and methods can enable determination of multiple safe trajectories and corresponding route plans based on the projection of the agent in the station-lateral-time domain, which significantly reduces the computational cost of such determination. Therefore, in some examples, the present disclosure is configured to generate obstacle data from a dynamic trace (e.g., a predicted dynamic trace of an agent), such as by compiling information used for each dynamic trace and generating station constraints and lateral constraints.

[0080] In one or more examples or embodiments, the system 500 obtains sensor data 506 (e.g., for detecting the environment around the system and / or the autonomous vehicle). For example, a perception system of the system (such as a sensor that is associated with the system) Figure 4 The sensor data 506 may be one or more of the following: Radar sensor data, non-Radar sensor data, camera sensor data, image sensor data, audio sensor, and LiDAR sensor data. The specific type of sensor data is not limiting. The sensor data 506 may indicate the environment around the autonomous vehicle. For example, the sensor data 506 may indicate an object and / or multiple objects in the environment around the autonomous vehicle. The object may include an agent in the environment. For example, an agent is an object that can have (e.g., be capable of) dynamic movement. Agents are, for example, vehicles, pedestrians, cyclists, etc. The sensor data 506 may indicate one or more agents, such as one or more of the following: vehicles, pedestrians, and cyclists.

[0081] In one or more examples or embodiments, the sensor is one or more sensors such as an onboard sensor. The sensor may be associated with an autonomous vehicle. The autonomous vehicle, for example, includes one or more sensors configured to monitor the environment in which the autonomous vehicle is operating, such as via a sensor, through sensor data 506. For example, monitoring provides sensor data 506 indicating what is happening in the environment around the autonomous vehicle (such as for determining a trajectory of the autonomous vehicle, etc.). The sensor may include Figure 2One or more of the illustrated sensors. The sensor may be one or more of the following: a Radar sensor, a camera sensor, a microphone, an infrared sensor, an image sensor, and a LiDAR sensor. In one or more example systems, the sensor may be selected from the group consisting of a Radar sensor, a camera sensor, and a LiDAR sensor.

[0082] While the system 500 is obtaining sensor data 506 indicative of an agent in the environment, in one or more examples or embodiments, the autonomous vehicle is configured to operate along a first trajectory in the environment. The autonomous vehicle may actively operate along the first trajectory while obtaining the sensor data 506. In some examples, the autonomous vehicle is configured to operate along the first trajectory (e.g., not actively operating along the first trajectory, but capable of operating along the first trajectory / ready to operate along the first trajectory). The system 500, for example, provides data associated with the first trajectory to cause the autonomous vehicle to operate along the first trajectory. The first trajectory may be, for example, for Figure 1 The trajectory in question.

[0083] In some examples, sensor data 506 indicates an agent in the environment. In one or more examples or embodiments, the system 500 is configured to determine a dynamic trace 504a associated with the agent. For example, the prediction system 504 of the system 500 may be configured to determine a dynamic trace 504a. The system 500 may be configured to determine a plurality of dynamic traces, each of which is associated with one of the plurality of agents. A dynamic trace is, for example, a prediction (e.g., estimate, projection) of the trajectory and / or position of an agent as a function of time. In other words, a dynamic trace indicates where the agent is moving within a specific time period (e.g., within a future time period). In one or more examples or embodiments, a dynamic trace is configured to track an agent (e.g., an agent trajectory) over time. For example, a dynamic trace represents the position of an agent over time. In one or more examples or embodiments, a dynamic trace is a prediction of the trajectory of an agent as a function of distance. For example, the system 500 is configured to determine the dynamic trace 504a by predicting where the agent is moving, such as based on the speed of the agent in one or more previous frames and the location of the agent. In some examples, the system 500 determines the dynamic trace 504a by predicting the trajectory of each agent over time and space. The system 500 can be configured to determine (e.g., update) the dynamic trace 504a at specific time intervals. The dynamic trace can be regarded as a decomposition of the 3-dimensional path of the agent to a 2-dimensional path (e.g., with the path length and lateral gap length of the agent). The dynamic trace is, for example, a two-dimensional indication of the potential path of the agent over time. In some examples, the dynamic trace changes in place based on time. In other words, as the agent moves, the dynamic trace of the agent is also updated.

[0084] In one or more examples or embodiments, the system 500 uses the dynamic trace to generate obstacle data 502a associated with the agent. The system 500 can be configured to generate a plurality of obstacle data 502a, each obstacle data being associated with one of the plurality of agents. The system 500, for example, converts the dynamic trace into an obstacle indicated by the obstacle data 502a. In some examples, the obstacle data indicates that the agent is an obstacle along the first trajectory of the autonomous vehicle. For example, if the agent is not an obstacle along the first trajectory, the system 500 is configured not to generate obstacle data associated with the agent. Generating obstacle data, for example, includes: taking sensor data 506 to the perception system 502 as input (e.g., sensor data 506), and using the system 500 to convert the dynamic trace from the prediction system 504 into obstacle data 502a. In one or more examples or embodiments, the obstacle data 502a provides one or more parameters that characterize the agent as an obstacle to the trajectory of the autonomous vehicle. The parameters include, for example, the dimensions (e.g., general dimensions) of the obstacle (e.g., agent). The dimensions can be a generalized set of dimensions, such as a polygon representing an agent, etc. In some examples, the obstacle data is a simplified data representation of the dynamic trace, which can be used to improve computational efficiency.

[0085] In one or more examples or embodiments, the system 500 uses the obstacle data 502a, such as via a planning system 520 (which may be associated with a Figure 4 The system 500 may be configured to plan actions to be taken by the autonomous vehicle, such as by planning system 404 of the present invention (same or similar to the planning system 404 of the present invention). In one or more examples or embodiments, based on obstacle data 502a, system 500 is configured to determine different constraints for the trajectory of the autonomous vehicle. Constraints may be understood as restrictions on the movement of the autonomous vehicle. For example, a constraint will prevent the autonomous vehicle from changing into a lane in which another vehicle is located.

[0086] As an example, the system 500 uses the constraint generation system 508 to determine one or more constraints 508a to apply to the trajectory of the autonomous vehicle. For example, the system 500 is configured to determine the station constraint 508a and / or the lateral constraint 508a based on the obstacle data. The station constraint can be regarded as a description of the station maneuver that can be characterized (e.g., parameterized) in time and space. In other words, via the station constraint, the autonomous vehicle is prevented from moving in a certain direction at a certain time. In one or more examples or embodiments, the station constraint is defined by the upper space / station limit and the lower space / station limit within which the autonomous vehicle should stay. The station constraint can be regarded as a constraint applied to the longitudinal maneuver of the autonomous vehicle. The station constraint is, for example, a spatiotemporal constraint (e.g., a station over time). In one or more examples or embodiments, a lateral constraint is a constraint characterized (e.g., parameterized) in space and position. For example, a lateral constraint is a lateral clearance with the station. In some examples, obstacle data is used to generate constraints that represent where the autonomous vehicle can be along the first trajectory (station) and how much space the autonomous vehicle has relative to its sides (laterally) at a specific time within the time frame. System 500 can be configured to provide data for controlling the rate of the autonomous vehicle and when to stop based on the clearance available along the path that the autonomous vehicle will interact with. Constraints can be provided by (e.g., Figure 4 The constraint generation system 508 included in the planning system 520 (which is the same as or similar to the planning system 404 of FIG. 6 and the planning system 604a of FIG. 6 ) is generated.

[0087] In other words, the system 500 uses obstacle data to determine one or more constraints for the movement of the autonomous vehicle in time. Since the autonomous vehicle should not hit the intelligent body trajectory represented by the obstacle data or otherwise interact with the intelligent body trajectory represented by the obstacle data, the obstacle data can be useful for determining the actions that the autonomous vehicle should not take. For example, the system 500 determines the station constraint 508a for longitudinal motion (such as the forward and backward (e.g., reverse) motion of the autonomous vehicle). For example, the system 500 determines the lateral constraint 508a for the lateral or sideways motion of the autonomous vehicle. These constraints can be dynamically adjusted as needed by the system based on any update of the obstacle data (such as via the update of the dynamic trace). In one or more examples or embodiments, by using the station constraint 508a and the lateral constraint 508a (which limits the movement of the autonomous vehicle), the system 500 can look forward in time (e.g., a single dimension). This can greatly improve the efficiency of the analysis for generating the second trajectory. The second trajectory should, for example, avoid any collision with any obstacle in the environment.

[0088] In one or more examples or embodiments, the system 500 is configured to take action at a specific time (e.g., a specific timestamp, a specific time interval). The timestamps can be at specific intervals, such as every millisecond, every second, etc., which will be advantageous. For example, the system 500 is configured to perform one or more of the following operations per timestamp: obtain sensor data 506, determine dynamic traces 504a, generate obstacle data 502a, and determine station constraints 508a and lateral constraints 508a. In one or more examples or embodiments, the system 500 is configured to determine whether there is a difference between the dynamic traces and / or obstacle data at a previous timestamp and the dynamic traces and / or obstacle data at a current timestamp. The system 500 can then be configured to update the station constraints 508a and / or lateral constraints 508a as needed.

[0089] Once the constraints are determined, in some examples, the system 500 is configured to generate (e.g., via the trajectory generation system 510) one or more additional trajectories 510b. The one or more additional trajectories 510b may be viewed as a second trajectory 510b, a third trajectory, etc. In one or more examples or embodiments, the system 500 is configured to generate a second trajectory 510b for the autonomous vehicle based on the station constraint 508a and / or the lateral constraint 508a. The second trajectory 510b is, for example, a corridor without a collision with one of the obstacles (e.g., an agent). In some examples, the second trajectory is the same as the first trajectory. In some examples, the second trajectory is different from the first trajectory. For example, if the obstacle data and / or the dynamic trace indicates that the agent interacts with the first trajectory in a manner that it will interact with the autonomous vehicle (e.g., enter a collision with the autonomous vehicle), it would be advantageous for the autonomous vehicle to change the trajectory. Optionally, system 500 includes a trajectory selector system 512 that can select a particular trajectory 512a generated by trajectory generation system 510. In one or more examples or embodiments, trajectory generation system 510 generates a single additional trajectory 510b (eg, a second trajectory), and trajectory selector system 512 is not required.

[0090] In one or more examples or embodiments, providing data associated with the second trajectory configured to cause operation of the autonomous vehicle includes generating a control system 513 (such as Figure 4For example, the system 500 is configured to provide data associated with the second trajectory so that the autonomous vehicle operates along the second trajectory. In some examples, providing data configured to cause the operation of the autonomous vehicle includes transmitting the control data to, for example, the control system and / or an external system of the autonomous vehicle. In one or more examples or embodiments, the system is configured to control the control system and / or the external system of the autonomous vehicle based on the control data. The autonomous vehicles discussed herein can be, for example, any of the L0, L1, L2, L3, L4, or L5 autonomous levels in the SAE (Society of Automotive Engineers) framework. For example, the autonomous vehicle is an unmanned vehicle, such as a fully autonomous vehicle, etc.

[0091] In one or more examples or embodiments, generating obstacle data includes determining a projection distance of the agent onto the first trajectory. In other words, Figure 7 As illustrated, the system can determine the projected distance between the agent and the path of the autonomous vehicle. For example, the system 500 is configured to determine and / or generate a polygon associated with the agent (e.g., indicating its shape). In one or more examples or embodiments, the system 500 projects the points of the polygon (e.g., shape) from the agent to the baseline path of the first track (e.g., the center of the lane forming part of the first track). In the case where the first track is a corridor that allows some maneuverability of the autonomous vehicle, the baseline path can be the center of such a corridor. The projection of the station on the first track provides one or more projection distances. In some examples, the system 500 selects the shortest distance as the projection distance. However, the baseline path is not the path that the autonomous vehicle will take in a strict sense (such as two parallel lanes, etc.). Through the projection distance, it is possible to know where the agent is relative to the two paths, thereby avoiding decomposition. When the lane is very wide, the system 500 is configured, for example, to move the baseline path more to the left or right. For example, the baseline path can be moved along the first track. Advantageously, in some examples, the first track is in 2D space.

[0092] In one or more examples or embodiments, generating obstacle data includes generating obstacle data including one or more of the following items: a projection distance, an agent type associated with the agent, and an environment type associated with the environment. In some examples, such as Figure 7As illustrated, the projection distance is the projection distance to the baseline path. In some examples, the projection distance is the projection distance to each obstacle (e.g., to each agent). For example, each agent is associated with a projection distance. In some examples, the projection distance is a projection distance related to the starting and ending points of the agent projection used to understand how much space such an obstacle occupies. In some examples, the projection distance is the projection distance to the nearest projection point that defines the most constrained point of the agent relative to the baseline projection (which can be used for clearance). In one or more examples or embodiments, the agent type associated with the agent includes one or more of a vehicle, a pedestrian, and a cyclist.

[0093] In one or more examples or embodiments, the environment type includes one or more aspects of the environment that can be related to the operation of the autonomous vehicle by the system. In some examples, the environment type can include one or more aspects of the environment that can be related to the movement of the AV in response to obstacle data and corresponding constraints to select a trajectory or constrain the AV. The environment type can indicate a drivable area and / or an urban area, for example. The environment type indicates a more specific type of environment, such as a highway, a surface street, etc. In one or more examples or embodiments, the system 500 is configured to determine the environment type corresponding to the area (e.g., a drivable area, a highway, a pick-up and unloading (PuDo) area) where the agent is located. In some examples, the projection of the system 500 captures metadata to generate the environment type. The metadata can include, for example, the semantics of the projection of the agent, which direction and / or orientation the agent is in the environment, etc. In some cases, the metadata can be used to add semantics to the determined station constraint and / or the determined lateral constraint. In some cases, the metadata can be used to modify the station constraint and / or lateral constraint determined based on the dynamic trace. In some examples, metadata may include an agent type for the agent and / or an environment type for the environment in which the AV and the agent operate. In some such examples, the constraint generation system 508 may generate a station constraint and / or a lateral constraint for the agent taking into account the agent type and the environment type. In some other examples, the trajectory generation system 510 and / or the trajectory selector system may generate or select a trajectory based at least in part on the agent type, environment type, or other semantics derived or extracted from the metadata. In some examples, the system 500 is configured to handle the agent in different ways. For example, when the agent is in open driving vs PuDo driving, different gaps relative to the agent may be advantageous. For example, in the PuDo scenario, the autonomous vehicle is usually driven very slowly, so in most cases, the system 500 may be configured to operate the autonomous vehicle to keep closer to pedestrians or vehicles (e.g., agents). But on highways (e.g., open driving), the system 500 is configured to operate the autonomous vehicle to not keep so close to pedestrians and / or vehicles. In some cases, metadata may be included as semantic information, for example, along with station constraints and / or lateral constraints to be used for further processing (eg, with a trajectory generation system, trajectory selector system, and / or control system 513).

[0094] In some examples, the projected distance is used to determine the constraints on the autonomous vehicle. In one or more embodiments or examples, determining the station constraint 508a and the lateral constraint 508a includes determining the station constraint 508a based on the projected distance. In one or more examples or embodiments, determining the station constraint 508a and the lateral constraint 508a includes determining the lateral constraint 508a based on the projected distance. For example, the station projection onto the first trajectory provides one or more projected distances. In some examples, the system 500 is configured to select the shortest distance as the projected distance for determining the station constraint 508a. In other words, the station projection is based on the obstacle decision option (e.g., obstacle action or behavior, e.g., where the agent is going), for example, if the system 500 determines that the autonomous vehicle should pass after the obstacle, the system 500 is configured to generate the station constraint 508a before the obstacle. For example, the distance of the dynamic trace associated with the agent is projected onto one or more portions of the first trajectory, for example, onto one or more transition segments of the route planning. In one or more examples or embodiments, there is also a lateral homotopy, so the system determines that the autonomous vehicle should pass to the left or right based on the projection of the leftmost point or the rightmost point of the object.

[0095] In one or more examples or embodiments, generating a second trajectory based on a position constraint and a lateral constraint includes: determining homotopy based on the position constraint and the lateral constraint. Homotopy can be regarded as a class describing a set of trajectories having the same starting place and the same ending place, for which there is a continuous deformation from one to another while remaining within the class. In one or more examples or embodiments, homotopy is regarded as describing a set of constraints, and a trajectory is generated according to the set of constraints based on a cost function and / or other parameters of the trajectory. In other words, homotopy can be regarded as a corridor in space and time. In some examples, homotopy can be regarded as one or more constraints applied to the potential trajectory of the vehicle. In some examples, these constraints are applied in 2D space, such as in x and y coordinate systems, etc. In some examples, these constraints are position constraints and / or lateral constraints (e.g., spatial constraints and / or spatial lateral constraints). In other words, homotopy can limit the scope of the potential trajectory by taking into account the constraints imposed by any obstacles (e.g., any object, agent) in the environment. In one or more examples or embodiments, the trajectory generation system 510 determines homotopy based on position constraints and lateral constraints.

[0096] In one or more examples or embodiments, generating a second trajectory based on the station constraint and the lateral constraint includes: generating a second trajectory of the autonomous vehicle based on homology. In some examples, the homology includes a set of trajectories, such as a plurality of trajectories, etc. The system can be configured to generate a plurality of homologous and select one of the plurality of homologous as the basis for the second trajectory. In some examples, the constraint generation system 508 generates the station constraint 508a and / or the lateral constraint 508a. In one or more embodiments or examples, the constraint generation system 508 provides the station constraint 508a and / or the lateral constraint 508a to the trajectory generation system 510. In one or more examples or embodiments, the trajectory generation system 510 generates a second trajectory 510b of the autonomous vehicle based on the homology.

[0097] In one or more examples or embodiments, the system 500 is configured to generate trajectories from each homotopy via a control optimization method (e.g., model predictive control). In one or more examples or embodiments, the system 500 (e.g., via model predictive control) uses the constraints contained in a given homotopy (e.g., station constraints 508a and / or lateral constraints 508a) to determine an optimized trajectory (e.g., a dynamically feasible trajectory) based on specific target parameters. In some examples, the system 500 is configured to generate trajectories from each homotopy (e.g., from a trajectory generation system 510), and then the trajectories are fed into a ranking system (e.g., a trajectory selector system 512) to score the trajectories. In some examples, the system 500 generates multiple homotopies specific to the decisions and / or constraints made for each projected trace on the route plan, and each homotopy in these homotopies describes a set of constraints. In some examples, the system 500 generates optimized trajectories from each homotopy, and the final result is a plurality of trajectories optimized from a plurality of homotopies that can be selected in the trajectory selector system 512. For example, since the homotopy specifies corridors of space on which the vehicle can operate, there is no guarantee that there are dynamically feasible (eg, taking into account the vehicle's motion constraints) trajectories within the constraints.

[0098] In one or more examples or embodiments, determining the station constraint and the lateral constraint includes: the determination may be further based on the agent type. For example, the constraint generation system 508 may determine the station constraint 508a and the lateral constraint 508a based on the agent type of the obstacle data. In other words, the agent type may be related to the constraints determined by the system. For example, the system 500 is configured to determine a different set of constraints when faced with obstacle data indicating a truck than when faced with obstacle data indicating a bicycle.

[0099] In one or more examples or embodiments, determining the station constraint and the lateral constraint is further based on the environment type. For example, the constraint generation system 508 can determine the station constraint 508a and the lateral constraint 508a based on the environment type of the obstacle data. In other words, the environment itself can provide potential constraints for the operation of the autonomous vehicle using the system 500. Examples of environment types include one or more types indicating the following items: drivable area, highway, PuDo, dirt road, and urban road. As an example, when the environment type indicates a highway, the system 500 can apply different constraints compared to the environment type indicating a dirt road.

[0100] FIG. 6A to FIG. 6B 6 is a diagram of an example implementation of a process for obstacle representation, such as that performed by vehicle 602. Vehicle 602 may be an autonomous vehicle and may be computed by AV computing 640 (which may be associated with Figure 5 AV computing 540 and / or system 500) is the same or similar to control. Fig. 6A As shown, in some examples, the AV computing 640 of the vehicle 602 is configured to obtain sensor data (613). The AV computing 640 is configured to take, for example, Figure 5 Again, the actions discussed above are referred to, and using the planning system 604a (which is Figure 5 Planning System 520 and / or Figure 4 The planning system 404 of the same or similar) generates data associated with the second trajectory and transmits the data to the control system 604b (which can be connected to the control system 604b) Figure 5 The control system 513 and / or Figure 4 control system 408 (same or similar) (616), such as for operation of vehicle 602. Figure 6B To illustrate a further implementation, the AV computing 640 may transmit control signals, such as for controlling the operation of the vehicle 602, from the control system 604b to the DBW system 606 (620).

[0101] Figure 7 is a diagram of an example projection 700 of an obstacle onto a path. In other words, Figure 7 The generation of obstacle data is illustrated. Figure 7 An obstacle is shown, illustrated as agent 702. For example, the obstacle can be a polygonal shape and is projected onto a baseline. Projecting an obstacle (such as agent 702, etc.) onto a baseline includes projecting points from the polygon of agent 702 onto a baseline 708 illustrating a first trajectory of the AV. In some examples, baseline 708 is the center of a lane, where an upper lane boundary is illustrated as 712A and a lower lane boundary is illustrated as 712B. For example, generating obstacle data (e.g., Figure 5Obstacle data 502a) includes information from a perception system (e.g., Figure 5 of the perception system 502) and the prediction system (e.g., Figure 5 The prediction system 504 of the present invention converts the dynamic trace of the AV into obstacle data. In one or more examples or embodiments, the agent 702 is an obstacle to the AV trajectory. For example, points 708A and 708B are the beginning and end of a lane and / or a portion of a lane, respectively. For example, the AV interacts with an obstacle (e.g., agent 702) between points 704A and 704B. In other words, the obstacle exists in the surrounding environment of the trajectory of the AV between points 704A and 704B. The distance 710 (e.g., between points 704A and 704B) is, for example, the distance along the lane that the AV interacts with the obstacle. In other words, points 704A and 704B are generated from the projection of the points of the polygon of the agent corresponding to, for example, the starting point and the ending point of the polygon (e.g., the starting edge and the ending edge of the polygon). For example, points 704A, 704B and the distance 710 generated from such points can be used by the disclosed system to determine the size of the agent. In some examples, points 704A, 704B are used to generate station constraints, such as constraints that allow the AV to make decisions (e.g., stop and / or pass behind and / or pass ahead and / or overtake) based on the agent's actions along the first trajectory of the AV. For example, point 706A is the shortest lateral gap. In other words, point 706A is a projection of point 706B that generates the shortest distance to the baseline (such as distance 713, etc.). For example, point 706B is the most constrained point of the agent relative to the baseline and is used to generate a spatial gap constraint. Generating a spatial gap constraint involves, for example, projecting the distance (e.g., the shortest distance relative to the baseline) of an obstacle (e.g., agent 702) of the dynamic track to a portion of the first trajectory (which corresponds to, for example, one or more transition segments of the route planning).

[0102] FIG. 8A to FIG. 8C 800, 820 are examples of examples of generating station constraints according to one or more embodiments of the present disclosure using system 500. In other words, FIG. 8A to FIG. 8C The generation of station constraints is illustrated. Specifically, Fig. 8A As an example, an autonomous vehicle 802 including the system 500 is approaching a crosswalk 806, and a pedestrian 812 (such as an agent) intends to cross the crosswalk 806. The autonomous vehicle 802 obtains sensor data (such as Figure 5 sensor data 506, etc.). FIG. 8B to FIG. 8C The example is based on the movement of pedestrian 812 and / or its predicted position-time (S0, S1, S2 represent also FIG. 8B to FIG. 8C The various stations of AV shown in the figure, T1, T2, T3 are also FIG. 8B to FIG. 8CFor example, the action of pedestrian 812 may be before autonomous vehicle 802 reaches crosswalk 806 or after autonomous vehicle 802 moves across crosswalk 806, pedestrian 812 crossing crosswalk 806.

[0103] Figure 8B The example is Fig. 8A Autonomous vehicle 802 reaches the boundary 806A Fig. 8A 806 and when pedestrian 812 crosses crosswalk 806. For example, autonomous vehicle 802 may pass through the crosswalk after the pedestrian crosses the crosswalk by increasing its speed and / or acceleration. Figure 8C The example is Fig. 8A Autonomous vehicle 802 moves across Fig. 8A After the pedestrian crossing 806, Fig. 8A 812 pedestrians crossed Fig. 8A ST analysis of the pedestrian crossing 806. For example, Fig. 8A The autonomous vehicle 802 may wait for the pedestrian 812 to cross the crosswalk by reducing its speed and / or by stopping before reaching the crosswalk. In some examples, the generation of the spatial constraint is based on the actions taken by obstacles (e.g., agents) such as pedestrian 812. For example, Fig. 8A Area 808 indicates an area that the autonomous vehicle may occupy when pedestrian 812 crosses crosswalk 806 before autonomous vehicle 802 reaches crosswalk 806. For example, Fig. 8A The area 810 indicates the area that a pedestrian 812 may occupy when crossing the crosswalk 806 after the autonomous vehicle 802 moves across the crosswalk 806. In other words, Fig. 8A The regions 808, 810 in are, for example, zones occupied by pedestrian 812 and / or autonomous vehicle 802 associated with actions taken by pedestrian 812, where such actions and zones are used to generate spatial constraints. FIG. 8B to FIG. 8C Regions 808A and 810A are respectively decision regions generated after the generation of the spatial constraints. FIG. 8A to FIG. 8C Region 804 depicts the area where the pedestrian contacts the autonomous vehicle when the pedestrian and the autonomous vehicle decide to cross the crosswalk at the same time. For example, crosswalk 806 includes a lower boundary (e.g., at Figure 8B 806A) and an upper boundary (e.g., Figure 8C 806B). Figure 8B The station S0 at times T2 and T3 shows an example illustration when the AV 802 passes after the pedestrian 812 crosses the crosswalk. Figure 8CThe stations S1 , S2 at times T2 and T3 show example illustrations when the AV 802 passes before the pedestrian 812 crosses the crosswalk.

[0104] Fig. 9 9 is a diagram illustrating an example of a lateral constraint generated 900 using system 500 in accordance with one or more embodiments of the present disclosure. In other words, Fig. 9 Illustrate the generation of spatial constraints. Fig. 9 Example of station-lateral clearance (SL) analysis. Fig. 9 An autonomous vehicle 906 is illustrated moving across lanes, obtaining sensor data (such as Figure 5 Sensor data 506, etc.). For example, 902A, 902B, 902C are the first agent (e.g., obstacle) at T seconds, T+1 seconds, and T+2 seconds, respectively. For example, 904A, 904B, 904C are the second agent (e.g., obstacle) at T seconds, T+1 seconds, and T+2 seconds, respectively. In some examples, the autonomous vehicle 906 moves through a lane that considers the first agent and the second agent as obstacles. For example, due to the interaction of the autonomous vehicle 906 with the first agent and the second agent, spatial constraints (e.g., spatial constraints 902AA, 902BB, 902CC, 904AA, 904BB, 904CC for the first agent 902A, 902B, 902C and the second agent 904A, 904B, 904C, respectively) are generated. For example, the autonomous vehicle can change direction (e.g., move left or right) based on the projection of the obstacle along the baseline path based on the dynamic trace of the obstacle. In other words, the autonomous vehicle can change direction based on the projection of the leftmost or rightmost points of the first agent and the second agent when the positions of the first agent and the second agent change relative to the baseline path. For example, the leftmost or rightmost point of the obstacle is Figure 7 The projection of point 706B in the image relative to the baseline is Figure 7 For example, the leftmost or rightmost point of the obstacle defines (e.g., generating Figure 7 The projection point 706A Figure 7 The shortest lateral gap distance 712) is the most constrained point of the obstacle, which is used to define the lateral constraint (e.g., the spatial gap or lateral constraint 902AA, 902BB, 902CC, 904AA, 904BB, 904CC).

[0105] Reference now Fig.10, illustrating a flow chart of a method or process 1000 for determining constraints associated with one or more obstacles (e.g., dynamic obstacles) and generating a trajectory based on the determined constraints, wherein the trajectory can be used to operate and / or control an AV in the presence of the obstacles. The method can be implemented by a system disclosed herein (such as Figure 2 AV Computing 202F and Figure 4 AV Computing 400, Figure 1 The vehicle 102 and Figure 2 200 vehicles, Figure 3 The device 300, and Figure 5 Systems 500 and AV Computing 540 and Figures 6A to 6B and Figures 7 to 9 The disclosed system may include at least one processor that may be configured to perform one or more than one of the operations of method 1000. Method 1000 may be performed by another device or device group (e.g., completely and / or partially, etc.) separate from the system disclosed herein or including the system disclosed herein.

[0106] In one or more embodiments or examples, method 1000 includes: at step 1002, using at least one processor to obtain sensor data. In one or more embodiments or examples, the sensor data indicates an agent in an environment in which an autonomous vehicle (AV) is configured to operate along a first trajectory (e.g., a baseline path). In some examples, the agent may be an object in the environment around the AV that may have dynamic behavior. In some examples, the sensor data may indicate one or more agents such as one or more of the following: a vehicle, a pedestrian, and a cyclist.

[0107] In one or more embodiments or examples, method 1000 includes: At step 1004, using at least one processor, determining a dynamic trace associated with an agent. For example, a dynamic trace is a prediction of an agent's trajectory as a function of time. For example, a dynamic trace tracks an agent over time (e.g., an agent trajectory). The prediction is, for example, based on the speed of the agent in one or more previous frames (e.g., a time frame) to predict where the agent is moving. The dynamic trace is, for example, a prediction of the trajectory of the agent over time and space.

[0108] In one or more embodiments or examples, method 1000 includes: at step 1006, using at least one processor, generating obstacle data associated with the agent based on the dynamic trace. In one or more embodiments or examples, the obstacle data indicates that the agent is an obstacle along the first trajectory of the autonomous vehicle. In one or more embodiments or examples, the obstacle data provides one or more parameters that characterize the agent as an obstacle to the trajectory of the AV. Generating the obstacle data includes, for example, converting the dynamic trace from a perception system (e.g., sensor data) and a prediction system into the obstacle data.

[0109] In one or more embodiments or examples, method 1000 includes: at step 1008, using at least one processor, determining a station constraint and a lateral constraint based on obstacle data to apply to the trajectory or generate a safe trajectory that does not intersect the dynamic trajectory of the agent. In one or more embodiments or examples, the station constraint is a station maneuver description characterized (e.g., parameterized) in time and space. The station constraint is defined, for example, by the upper space / station limit and the lower space / station limit of the corridor in which the AV should stay. The station constraint may include a constraint applied to the longitudinal maneuver of the vehicle (e.g., changing speed along the baseline path). In some examples, the station constraint may be a spatiotemporal constraint (e.g., a time-varying station). The lateral constraint may be a constraint associated with the lateral distance of the agent relative to the baseline path. In some cases, the lateral constraint may be characterized in the station-lateral (SL) domain. In some cases, the lateral constraint may be characterized in the station-time (ST) domain by defining the spatiotemporal boundaries of the agent. In some examples, the lateral constraint may be a lateral gap with the station. In some cases, obstacle data is used to generate constraints that represent where the AV can be along a first trajectory (station) and how much space the AV has relative to its sides (laterally) at a specific time in the time frame. Actions such as acceleration, deceleration, and when to stop can be designed based on the clearance available along the path (such as a lane) that the AV will interact with.

[0110] In one or more embodiments or examples, method 1000 includes: at step 1010, using at least one processor, generating a second trajectory of the autonomous vehicle based on the station constraint and the lateral constraint. In some examples, the second trajectory can be different from or the same as the first trajectory. The second trajectory is, for example, a corridor. In one or more embodiments or examples, method 1000 includes: at step 1012, using at least one processor to provide data associated with the second trajectory. In one or more embodiments or examples, the data associated with the second trajectory is configured to cause the autonomous vehicle to operate along the second trajectory.

[0111] In one or more embodiments or examples, generating obstacle data at step 1006 includes determining a projection of a representation of the agent onto the first trajectory. Figure 7 As illustrated, the agent can be parameterized as a polygon, and generating obstacle data includes projecting selected points (e.g., vertices) of the polygon onto a baseline path of the first trajectory (e.g., the center of a lane forming part of the first trajectory). In one or more embodiments or examples, the projection of the position onto the first trajectory (e.g., the projection of the polygon-shaped agent) provides one or more projection distances. For example, the shortest distance is selected as the projection distance. In some examples, the baseline path is not strictly a path that the AV will take (such as two parallel lanes, etc.). Using the projection points, the trajectory of the agent relative to two paths (such as two parallel lanes, etc.) can be determined without having to decompose the trajectory of the agent into time, position, and lateral clearance parameters (e.g., treating obstacle avoidance as a 3D dimensional problem). In some cases, generating obstacle data may include determining the distance traveled along the path based on the projection of the agent (such as as a Figure 7 In some examples, generating constraints (such as station constraints and lateral constraints, etc.) includes generating constraints within the determined distance traveled along the path distance (rather than the distance associated with the time range in which the AV is configured to operate). In other words, generating constraints includes generating constraints for the time intervals in which the agent is detected by the AV. In some examples, the baseline path is the center of the lane. In some other cases, for example, when the lane is very wide, the baseline path can be closer to the left or right boundary of the lane. In some examples, the trajectory is in 2D space.

[0112] In one or more embodiments or examples, generating obstacle data at step 1006 includes generating obstacle data including one or more of the following items: a projected distance, an agent type associated with the agent, and an environment type associated with the environment surrounding the agent and / or the AV. For example, the projected distance can be a distance relative to a baseline path. In some cases, an agent polygon can be estimated for the agent, where the agent polygon represents the spatial boundary of the agent. In some such cases, the projected distance can be generated by determining the lateral distance of the vertices of the agent polygon relative to the baseline path associated with the first trajectory. Additionally or alternatively, the projected distance can be generated by a normal projection of the vertices of the agent polygon on the baseline path associated with the first trajectory. The normal projection can include projecting a point (e.g., a vertex of the agent polygon) onto the baseline path along a direction normal (perpendicular) to the baseline path.

[0113] For example, each obstacle (e.g., each agent) is associated with a plurality of projection distances. For example, the projection distance is the shortest projection distance relative to the baseline path. Distances (such as the projection distances associated with the starting and ending points (such as the starting point 704A and the ending point 704B) of the agent projection are also associated with the distances (such as the projection distances associated with the starting and ending points (such as the starting point 704A and the ending point 704B) of the agent projection. Figure 7 The distance 710, etc.) is used, for example, to determine the width of an agent (e.g., how much space such an agent (such as an obstacle, etc.) occupies. The distance associated with the closest projected point relative to the baseline, for example, defines the most constrained point of the agent and is used, for example, for lateral clearance. For example, the distance associated with the closest projected point relative to the baseline is the projected distance.

[0114] In some embodiments, metadata can be combined with the determined projection of the agent or attached to the determined projection of the agent. In some examples, metadata includes agent type and / or environment type. In some examples, agent type includes vehicles and / or pedestrians and / or cyclists. In some examples, the environment type corresponds to the area where the agent is located (e.g., drivable area, highway, PuDo). In some cases, metadata attached to the projection of the agent can be used to determine the type of driving behavior taken by AV. In some examples, metadata can be attached to the projection of the agent as semantics (e.g., the projection of the agent to the station-time domain or the station-lateral-time domain). In some examples, the presence of agents (e.g., obstacles) near or in different types of environments can make it possible to process agents (e.g., projections of agents) in different ways to generate station constraints and lateral constraints. For example, when the agent is in open driving vs pick-up and unloading (PuDo) driving, different gaps to the obstacle can be generated. For example, in the PuDo scenario, the AV drives very slowly, which implies that the AV can stay closer to pedestrians, vehicles, or cyclists (e.g., agents) for most of the driving time. However, on highways (e.g., open driving), driving close to pedestrians and / or vehicles is undesirable. In some cases, metadata can be provided to the constraint generation system 508 and can be used to generate accurate station constraints and lateral constraints. In some examples, the metadata attached to the agent's projection can be transmitted to other processing systems of the AV for further processing.

[0115] In one or more embodiments or examples, determining the station constraint and the lateral constraint at step 1008 includes determining the station constraint based on a projected distance. In one or more embodiments or examples, determining the station constraint and the lateral constraint at step 1008 includes determining the lateral constraint based on a projected distance. For example, one or more projected distances are provided relative to a baseline projected agent (e.g., a polygonal shape). For example, the station projection onto the first trajectory provides one or more projected distances. In other words, the station projection is based on agent actions along the first trajectory of the AV (e.g., obstacle decision options). For example, the station constraint allows the AV to make decisions (e.g., stop and / or pass after and / or pass before and / or overtake) based on agent actions along the first trajectory of the AV. For example, when the AV intends to pass through a crosswalk after an obstacle (e.g., a pedestrian) crosses the crosswalk, a station constraint is implied to force the AV to slow down or even stop (e.g., Figure 7 B). For example, the shortest distance is selected as the projected distance for determining the lateral constraint. For example, the autonomous vehicle may change direction (e.g., move left or right) based on the projection of the obstacle along the dynamic track relative to the baseline. In other words, the autonomous vehicle may change direction based on the projection of the leftmost or rightmost point of the obstacle. For example, the leftmost or rightmost point of the obstacle defines (e.g., generates a distance to the horizontal constraint). Figure 7 The projection point 706A Figure 7 The shortest lateral gap distance 712) is the most constrained point of the obstacle, which is used to define the spatial gap constraint.

[0116] In one or more embodiments or examples, generating the second trajectory based on the station constraint and the lateral constraint at step 1010 includes: determining a homotopy (e.g., a set of trajectories) based on the station constraint and the lateral constraint. In one or more embodiments or examples, generating the second trajectory based on the station constraint and the lateral constraint at step 1010 includes: generating a second trajectory for the autonomous vehicle based on the homotopy.

[0117] In one or more embodiments or examples, determining the station constraint and the lateral constraint at step 1008 is further based on the agent type. In one or more embodiments or examples, determining the station constraint and the lateral constraint at step 1008 is further based on the environment type.

[0118] In the previous description, aspects and embodiments of the present disclosure have been described with reference to many specific details, which may vary depending on the implementation. Therefore, the description and the accompanying drawings should be regarded as illustrative, not restrictive. The only and exclusive indication of the scope of the invention, and the applicant's expectation that the content of the scope of the invention is the literal and equivalent scope of the claims issued from this application in the specific form of the claims, including any subsequent amendments. Any definition of the terms used to be included in such claims that are clearly set forth herein should be based on the meaning of such terms as used in the claims. In addition, when the term "also includes" is used in the previous description or the attached claims, the following of the phrase may be an additional step or entity, or a sub-step / sub-entity of the previously described step or entity.

[0119] A non-transitory computer-readable medium is disclosed, comprising instructions stored thereon, which, when executed by at least one processor, cause the at least one processor to perform one or more operations according to the method disclosed herein.

[0120] Example Autonomous Vehicle (AV) Computing System

[0121] In some cases, the AV computing 540 can be configured to provide a trajectory used by the control system 513 to avoid collisions or harmful interactions with agents in the environment when navigating the AV in the environment. In some cases, the control system 513 can receive commands to move the AV along a baseline path. In some cases, the baseline path can be a path determined based on a starting point and a destination and one or more roads or streets connecting the starting point to the destination. In some examples, the baseline path can be the centerline of a lane or road.

[0122] In some cases, the control system can operate the AV along a first trajectory associated with the baseline path. In some cases, the first trajectory can be a trajectory generated by the planning system 520, or an initial trajectory generated by another system of the AV based on the baseline path and certain predetermined constraints (e.g., station constraints) associated with the baseline path (e.g., boundaries and lanes of a road or certain structures and / or obstacles along the road).

[0123] In some cases, the perception system 502 can be configured to receive sensor data 506 from a sensor (e.g., a camera or LiDAR) and use the received sensor data 506 to generate obstacle data 502a indicating one or more agents (e.g., agents near the baseline path) that can potentially interact with the AV. In some cases, these agents can be dynamic and change their positions over time. As such, the obstacle data can be time-dependent and, in some cases, collected sequentially (e.g., at equal time intervals) to capture changes in the location and / or speed of the agents (e.g., relative to the baseline path).

[0124] In some cases, the prediction system 504 may use obstacle data to predict the dynamic trace 504a of the agent. In some examples, the dynamic trace 504a may be a predicted trajectory of the agent in space and time. In some implementations, the prediction system 504 may extract the dynamic trace 504a of the agent from a sequence of frames or data points captured by a sensor (e.g., a camera or LiDAR of an AV) at continuous time steps during a measurement time interval. For example, the prediction system 504 may use previously received sensor data to estimate the position, velocity (e.g., linear velocity or angular velocity) and / or direction of motion (or rotation) of the agent, and then use the estimated position, velocity and / or direction of motion to determine or predict the future location of the agent at a later time. In some implementations, the prediction system 504 determines the position, velocity, direction of motion, and future location of the agent relative to the baseline path.

[0125] In some implementations, planning system 520 may capture or receive metadata indicating characteristics of an agent or environment. In some examples, such metadata may be used to generate or customize constraints generated to avoid collisions or interactions with an agent.

[0126] In some cases, metadata may be generated by the perception system 502 based on the sensor data 506. In some cases, metadata may be generated based at least in part on data stored in a memory of the planning system 520 or received from another system of the AV. In some examples, metadata may be generated by comparing the sensor data 506 with reference data stored in a memory of the AV. In some cases, the perception system 502 may generate metadata and append the metadata to the obstacle data 502a. In various implementations, the metadata may include an agent type, an environment type, or both. In some cases, the planning system 520 may generate metadata based at least in part on a dynamic trajectory 504a predicted for the agent.

[0127] In some cases, the constraint generation system 508 can receive the obstacle data 502a and the dynamic trace 504a and generate constraints 508a, which can be used to determine the spatiotemporal zone occupied by the agent and / or the safe spatiotemporal zone that the AV can occupy without interacting or colliding with the agent. In some cases, the constraints 508a can be determined relative to time, a longitudinal direction (e.g., a direction along the baseline path and / or along the direction of motion of the VA), and / or a lateral direction that can be substantially perpendicular to the longitudinal direction and parallel to the ground. In some examples, the longitudinal constraint can indicate the distance between the AV and the agent along the baseline path at a specific time and is referred to as a station constraint. In some examples, the lateral constraint indicates the lateral distance between the AV and the agent at a specific time. In some examples, the lateral constraint is determined based on the minimum lateral distance between the AV and the agent.

[0128] In some cases, the constraint generation system 508 may use a set of generalized parameters to represent the agent, and use the generalized parameters to determine the lateral constraints and the station constraints. For example, the generalized parameters may include a polygon representing the boundary of the agent (also referred to as the agent polygon) so that overlap with the polygon in the station-lateral domain is avoided, which prevents interaction or collision with the agent. In some cases, the generalized parameters may be generated by the perception system 502 and may be provided to the constraint generation system 512 (e.g., as part of the obstacle data 502a). In some cases, the station constraints may be generated based on the projection of the polygon on the baseline path. For example, some of the vertices of the polygon may be projected to points along the baseline, and these points may be used as station constraints or for generating station constraints. In some cases, the constraint generation system 508 may use the projection of two vertices of the polygon defining the longitudinal boundary of the agent and / or the projection of the third vertex with the shortest lateral distance (e.g., normal lateral distance) relative to the baseline path to generate the station constraints and lateral constraints. In some cases, the third vertex may be referred to as the most constrained point of the agent.

[0129] In some examples, the constraint generation system 508 generates constraints by determining the constraint boundaries of the agent in the station, lateral, and time (SLT) domain. In some examples, the constraint generation system 508 generates constraints by determining the constraint boundaries of the agent in the station and time (ST) domain based at least in part on the determined lateral constraints. In some cases, the constraint generation system 508 generates corridors in the station and lateral (SL) domain. In some cases, considering the dynamic nature of the agent, the corridors in the SL domain can be time-dependent. In some cases, time-dependent corridors can be used to generate constraint boundaries for the agent in the ST domain. It should be understood that the constraint boundaries of the agent in the SLT domain can be larger than the actual physical boundaries of the agent because the constraint generation system 508 can determine larger boundaries based on safety requirements, limitations of the AV and / or corresponding metadata.

[0130] In some cases, constraint generation system 508 may generate station constraints and lateral constraints based at least in part on metadata associated with obstacle data 502a. In some cases, metadata may be attached to obstacle data 502a and received from perception system 502. In some examples, constraint generation system 508 may generate station constraints and lateral constraints for an agent based on the agent type and / or the environment surrounding the agent. In some cases, the agent type and / or the environment type may affect the determination of the constraint boundaries of the agent in space and time.

[0131] In some implementations, once the constraint boundaries of the agent are determined in the spatiotemporal domain (e.g., the station-time domain), the trajectory generation system 510 can generate a safe spatiotemporal zone based on the determined constraint boundaries of the agent in the spatiotemporal domain. In some cases, the trajectory generation system 510 can generate a safe spatiotemporal zone based on the station constraints and lateral constraints received from the constraint generation system 508. The safe spatiotemporal zone can be a zone in the ST domain (a zone in a station-time map) that can be occupied by an AV. Additionally, in some cases, the trajectory generation system 510 can generate a safe spatiotemporal zone based on metadata (e.g., received from the constraint generation system 508). Next, the trajectory generation system 510 can generate one or more trajectories (e.g., safe trajectories) that pass through the safe spatiotemporal zone. The safe trajectory can be a spatiotemporal trajectory of the AV that does not intersect the spatiotemporal trajectory of the agent.

[0132] In some cases, projecting the selected points of a parameterized dynamic agent (e.g., the vertices of any agent polygon) along a baseline path and determining the lateral distance relative to the baseline path can be referred to as projecting the agent into a station-lateral-time (SLT) domain. In some cases, generating constraints (e.g., lateral constraints and station constraints) can include an analysis of the projection of the agent into the station-lateral-time (SLT) domain. In some implementations, such an analysis can be decomposed into an analysis in the station-lateral (SL) domain, and the results in the station-time domain (ST) analysis are used to generate the projection of the agent in the (SLT) domain. In these cases, the lateral constraint can be embedded in the station-time domain (ST) as a constraint boundary of the agent projection. In some embodiments, the result of projecting the agent in the (SLT) domain can be an ST diagram, which includes an agent ST zone corresponding to the constraint boundary and a safe ST zone available for AV navigation. In some cases, the agent ST zone can be determined at least in part based on metadata (e.g., agent type and / or environment type). In some cases, trajectory generation system 510 generates homotopies based on safe ST zones, and trajectory selector system 512 selects a trajectory (eg, a safe trajectory) based on the homotopies and provides the selected trajectory to control system 513 for navigating the AV along the trajectory.

[0133] Fig.11An example station-lateral constraint and station-time constraint analysis based on sensor data associated with an agent moving toward a road along which an AV is moving is illustrated. As shown in SL diagram 1100, the road is defined by two road boundary lines 1104 (a) and road boundary line 1104 (b), and the AV is initially moving along a baseline path 1106 (e.g., the centerline of the road defined by boundary lines 1104 (a) and 1104 (b)). The agent is represented as an agent polygon 1101 that defines the spatial boundary of the agent. As indicated by the agent's velocity vector (v), the agent (and thus the agent polygon 1101) moves along a longitudinal direction (station direction S) as its lateral distance (e.g., normal lateral distance) to the baseline path 1106 decreases. SL diagram 1100 depicts the position of the agent polygon 1101 at three different times T1, T2, and T3 and the projections (e.g., normal projections) of the three vertices of the agent polygon 1101 onto the baseline path 1106. The two vertices define the projected length of the polygonal agent, and a third vertex 1122 located between the first vertex and the second vertex is the most constrained vertex with the closest lateral distance 1120 to the baseline path 1106. In some cases, the lateral position of the third vertex 1122 can be used to generate a stance constraint and a lateral constraint, and the first vertex and the second vertex can further restrict the movement of the AV along the longitudinal direction (along the stance direction). In the example shown, the first vertex is projected onto the baseline path 1106 at times T1, T2, and T3 as points 1108 (a), 1108 (b), and 1108 (c), respectively, and the second vertex is projected onto the baseline path 1106 at times T1, T2, and T3 as points 1112 (a), 1112 (b), and 1112 (c), respectively. Similarly, the third vertex 1122 is projected onto the baseline path 1106 at times T1, T2, and T3 as points 1110 (a), 1110 (b), and 1110 (c), respectively. In this way, the two-dimensional motion of the agent is reduced to one-dimensional motion of projection points 1108(a), 1110(a), and 1112(a) along the baseline path 1106. In some cases, the motion of one or more of these projection points can be used to generate a station constraint, and the corresponding lateral distance can be used to generate a lateral constraint. The SL graph 1100 and the corresponding analysis can be used to generate a corresponding ST graph for generating a safe trajectory (e.g., a safe homology) used by the AV. In some cases, a safe trajectory can be a trajectory that does not intersect the trajectory of the agent at any point in time. The longitudinal positions of the projected vertices of the agent polygon 1101 can be represented as traces 1114, 1116, and 1118 on the ST graph 1102 (in this example, the velocity of the corresponding agent is assumed to be constant).

[0134] In some cases, the constraint generation system 508 can determine the agent zone on the ST graph 1102 based on the following items: the evolution of the lateral position of the agent polygon; the criteria for constraint generation; and in some cases metadata associated with the corresponding agent and / or environment. In some cases, the criteria for constraint generation can include the shortest lateral distance of the agent polygon 1101 relative to the baseline path 1106. For example, when the lateral distance (e.g., normal lateral distance) 1120 from the most constrained vertex 1122 to the baseline path 1106 becomes less than a first threshold, the AV should slow down, and when the lateral distance 1120 becomes less than a second threshold that is smaller than the first threshold, the AV should stop to avoid colliding with the agent. As another example, constraints on the movement of the AV can be imposed based on the portion of the agent polygon 1101 that overlaps with the road. In some examples, the overlapping portion can be quantified as the distance between the intercepts where the agent polygon intersects the right boundary line 1104 (a) of the road. Therefore, a constrained agent zone 1124 on the ST diagram can be defined so that the AV stops at the first longitudinal location where the agent polygon 1101 extends beyond the right road boundary line 1104 (a).

[0135] In some cases, the constraint agent zone can extend beyond the constraint agent zone 1124 that depicts the spatiotemporal zone associated with the constraint criteria. In some examples, the constraint agent zone can be extended beyond the spatiotemporal zone associated with the constraint criteria based on the rate at which the agent is moving toward the baseline path, various safety measures or metadata, and other parameters and factors. In this way, the constraint agent zone (such as the constraint agent zone 1124, etc.) on the ST graph 1102 can capture certain aspects of both the station constraint and the lateral constraint. Therefore, the ST graph 1102 can be used by the trajectory generation system 510 to generate a safe trajectory used by the AV. In some examples, the safe trajectory for the ST graph 1102 can be a spatiotemporal trajectory that does not intersect with the constraint agent zone 1124. The safe trajectory can constitute a homology, and the trajectory selector system 512 can use additional criteria (including criteria associated with metadata) to select a preferred safe trajectory from the homology and provide the preferred safe trajectory to the control system 513, and cause the AV to adjust its trajectory from the first trajectory to the preferred trajectory to avoid any interaction with the agent. In various implementations, the safe trajectory may be a trajectory in the spatiotemporal (e.g., station-time) domain. Thus, controlling the AV based on the preferred safe trajectory may include decelerating, accelerating, stopping, and / or changing lateral position relative to the road boundary lines 1104(a) and 1104(b), among other actions.

[0136] Example Embodiments

[0137] The example embodiments described herein have several features, no single one of which is essential or solely responsible for its desirable attributes. Various example systems and methods are provided below.

[0138] Also disclosed are methods, non-transitory computer readable media, and systems according to any of the following:

[0139] Example 1. A method comprising:

[0140] Using at least one processor, obtaining sensor data indicative of an agent in an environment in which the autonomous vehicle is configured to operate along a first trajectory;

[0141] determining, using the at least one processor, a dynamic trace associated with the agent;

[0142] generating, using the at least one processor, obstacle data associated with the agent based on the dynamic trace, wherein the obstacle data indicates that the agent is an obstacle along the first trajectory of the autonomous vehicle;

[0143] determining, using the at least one processor, a station constraint and a lateral constraint to apply to a trajectory based on the obstacle data;

[0144] generating, using the at least one processor, a second trajectory for the autonomous vehicle based on the station constraint and the lateral constraint; and

[0145] Using the at least one processor, data associated with the second trajectory is provided, the data associated with the second trajectory being configured to cause the autonomous vehicle to operate along the second trajectory.

[0146] Example 2. The method according to Example 1, wherein generating the obstacle data comprises: determining a projection distance of the intelligent agent onto the first trajectory.

[0147] Example 3. A method according to Example 2, wherein generating the obstacle data includes: generating obstacle data including one or more of the following items: the projection distance, the agent type associated with the agent, and the environment type associated with the environment.

[0148] Example 4. The method according to Example 2 or 3, wherein determining the station constraint and the lateral constraint comprises: determining the station constraint based on the projection distance.

[0149] Example 5. A method according to any one of Examples 2 to 4, wherein determining the station constraint and the lateral constraint includes: determining the lateral constraint based on the projection distance.

[0150] Example 6. The method of any of the preceding examples, wherein generating the second trajectory based on the station constraint and the lateral constraint comprises:

[0151] determining homotopy based on the station constraint and the lateral constraint; and

[0152] The second trajectory of the autonomous vehicle is generated based on the homotopy.

[0153] Example 7. A method according to any one of Examples 3 to 6, wherein determining the station constraint and the lateral constraint is further based on the agent type.

[0154] Example 8. A method according to any one of Examples 3 to 7, wherein determining the station constraint and the lateral constraint is further based on the environment type.

[0155] Example 9. A system comprising:

[0156] at least one processor; and

[0157] at least one memory having instructions stored thereon, wherein when the instructions are executed by the at least one processor, the at least one processor is caused to perform operations, the operations comprising:

[0158] obtaining sensor data indicative of an agent in an environment in which the autonomous vehicle is configured to operate along a first trajectory;

[0159] determining a dynamic trace associated with the agent;

[0160] generating obstacle data associated with the agent based on the dynamic trace, wherein the obstacle data indicates that the agent is an obstacle along the first trajectory of the autonomous vehicle;

[0161] determining a station constraint and a lateral constraint to apply to the trajectory based on the obstacle data;

[0162] generating a second trajectory for the autonomous vehicle based on the station constraint and the lateral constraint; and

[0163] Data associated with the second trajectory is provided, the data associated with the second trajectory being configured to cause the autonomous vehicle to operate along the second trajectory.

[0164] Example 10. The system of Example 9, wherein generating the obstacle data comprises determining a projection distance of the agent onto the first trajectory.

[0165] Example 11. A system according to Example 10, wherein generating the obstacle data includes: generating obstacle data including one or more of the following items: the projection distance, the agent type associated with the agent, and the environment type associated with the environment.

[0166] Example 12. The system of Example 10 or 11, wherein determining the station constraint and the lateral constraint comprises determining the station constraint based on the projection distance.

[0167] Example 13. A system according to any one of Examples 10 to 12, wherein determining the station constraint and the lateral constraint includes: determining the lateral constraint based on the projection distance.

[0168] Example 14. The system of any one of Examples 9 to 13, wherein generating the second trajectory based on the station constraint and the lateral constraint comprises:

[0169] determining homotopy based on the station constraint and the lateral constraint; and

[0170] The second trajectory of the autonomous vehicle is generated based on the homotopy.

[0171] Example 15. A system according to any one of Examples 11 to 14, wherein determining the station constraint and the lateral constraint is further based on the agent type.

[0172] Example 16. A system according to any one of Examples 11 to 15, wherein determining the station constraint and the lateral constraint is further based on the environment type.

[0173] Example 17. A non-transitory computer-readable medium comprising instructions stored thereon, which, when executed by at least one processor, cause the at least one processor to perform operations comprising:

[0174] obtaining sensor data indicative of an agent in an environment in which the autonomous vehicle is configured to operate along a first trajectory;

[0175] determining a dynamic trace associated with the agent;

[0176] generating obstacle data associated with the agent based on the dynamic trace, wherein the obstacle data indicates that the agent is an obstacle along the first trajectory of the autonomous vehicle;

[0177] determining a station constraint and a lateral constraint to apply to the trajectory based on the obstacle data;

[0178] generating a second trajectory for the autonomous vehicle based on the station constraint and the lateral constraint; and

[0179] Data associated with the second trajectory is provided, the data associated with the second trajectory being configured to cause the autonomous vehicle to operate along the second trajectory.

[0180] Example 18. The non-transitory computer-readable medium of Example 17, wherein generating the obstacle data comprises determining a projection distance of the agent onto the first trajectory.

[0181] Example 19. A non-transitory computer-readable medium according to Example 18, wherein generating the obstacle data includes: generating obstacle data including one or more of the following items: the projection distance, the agent type associated with the agent, and the environment type associated with the environment.

[0182] Example 20. The non-transitory computer-readable medium of Example 18 or 19, wherein determining the station constraint and the lateral constraint comprises determining the station constraint based on the projection distance.

[0183] Example 21. A non-transitory computer-readable medium according to any one of Examples 18 to 20, wherein determining the station constraint and the lateral constraint includes: determining the lateral constraint based on the projection distance.

[0184] Example 22. The non-transitory computer-readable medium of any one of Examples 17 to 21, wherein generating the second trajectory based on the station constraint and the lateral constraint comprises:

[0185] determining homotopy based on the station constraint and the lateral constraint; and

[0186] The second trajectory of the autonomous vehicle is generated based on the homotopy.

[0187] Example 23. A non-transitory computer-readable medium according to any one of Examples 19 to 22, wherein determining the station constraint and the lateral constraint is further based on the agent type.

[0188] Example 24. A non-transitory computer-readable medium according to any one of Examples 19 to 23, wherein determining the station constraint and the lateral constraint is further based on the environment type.

[0189] Example 25. A method according to Example 2, wherein determining the projection distance includes: determining an agent polygon associated with the agent, wherein the agent polygon represents a spatial boundary of the agent.

[0190] Example 26. The method of Example 25, wherein determining the projection distance comprises determining lateral distances of vertices of the agent polygon relative to a baseline path associated with the first trajectory.

[0191] Example 27. According to the method of Example 25, determining the projection distance includes: determining the normal projection of the vertices of the agent polygon on the baseline path associated with the first trajectory.

[0192] Example 28. A system according to Example 9, wherein determining the projection distance includes: determining a lateral distance of the intelligent body relative to a baseline path or a projection length of the intelligent body on the baseline path, wherein the baseline path is associated with the first trajectory.

Claims

1. A method, include: Using at least one processor, obtaining sensor data indicative of an agent in an environment in which the autonomous vehicle is configured to operate along a first trajectory; determining, using the at least one processor, a dynamic trace associated with the agent; generating, using the at least one processor, obstacle data associated with the agent based on the dynamic trace, wherein the obstacle data indicates that the agent is an obstacle along the first trajectory of the autonomous vehicle; determining, using the at least one processor, a station constraint and a lateral constraint based on the obstacle data usable to generate a trajectory for use by the autonomous vehicle; generating, using the at least one processor, a second trajectory for the autonomous vehicle based on the station constraint and the lateral constraint; as well as Using the at least one processor, data associated with the second trajectory is provided, the data associated with the second trajectory being configured to cause the autonomous vehicle to operate along the second trajectory.

2. The method according to claim 1, in, Generating the obstacle data includes: determining a projection distance of the agent onto the first trajectory.

3. The method according to claim 2, in, Determining the projection distance includes determining an agent polygon associated with the agent, wherein the agent polygon represents a spatial boundary of the agent.

4. The method according to claim 3, in, Determining the projected distance includes determining lateral distances of vertices of the agent polygon relative to a baseline path associated with the first trajectory.

5. The method according to claim 3, determining the projection distance include: Normal projections of vertices of the agent polygon onto a baseline path associated with the first trajectory are determined.

6. The method according to claim 2, in, Generating the obstacle data includes generating obstacle data including one or more of: the projection distance, an agent type associated with the agent, and an environment type associated with the environment.

7. The method according to any one of claims 2 to 6, in, Determining the position constraint and the lateral constraint includes determining the position constraint based on the projection distance.

8. The method according to any one of claims 2 to 7, in, Determining the station constraint and the lateral constraint includes: determining the lateral constraint based on the projection distance.

9. The method according to any one of the preceding claims, in, Generating the second trajectory based on the station constraint and the lateral constraint includes: determining homotopy based on the station constraint and the lateral constraint; and The second trajectory of the autonomous vehicle is generated based on the homotopy.

10. The method according to any one of claims 6 to 9, in, Determining the station constraint and the lateral constraint is further based on the agent type.

11. The method according to any one of claims 6 to 10, in, Determining the station constraint and the lateral constraint is further based on the environment type.

12. A system, include: at least one processor; as well as at least one memory having instructions stored thereon, wherein when the instructions are executed by the at least one processor, the at least one processor is caused to perform operations, the operations comprising: obtaining sensor data indicative of an agent in an environment in which the autonomous vehicle is configured to operate along a first trajectory; determining a dynamic trace associated with the agent; generating obstacle data associated with the agent based on the dynamic trace, wherein the obstacle data indicates that the agent is an obstacle along the first trajectory of the autonomous vehicle; determining a station constraint and a lateral constraint to apply to the trajectory based on the obstacle data; generating a second trajectory for the autonomous vehicle based on the station constraint and the lateral constraint; and Data associated with the second trajectory is provided, the data associated with the second trajectory being configured to cause the autonomous vehicle to operate along the second trajectory.

13. The system according to claim 12, in, Generating the obstacle data includes: determining a projection distance of the agent onto the first trajectory.

14. The system according to claim 13, in, Determining the projection distance includes determining a lateral distance of the agent relative to a baseline path or a projection length of the agent on the baseline path, wherein the baseline path is associated with the first trajectory.

15. A system according to claim 13 or 14, in, Generating the obstacle data includes generating obstacle data including one or more of: the projection distance, an agent type associated with the agent, and an environment type associated with the environment.

16. A system according to any one of claims 13 to 15, in, Determining the position constraint and the lateral constraint includes determining the position constraint based on the projection distance.

17. A system according to any one of claims 13 to 16, in, Determining the station constraint and the lateral constraint includes: determining the lateral constraint based on the projection distance.

18. A system according to any one of claims 12 to 17, in, Generating the second trajectory based on the station constraint and the lateral constraint includes: determining homotopy based on the station constraint and the lateral constraint; and The second trajectory of the autonomous vehicle is generated based on the homotopy.

19. A system according to any one of claims 15 to 18, in, Determining the station constraint and the lateral constraint is further based on the agent type.

20. The system according to any one of claims 15 to 18, in, Determining the station constraint and the lateral constraint is further based on the environment type.

21. A non-transitory computer readable medium comprising instructions stored thereon, which instructions, when executed by at least one processor, cause the at least one processor to perform operations, the operations include: obtaining sensor data indicative of an agent in an environment in which the autonomous vehicle is configured to operate along a first trajectory; determining a dynamic trace associated with the agent; generating obstacle data associated with the agent based on the dynamic trace, wherein the obstacle data indicates that the agent is an obstacle along the first trajectory of the autonomous vehicle; determining a station constraint and a lateral constraint to apply to the trajectory based on the obstacle data; generating a second trajectory for the autonomous vehicle based on the station constraint and the lateral constraint; as well as Data associated with the second trajectory is provided, the data associated with the second trajectory being configured to cause the autonomous vehicle to operate along the second trajectory.

22. The non-transitory computer readable medium of claim 21, in, Generating the obstacle data includes: determining a projection distance of the agent onto the first trajectory.

23. The non-transitory computer readable medium of claim 22, in, Generating the obstacle data includes generating obstacle data including one or more of: the projection distance, an agent type associated with the agent, and an environment type associated with the environment.

24. The non-transitory computer readable medium according to claim 22 or 23, in, Determining the position constraint and the lateral constraint includes determining the position constraint based on the projection distance.

Images