Guided generation of trajectory for remote vehicle assistance

Abstract

A method for generating a suggested trajectory for a vehicle at a remote vehicle assistance system is provided. One or more user inputs defining the suggested trajectory may be received at the remote vehicle assistance system. The suggested trajectory may be validated to comply with one or more constraints imposed by a motion planner of the vehicle. Examples of constraints include that the suggested trajectory is within a drivable surface and does not collide with one or more objects present in an environment of the vehicle. If a remote vehicle assistance system fails to validate the suggested trajectory, the suggested trajectory may be redrawn instead of sending it to the vehicle. A system and a computer program product are also provided.

Description

[0001] Cross - Reference to Related Applications

[0002] This application claims the benefit of priority to U.S. Provisional Application No. 63 / 416,868, filed on October 17, 2022, entitled "GUIDED GENERATION OF TRAJECTORIES FOR REMOTE VEHICLE ASSISTANCE" and U.S. Application No. 18 / 071,360, filed on November 29, 2022, entitled "GUIDED GENERATION OF TRAJECTORIES FOR REMOTE VEHICLE ASSISTANCE", the disclosures of which are hereby incorporated by reference in their entireties. Background Art

[0003] Autonomous vehicles are capable of sensing and navigating through their surrounding environment with little to no human input. To safely navigate a vehicle along a selected path, the vehicle can rely on motion planning processes to generate, update, and execute one or more trajectories through its immediate surrounding environment. A vehicle's trajectory can be generated based on the vehicle's own current state and the conditions present in the vehicle's surrounding environment, which can include moving objects such as other vehicles and pedestrians, as well as stationary objects such as buildings and street poles. For example, a trajectory can be generated to avoid collisions between the vehicle and the objects present in its surrounding environment. Additionally, a trajectory can be generated such that the vehicle operates according to other desired characteristics such as path length, ride quality or comfort, required travel time, compliance with traffic rules, and / or adherence to driving practices. Brief Description of the Drawings

[0004] Figure 1 is an example environment of a vehicle that can implement one or more components of an autonomous system;

[0005] Figure 2 is a diagram of one or more systems of a vehicle including an autonomous system;

[0006] Figure 3 is Figure 1 and Figure 2 a diagram of one or more devices and / or components of one or more systems;

[0007] Figure 4A is a diagram of certain components of an autonomous system;

[0008] Figure 4B is a diagram of an implementation of a Remote Vehicle Assistance (RVA) system;

[0009] Figure 5 Is a sequence diagram that is an example of a process for generating a trajectory at a remote vehicle assistance system for ingestion by a planning system of an autonomous vehicle;

[0010] Figure 6A A screenshot depicting an example of a user interface for generating a trajectory;

[0011] Figure 6B A screenshot depicting another example of a user interface for generating a trajectory;

[0012] Figure 6C A screenshot depicting another example of a user interface for generating a trajectory;

[0013] Figure 6D A screenshot depicting another example of a user interface for generating a trajectory; and

[0014] Figure 7 Is a flowchart of a process for generating a trajectory at a remote vehicle assistance system for ingestion by a planning system of an autonomous vehicle. Detailed Description

[0015] In the following description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the present disclosure. It will be evident, however, that the embodiments described herein may be practiced without these specific details. In some instances, well-known structures and devices are illustrated in block diagram form in order to avoid unnecessarily obscuring aspects of the present disclosure.

[0016] In the drawings, for ease of description, a particular arrangement or order of schematic elements (such as those representing systems, devices, modules, instruction blocks, and / or data elements, etc.) is illustrated. However, those skilled in the art will understand that unless explicitly described, the particular order or arrangement of schematic elements in the drawings is not intended to imply a requirement for a particular processing order or sequence, or a separation of processes. Additionally, unless explicitly described, the inclusion of schematic elements in the drawings is not intended to imply that such elements are required in all embodiments, nor that the features represented by such elements cannot be included in some embodiments or combined with other elements in some embodiments.

[0017] In addition, in the drawings, connecting elements (such as solid lines, dashed lines, or arrows) are used to illustrate a connection, relationship, or association between or among two or more other schematic elements. The absence of any such connecting element is not intended to mean that a connection, relationship, or association cannot exist. In other words, some connections, relationships, or associations between elements are not illustrated in the drawings so as not to obscure the present disclosure. In addition, for ease of illustration, a single connecting element may be used to represent multiple connections, relationships, or associations between elements. For example, if a connecting element represents the communication of a signal, data, or instruction (e.g., “software instruction”), those skilled in the art will understand that such an element may represent one or more signal paths (e.g., a bus) that may be required to affect the communication.

[0018] Although terms such as “first,” “second,” and / or “third” etc. are used to describe various elements, these elements should not be limited by these terms. The terms “first,” “second,” and / or “third” are only used to distinguish one element from another. For example, without departing from the scope of the described embodiments, a first contact may be referred to as a second contact, and similarly, a second contact may be referred to as a first contact. Both the first contact and the second contact are contacts, but they are not the same contact.

[0019] The terms used in the description of the various embodiments herein are included only for the purpose of describing particular embodiments and are not intended to be limiting. As used in the description of the various embodiments and the appended claims, the singular forms “a,” “an,” and “the” are also intended to include the plural forms and may be used interchangeably with “one or more” or “at least one” unless the context clearly dictates otherwise. It will also be understood that the term “and / or” as used herein refers to and includes any and all possible combinations of one or more of the associated listed items. It will also be understood that when the terms “comprises,” “comprising,” “includes,” and / or “including” are used in this specification, they specify the presence of the stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0020] As used herein, the terms "communicate" and "communicating" refer to at least one of receiving, receipt, transmission, conveyance, and / or provision of information (or information represented by, for example, data, signals, messages, instructions, and / or commands, etc.). For a unit (e.g., a device, a system, a component of a device or system, and / or a combination thereof, etc.) that is to communicate with another unit, this means that the unit can receive information from the other unit directly or indirectly and / or send (e.g., transmit) information to the other unit. This can refer to a direct or indirect connection that is inherently wired and / or wireless. Additionally, two units can communicate with each other even if the information transmitted therebetween is modified, processed, relayed, and / or routed. For example, even if a first unit receives information passively and does not actively transmit information to a second unit, the first unit can communicate with the second unit. As another example, if at least one intermediate unit (e.g., a third unit located between the first unit and the second unit) processes the information received from the first unit and transmits the processed information to the second unit, the first unit can communicate with the second unit. In some embodiments, a message can refer to a network packet (e.g., a data packet, etc.) that includes data.

[0021] As used herein, depending on the context, the term "if" is optionally interpreted to mean "when", "at the time of", "in response to determining that", and / or "in response to detecting", etc. Similarly, depending on the context, the phrase "if it has been determined" or "if [the stated condition or event] has been detected" is optionally interpreted to mean "at the time of determining...", "in response to determining that", or "at the time of detecting [the stated condition or event]" and / or "in response to detecting [the stated condition or event]", etc. Further, as used herein, the terms "have", "having", or "possess", etc. are intended to be open-ended terms. Additionally, unless otherwise explicitly stated, the phrase "based on" is intended to mean "at least partially based on".

[0022] Reference will now be made in detail to embodiments, examples of which are illustrated in the accompanying drawings. In the following detailed description, numerous specific details are set forth in order to provide a thorough understanding of the various described embodiments. However, it will be apparent to those of ordinary skill in the art that the various described embodiments may be practiced without these specific details. In other instances, well-known methods, procedures, components, circuits, and networks have not been described in detail so as not to unnecessarily obscure aspects of the embodiments.

[0023] General Overview

[0024] In some aspects and / or embodiments, the systems, methods, and computer program products described herein include and / or implement generating a trajectory at a Remote Vehicle Assist (RVA) system for ingestion by a planning system of a vehicle (e.g., an autonomous vehicle). For example, the Remote Vehicle Assist system may receive from a remote assist operator one or more user inputs specifying a proposed trajectory for the vehicle. Instead of sending the proposed trajectory directly to the vehicle's planning system, the Remote Vehicle Assist system may first verify the proposed trajectory to ensure that the proposed trajectory satisfies one or more constraints imposed by the vehicle's planning system. The Remote Vehicle Assist system may determine one or more constraints imposed by the vehicle's planning system based at least on the current condition of the vehicle and / or conditions present in the vehicle's surrounding environment. For example, in some cases, one or more constraints may be determined based on a map and / or one or more objects tracked by the vehicle. Additionally, in some cases, one or more constraints may include that the proposed trajectory is within a drivable surface, does not collide with one or more objects present in the vehicle's surrounding environment, and / or is within lane boundaries, etc. In the case where it is determined that the proposed trajectory satisfies one or more constraints imposed by the vehicle's planning system, the Remote Vehicle Assist system may send the proposed trajectory to the vehicle's planning system. Conversely, if the proposed trajectory fails to satisfy one or more constraints imposed by the vehicle's planning system, the Remote Vehicle Assist system may prevent the proposed trajectory from being sent to the vehicle's planning system.

[0025] Techniques for generating, at a Remote Vehicle Assist (RVA) system, a trajectory for a vehicle (e.g., an autonomous vehicle) that satisfies one or more constraints imposed by a planning system of the vehicle, by implementation of the systems, methods, and computer program products described herein. Traditional solutions directly send a proposed trajectory generated at a remote vehicle assist system to a planning system of the vehicle without any verification that the proposed trajectory satisfies one or more constraints imposed by the planning system of the vehicle. Thus, in traditional solutions, the proposed trajectory may be sent to the planning system of the vehicle even when the proposed trajectory includes errors such as overlapping a curb, leaving a drivable surface, and / or colliding with an object. In the case where the proposed trajectory violates a constraint imposed by the planning system of the vehicle, the proposed trajectory may be rejected by the planning system of the vehicle. The total time cost associated with a proposed path that is rejected by the planning system of the vehicle and subsequently redrawn at the remote assist system is high (e.g., typically between one and three minutes). Verifying the proposed trajectory before sending it to the planning system to ensure that the proposed trajectory satisfies the constraints imposed by the planning system of the vehicle can minimize the likelihood of the proposed trajectory being rejected by the planning system, thus avoiding unnecessary delays and improving the experience of the vehicle's drivers and passengers as well as the experience of remote assist operators.

[0026] Now refer to Figure 1 , to illustrate example environment 100, in which vehicles including autonomous systems and vehicles not including autonomous systems operate. As illustrated, environment 100 includes vehicles 102a - 102n, objects 104a - 104n, routes 106a - 106n, area 108, vehicle - to - infrastructure (V2I) devices 110, network 112, remote autonomous vehicle (AV) system 114, queue management system 116, and V2I system 118. Vehicles 102a - 102n, vehicle - to - infrastructure (V2I) devices 110, network 112, autonomous vehicle (AV) system 114, queue management system 116, and V2I system 118 are interconnected via a wired connection, a wireless connection, or a combination of wired or wireless connections (e.g., establishing a connection for communication, etc.). In some embodiments, objects 104a - 104n are interconnected with at least one of vehicles 102a - 102n, vehicle - to - infrastructure (V2I) devices 110, network 112, autonomous vehicle (AV) system 114, queue management system 116, and V2I system 118 via a wired connection, a wireless connection, or a combination of wired or wireless connections.

[0027] Vehicles 102a - 102n (individually referred to as vehicle 102 and collectively referred to as vehicles 102) include at least one device configured to transport goods and / or people. In some embodiments, vehicle 102 is configured to communicate with V2I device 110, remote AV system 114, queue management system 116, and / or V2I system 118 via network 112. In some embodiments, vehicles 102 include cars, buses, trucks, and / or trains, etc. In some embodiments, vehicle 102 is the same as or similar to vehicle 200 described herein (see Figure 2 ). In some embodiments, vehicles 200 in the set of vehicles 200 are associated with an autonomous queue manager. In some embodiments, as described herein, vehicle 102 travels along corresponding routes 106a - 106n (individually referred to as route 106 and collectively referred to as routes 106). In some embodiments, one or more vehicles 102 include an autonomous system (e.g., an autonomous system that is the same as or similar to autonomous system 202).

[0028] Objects 104a - 104n (individually referred to as object 104 and collectively referred to as objects 104) include, for example, at least one vehicle, at least one pedestrian, at least one cyclist, and / or at least one structure (e.g., a building, a sign, a fire hydrant, etc.), etc. Each object 104 (e.g., located at a fixed location and over a period of time) is stationary or (e.g., having a speed and associated with at least one trajectory) moving. In some embodiments, object 104 is associated with a corresponding location in area 108.

[0029] Routes 106a - 106n (each referred to individually as Route 106 and collectively as Routes 106) are each associated with (e.g., define) a series of actions (also referred to as a trajectory) along which an AV can be navigated. Each Route 106 begins at an initial state (e.g., a state corresponding to a first spatio - temporal location and / or speed, etc.) and ends at a final goal state (e.g., a state corresponding to a second spatio - temporal location different from the first) or a goal region (e.g., a subspace of acceptable states (e.g., a termination state)). In some embodiments, the first state includes a location where one or more individuals will board the AV, and the second state or region includes one or more locations where one or more individuals boarding the AV will disembark. In some embodiments, Route 106 includes multiple acceptable sequences of states (e.g., multiple sequences of spatio - temporal locations) that are associated with (e.g., define) multiple trajectories. In an example, Route 106 includes only high - level actions or imprecise state locations, such as a series of connecting roads indicating a direction change at a roadway intersection, etc. Additionally or alternatively, Route 106 can include more precise actions or states, such as, for example, a specific target lane or precise location within a lane region and a target rate at those locations. In an example, Route 106 includes multiple precise state sequences along at least one high - level action with a limited look - ahead horizon for reaching an intermediate goal, where the combination of successive iterations of the limited - horizon state sequences cumulatively corresponds to multiple trajectories that together form a high - level route terminating at the final goal state or region.

[0030] Region 108 includes a physical region (e.g., a geographic region) in which vehicle 102 can be navigated. In an example, Region 108 includes at least one state (e.g., a country, a province, an individual state among multiple states included in a country, etc.), at least a portion of a state, at least one city, at least a portion of a city, etc. In some embodiments, Region 108 includes at least one named arterial road (referred to herein as a “road”), such as a highway, an interstate highway, a parkway, an urban street, etc. Additionally or alternatively, in some examples, Region 108 includes at least one unnamed road, such as a lane, a section of a parking lot, a section of an open space and / or undeveloped area, a dirt road, etc. In some embodiments, a road includes at least one lane (e.g., a portion of the road through which vehicle 102 can pass). In an example, a road includes at least one lane associated with (e.g., identified based on) at least one lane marking line.

[0031] A vehicle-to-infrastructure (V2I) device 110 (sometimes referred to as a vehicle-to-infrastructure or vehicle-to-everything (V2X) device) includes at least one device configured to communicate with a vehicle 102 and / or a V2I system 118. In some embodiments, the V2I device 110 is configured to communicate with the vehicle 102, a remote AV system 114, a queue management system 116, and / or the V2I system 118 via a network 112. In some embodiments, the V2I device 110 includes a radio frequency identification (RFID) device, a sign, a camera (e.g., a two-dimensional (2D) and / or three-dimensional (3D) camera), lane markings, streetlights, a parking meter, etc. In some embodiments, the V2I device 110 is configured to communicate directly with the vehicle 102. Additionally or alternatively, in some embodiments, the V2I device 110 is configured to communicate with the vehicle 102, the remote AV system 114, and / or the queue management system 116 via the V2I system 118. In some embodiments, the V2I device 110 is configured to communicate with the V2I system 118 via the network 112.

[0032] The network 112 includes one or more wired and / or wireless networks. In an example, the network 112 includes a cellular network (e.g., a Long-Term Evolution (LTE) network, a third-generation (3G) network, a fourth-generation (4G) network, a fifth-generation (5G) network, a Code Division Multiple Access (CDMA) network, etc.), a Public Land Mobile Network (PLMN), a Local Area Network (LAN), a Wide Area Network (WAN), a Metropolitan Area Network (MAN), a telephone network (e.g., a Public Switched Telephone Network (PSTN)), a private network, an ad hoc network, an intranet, the Internet, a fiber-based network, a cloud computing network, etc., and / or a combination of some or all of these networks.

[0033] The remote AV system 114 includes at least one device configured to communicate with the vehicle 102, the V2I device 110, the network 112, the queue management system 116, and / or the V2I system 118 via the network 112. In an example, the remote AV system 114 includes a server, a server group, and / or other similar devices. In some embodiments, the remote AV system 114 is co-located with the queue management system 116. In some embodiments, the remote AV system 114 participates in the installation of some or all of the components of the vehicle (including autonomous systems, autonomous vehicle computing, and / or software implemented by autonomous vehicle computing). In some embodiments, the remote AV system 114 maintains (e.g., updates and / or replaces) these components and / or software during the life of the vehicle.

[0034] The queue management system 116 includes at least one device configured to communicate with the vehicle 102, the V2I device 110, the remote AV system 114, and / or the V2I system 118. In an example, the queue management system 116 includes a server, a server group, and / or other similar devices. In some embodiments, the queue management system 116 is associated with a ridesharing company (e.g., an organization for controlling the operation of multiple vehicles (e.g., vehicles including autonomous systems and / or vehicles not including autonomous systems), etc.).

[0035] In some embodiments, the V2I system 118 includes at least one device configured to communicate with the vehicle 102, the V2I device 110, the remote AV system 114, and / or the queue management system 116 via the network 112. In some examples, the V2I system 118 is configured to communicate with the V2I device 110 via a connection different from the network 112. In some embodiments, the V2I system 118 includes a server, a server group, and / or other similar devices. In some embodiments, the V2I system 118 is associated with a municipal authority or a private institution (e.g., a private institution for maintaining the V2I device 110, etc.).

[0036] Provide Figure 1 The number and arrangement of the illustrated elements are examples. Compared with Figure 1 the illustrated elements, there may be additional elements, fewer elements, different elements, and / or elements with different arrangements. Additionally or alternatively, at least one element of the environment 100 may perform one or more functions described as being performed by Figure 1 at least one different element. Additionally or alternatively, at least one set of elements of the environment 100 may perform one or more functions described as being performed by at least one different set of elements of the environment 100.

[0037] Now refer to Figure 2 , the vehicle 200 (which may be the same as or similar to Figure 1 the vehicle 102) includes an autonomous system 202, a powertrain control system 204, a steering control system 206, and a braking system 208, or is associated with the autonomous system 202, the powertrain control system 204, the steering control system 206, and the braking system 208. In some embodiments, the vehicle 200 is the same as the vehicle 102 (see Figure 1)Same or similar. In some embodiments, the autonomous system 202 is configured to endow the vehicle 200 with autonomous driving capabilities (e.g., implement at least one of the following driving automation or maneuver-based functions, features, and / or devices, etc., where the at least one driving automation or maneuver-based function, feature, and / or device enables the vehicle 200 to operate partially or fully without human intervention, including but not limited to fully autonomous vehicles (e.g., vehicles that abandon reliance on human intervention, such as level 5 ADS-operated vehicles, etc.), highly autonomous vehicles (e.g., vehicles that abandon reliance on human intervention in certain situations, such as level 4 ADS-operated vehicles, etc.), and / or conditionally autonomous vehicles (e.g., vehicles that abandon reliance on human intervention in limited situations, such as level 3 ADS-operated vehicles, etc.), etc.). In one embodiment, the autonomous system 202 includes the operational or tactical functionality required to operate the vehicle 200 in road traffic and continuously perform part or all of the dynamic driving task (DDT). In another embodiment, the autonomous system 202 includes an advanced driver assistance system (ADAS) that includes driver support features. The autonomous system 202 supports various levels of driving automation ranging from no driving automation (e.g., level 0) to full driving automation (e.g., level 5). For a detailed description of fully autonomous vehicles and highly autonomous vehicles, reference can be made to SAE International standard J3016: Taxonomy and Definitions for Terms Related to On-Road Motor Vehicle Automated Driving Systems, the entire content of which is incorporated by reference. In some embodiments, the vehicle 200 is associated with an autonomous queue manager and / or a ridesharing company.

[0038] The autonomous system 202 includes a sensor suite that includes one or more devices such as a camera 202a, a LiDAR sensor 202b, a Radar sensor 202c, and a microphone 202d. In some embodiments, the autonomous system 202 may include more or fewer devices and / or different devices (e.g., ultrasonic sensors, inertial sensors, GPS receivers (discussed below), and / or odometer sensors for generating data associated with an indication of the distance traveled by the vehicle 200, etc.). In some embodiments, the autonomous system 202 uses one or more devices included in the autonomous system 202 to generate data associated with the environment 100 described herein. The data generated by one or more devices of the autonomous system 202 can be used by one or more systems described herein to observe the environment (e.g., environment 100) in which the vehicle 200 is located. In some embodiments, the autonomous system 202 includes a communication device 202e, an autonomous vehicle computing 202f, a drive-by-wire (DBW) system 202h, and a safety controller 202g.

[0039] The camera 202a includes at least one device configured to communicate with the communication device 202e, the autonomous vehicle computing 202f, and / or the safety controller 202g via a bus (e.g., a bus 302 that is the same as or similar to Figure 3 the bus). The camera 202a includes at least one camera (e.g., a digital camera using an optical sensor such as a charge-coupled device (CCD), a thermal camera, an infrared (IR) camera, and / or an event camera, etc.) for capturing images including physical objects (e.g., cars, buses, curbs, and / or people, etc.). In some embodiments, the camera 202a generates camera data as an output. In some examples, the camera 202a generates camera data that includes image data associated with the image. In this example, the image data may specify at least one parameter corresponding to the image (e.g., image characteristics such as exposure, brightness, etc., and / or an image timestamp, etc.). In such an example, the image may be in a format (e.g., RAW, JPEG, and / or PNG, etc.). In some embodiments, the camera 202a includes a plurality of independent cameras configured on (e.g., located on) the vehicle for capturing images for the purpose of stereovision (stereo vision). In some examples, the camera 202a includes generating image data and transmitting the image data to the autonomous vehicle computing 202f and / or a queue management system (e.g., the same as Figure 1a plurality of cameras of the same or similar queuing management system 116 as the queuing management system). In such an example, the autonomous vehicle computing 202f determines the depth to one or more objects in the fields of view of at least two of the plurality of cameras based on image data from at least two cameras. In some embodiments, the camera 202a is configured to capture images of objects within a distance relative to the camera 202a (e.g., up to 100 meters and / or up to 1 kilometer, etc.). Thus, the camera 202a includes features such as sensors and lenses optimized for sensing objects at one or more distances relative to the camera 202a.

[0040] In an embodiment, the camera 202a includes at least one camera configured to capture one or more images associated with one or more traffic lights, street signs, and / or other physical objects that provide visual navigation information. In some embodiments, the camera 202a generates traffic light data associated with one or more images. In some examples, the camera 202a generates TLD (Traffic Light Detection) data associated with one or more images including a format (e.g., RAW, JPEG, and / or PNG, etc.). In some embodiments, the camera 202a that generates TLD data is different from other systems incorporating cameras described herein in that the camera 202a may include one or more cameras having a wide field of view (e.g., a wide-angle lens, a fish-eye lens, and / or a lens having a viewing angle of about 120 degrees or greater, etc.) to generate images related to as many physical objects as possible.

[0041] The Light Detection and Ranging (LiDAR) sensor 202b includes being configured to communicate with the communication device 202e, the autonomous vehicle computing 202f, and / or the safety controller 202g via a bus (e.g., with Figure 3at least one device configured to communicate via a bus (e.g., a bus identical or similar to bus 302). The LiDAR sensor 202b includes a system configured to emit light from a light emitter (e.g., a laser emitter). The light emitted by the LiDAR sensor 202b includes light outside the visible spectrum (e.g., infrared light, etc.). In some embodiments, during operation, the light emitted by the LiDAR sensor 202b encounters a physical object (e.g., a vehicle) and is reflected back to the LiDAR sensor 202b. In some embodiments, the light emitted by the LiDAR sensor 202b does not penetrate the physical object it encounters. The LiDAR sensor 202b also includes at least one light detector that detects the light after the light emitted from the light emitter encounters a physical object. In some embodiments, at least one data processing system associated with the LiDAR sensor 202b generates an image (e.g., a point cloud and / or a combined point cloud, etc.) representing the objects included in the field of view of the LiDAR sensor 202b. In some examples, at least one data processing system associated with the LiDAR sensor 202b generates an image representing the boundary of a physical object and / or the surface of a physical object (e.g., the topology of the surface, etc.). In such examples, the image is used to determine the boundary of the physical object in the field of view of the LiDAR sensor 202b.

[0042] A Radio Detection and Ranging (Radar) sensor 202c includes at least one device configured to communicate with a communication device 202e, an autonomous vehicle computer 202f, and / or a safety controller 202g via a bus (e.g., a bus identical or similar to Figure 3 bus 302). The Radar sensor 202c includes a system configured to emit (pulsed or continuous) radio waves. The radio waves emitted by the Radar sensor 202c include radio waves within a predetermined spectrum. In some embodiments, during operation, the radio waves emitted by the Radar sensor 202c encounter a physical object and are reflected back to the Radar sensor 202c. In some embodiments, the radio waves emitted by the Radar sensor 202c are not reflected by some objects. In some embodiments, at least one data processing system associated with the Radar sensor 202c generates a signal representing the objects included in the field of view of the Radar sensor 202c. For example, at least one data processing system associated with the Radar sensor 202c generates an image representing the boundary of a physical object and / or the surface of a physical object (e.g., the topology of the surface, etc.). In some examples, the image is used to determine the boundary of the physical object in the field of view of the Radar sensor 202c.

[0043] The microphone 202d includes at least one device configured to communicate with the communication device 202e, the autonomous vehicle computing 202f, and / or the safety controller 202g via a bus (e.g., a bus the same or similar to Figure 3 bus 302). The microphone 202d includes one or more microphones (e.g., an array microphone and / or an external microphone, etc.) that capture an audio signal and generate data associated with (e.g., representing) the audio signal. In some examples, the microphone 202d includes a transducer device and / or a similar device. In some embodiments, one or more of the systems described herein may receive the data generated by the microphone 202d and determine the position (e.g., distance, etc.) of an object relative to the vehicle 200 based on the audio signal associated with the data.

[0044] The communication device 202e includes at least one device configured to communicate with the camera 202a, the LiDAR sensor 202b, the Radar sensor 202c, the microphone 202d, the autonomous vehicle computing 202f, the safety controller 202g, and / or the DBW (drive-by-wire) system 202h. For example, the communication device 202e may include a device the same or similar to Figure 3 communication interface 314. In some embodiments, the communication device 202e includes a vehicle-to-vehicle (V2V) communication device (e.g., a device for enabling wireless communication of data between vehicles).

[0045] The autonomous vehicle computing 202f includes at least one device configured to communicate with the camera 202a, the LiDAR sensor 202b, the Radar sensor 202c, the microphone 202d, the communication device 202e, the safety controller 202g, and / or the DBW system 202h. In some examples, the autonomous vehicle computing 202f includes devices such as a client device, a mobile device (e.g., a cellular phone and / or a tablet, etc.), and / or a server (e.g., a computing device including one or more central processing units and / or graphics processing units, etc.). In some embodiments, the autonomous vehicle computing 202f is the same or similar to the autonomous vehicle computing 400 described herein. Additionally or alternatively, in some embodiments, the autonomous vehicle computing 202f is configured to communicate with an autonomous vehicle system (e.g., an autonomous vehicle system the same or similar to Figure 1 the remote AV system 114), a queue management system (e.g., a queue management system the same or similar to Figure 1 queue management system 116), a V2I device (e.g., a V2I device the same or similar to Figure 1 V2I device 110), and / or a V2I system (e.g., a V2I system the same or similar to Figure 1communicate with a V2I system 118 that is the same as or similar to the V2I system.

[0046] The safety controller 202g includes at least one device configured to communicate with the camera 202a, the LiDAR sensor 202b, the Radar sensor 202c, the microphone 202d, the communication device 202e, the autonomous vehicle computing 202f, and / or the DBW system 202h. In some examples, the safety controller 202g includes one or more controllers (such as an electrical controller and / or an electromechanical controller, etc.) configured to generate and / or transmit control signals to operate one or more devices of the vehicle 200 (such as the powertrain control system 204, the steering control system 206, and / or the braking system 208, etc.). In some embodiments, the safety controller 202g is configured to generate control signals that take precedence over (e.g., override) the control signals generated and / or transmitted by the autonomous vehicle computing 202f.

[0047] The DBW system 202h includes at least one device configured to communicate with the communication device 202e and / or the autonomous vehicle computing 202f. In some examples, the DBW system 202h includes one or more controllers (such as an electrical controller and / or an electromechanical controller, etc.) configured to generate and / or transmit control signals to operate one or more devices of the vehicle 200 (such as the powertrain control system 204, the steering control system 206, and / or the braking system 208, etc.). Additionally or alternatively, one or more controllers of the DBW system 202h are configured to generate and / or transmit control signals to operate at least one different device of the vehicle 200 (such as turn signals, headlights, door locks, and / or windshield wipers, etc.).

[0048] The powertrain control system 204 includes at least one device configured to communicate with the DBW system 202h. In some examples, the powertrain control system 204 includes at least one controller and / or actuator, etc. In some embodiments, the powertrain control system 204 receives control signals from the DBW system 202h, and the powertrain control system 204 causes the vehicle 200 to perform longitudinal vehicle movements (such as starting to move forward, stopping moving forward, starting to move backward, stopping moving backward, accelerating in a certain direction, decelerating in a certain direction, etc.), or perform lateral vehicle movements (such as making a left turn and / or making a right turn, etc.). In an example, the powertrain control system 204 increases, maintains the same, or decreases the energy (such as fuel and / or electricity, etc.) provided to the motor of the vehicle, thereby causing at least one wheel of the vehicle 200 to rotate or not rotate.

[0049] The steering control system 206 includes at least one device configured to rotate one or more wheels of the vehicle 200. In some examples, the steering control system 206 includes at least one controller and / or actuator, etc. In some embodiments, the steering control system 206 rotates two front wheels and / or two rear wheels of the vehicle 200 left or right to turn the vehicle 200 left or right. In other words, the steering control system 206 causes the activities required to regulate the y-axis component of the vehicle's movement.

[0050] The braking system 208 includes at least one device configured to actuate one or more brakes to decelerate the vehicle 200 and / or keep it stationary. In some examples, the braking system 208 includes at least one controller and / or actuator configured to close one or more calipers associated with one or more wheels of the vehicle 200 on the corresponding rotors of the vehicle 200. Additionally or alternatively, in some examples, the braking system 208 includes an automatic emergency braking (AEB) system and / or a regenerative braking system, etc.

[0051] In some embodiments, the vehicle 200 includes at least one platform sensor (not explicitly illustrated) for measuring or inferring the nature of the state or condition of the vehicle 200. In some examples, the vehicle 200 includes platform sensors such as a global positioning system (GPS) receiver, an inertial measurement unit (IMU), a wheel speed sensor, a wheel brake pressure sensor, a wheel torque sensor, an engine torque sensor, and / or a steering angle sensor. Although the braking system 208 is illustrated as being located Figure 2 proximal to the vehicle 200, the braking system 208 can be located anywhere in the vehicle 200.

[0052] Now refer to Figure 3, A schematic diagram of an exemplary device 300. As illustrated, device 300 includes a processor 304, a memory 306, a storage component 308, an input interface 310, an output interface 312, a communication interface 314, and a bus 302. In some embodiments, device 300 corresponds to: at least one device of vehicle 102 (e.g., at least one device of a system of vehicle 102); at least one device of vehicle 200 (e.g., at least one device of autonomous system 202, powertrain control system 204, steering control system 206, and / or braking system 208 of vehicle 200); and / or one or more devices of network 112 (e.g., one or more devices of a system of network 112). In some embodiments, one or more devices of vehicle 102 (e.g., one or more devices of a system of vehicle 102), vehicle 200, and / or one or more devices of network 112 (e.g., one or more devices of a system of network 112) include at least one device 300 and / or at least one component of device 300. As Figure 3 shown, device 300 includes a bus 302, a processor 304, a memory 306, a storage component 308, an input interface 310, an output interface 312, and a communication interface 314.

[0053] The bus 302 includes components that permit communication between the components of device 300. In some cases, the processor 304 includes a processor (e.g., a central processing unit (CPU), a graphics processing unit (GPU), and / or an accelerated processing unit (APU), etc.), a microphone, a digital signal processor (DSP), and / or any processing component that can be programmed to perform at least one function (e.g., a field programmable gate array (FPGA) and / or an application specific integrated circuit (ASIC), etc.). The memory 306 includes random access memory (RAM), read only memory (ROM), and / or another type of dynamic and / or static storage device that stores data and / or instructions for use by the processor 304 (e.g., flash memory, magnetic memory, and / or optical memory, etc.).

[0054] The storage component 308 stores data and / or software related to the operation and use of device 300. In some examples, the storage component 308 includes a hard disk (e.g., a magnetic disk, an optical disk, a magneto-optical disk, and / or a solid state disk, etc.), a compact disc (CD), a digital versatile disc (DVD), a floppy disk, a cassette tape, a magnetic tape, a CD-ROM, a RAM, a PROM, an EPROM, a FLASH-EPROM, an NV-RAM, and / or another type of computer-readable medium, and corresponding drives.

[0055] The input interface 310 includes components of the enabling device 300 that receive information via a user input (e.g., a touch screen display, a keyboard, a keypad, a mouse, buttons, switches, a microphone, and / or a camera, etc.). Additionally or alternatively, in some embodiments, the input interface 310 includes sensors for sensing information (e.g., a Global Positioning System (GPS) receiver, an accelerometer, a gyroscope, and / or an actuator, etc.). The output interface 312 includes components for providing output information from the device 300 (e.g., a display, a speaker, and / or one or more Light Emitting Diodes (LEDs), etc.).

[0056] In some embodiments, the communication interface 314 includes transceiver-like components (e.g., a transceiver and / or separate receivers and transmitters, etc.) that enable the device 300 to communicate with other devices via a wired connection, a wireless connection, or a combination of a wired connection and a wireless connection. In some examples, the communication interface 314 enables the device 300 to receive information from another device and / or provide information to another device. In some examples, the communication interface 314 includes an Ethernet interface, an optical interface, a coaxial interface, an infrared interface, a Radio Frequency (RF) interface, a Universal Serial Bus (USB) interface, an interface, and / or a cellular network interface, etc.

[0057] In some embodiments, the device 300 performs one or more of the processes described herein. The device 300 performs these processes based on software instructions stored by a computer-readable medium such as the memory 306 and / or the storage component 308 and executed by the processor 304. A computer-readable medium (e.g., a non-transitory computer-readable medium) is defined herein as a non-transitory memory device. A non-transitory memory device includes a storage space located within a single physical storage device or a storage space distributed across multiple physical storage devices.

[0058] In some embodiments, software instructions are read into the memory 306 and / or the storage component 308 from another computer-readable medium or from another device via the communication interface 314. When executed, the software instructions stored in the memory 306 and / or the storage component 308 cause the processor 304 to perform one or more of the processes described herein. Additionally or alternatively, hardwired circuitry is used instead of or in combination with software instructions to perform one or more of the processes described herein. Thus, unless otherwise explicitly stated, the embodiments described herein are not limited to any particular combination of hardware circuitry and software.

[0059] The memory 306 and / or the storage component 308 includes a data store or at least one data structure (e.g., a database, etc.). The apparatus 300 is capable of receiving information from the data store or at least one data structure in the memory 306 or the storage component 308, storing the information in the data store or at least one data structure, communicating the information to the data store or at least one data structure, or searching for the information stored in the data store or at least one data structure. In some examples, the information includes network data, input data, output data, or any combination thereof.

[0060] In some embodiments, the apparatus 300 is configured to execute software instructions stored in the memory 306 and / or the memory of another apparatus (e.g., another apparatus that is the same as or similar to the apparatus 300). As used herein, the term "module" refers to at least one instruction stored in the memory 306 and / or the memory of another apparatus, which when executed by the processor 304 and / or the processor of another apparatus (e.g., another apparatus that is the same as or similar to the apparatus 300), causes the apparatus 300 (e.g., at least one component of the apparatus 300) to perform one or more than one process described herein. In some embodiments, the module is implemented in software, firmware, and / or hardware, etc.

[0061] Provided Figure 3 The number and arrangement of the illustrated components are provided as examples. In some embodiments, compared with Figure 3 the illustrated components, the apparatus 300 may include additional components, fewer components, different components, or components arranged differently. Additionally or alternatively, a set of components of the apparatus 300 (e.g., one or more than one component) may perform one or more than one function described as being performed by another component or another set of components of the apparatus 300.

[0062] Now refer Figure 4A, an example block diagram of an autonomous vehicle computing 400 (sometimes referred to as an "AV stack") is illustrated. As illustrated, the autonomous vehicle computing 400 includes a perception system 402 (sometimes referred to as a perception module), a planning system 404 (sometimes referred to as a planning module), a positioning system 406 (sometimes referred to as a positioning module), a control system 408 (sometimes referred to as a control module), and a database 410. In some embodiments, the perception system 402, the planning system 404, the positioning system 406, the control system 408, and the database 410 are included in and / or implemented in the vehicle's automatic navigation system (e.g., the autonomous vehicle computing 202f of the vehicle 200). Additionally or alternatively, in some embodiments, the perception system 402, the planning system 404, the positioning system 406, the control system 408, and the database 410 are included in one or more independent systems (e.g., one or more systems identical or similar to the autonomous vehicle computing 400, etc.). In some examples, the perception system 402, the planning system 404, the positioning system 406, the control system 408, and the database 410 are included in one or more independent systems located in the vehicle and / or at least one remote system as described herein. In some embodiments, any and / or all of the systems included in the autonomous vehicle computing 400 are implemented in software (e.g., software instructions stored in a memory), computer hardware (e.g., via a microprocessor, a microcontroller, an application-specific integrated circuit (ASIC), and / or a field-programmable gate array (FPGA), etc.), or a combination of computer software and computer hardware. It will also be understood that in some embodiments, the autonomous vehicle computing 400 is configured to communicate with remote systems (e.g., an autonomous vehicle system identical or similar to the remote AV system 114, a queue management system identical or similar to the queue management system 116, and / or a V2I system identical or similar to the V2I system 118, etc.).

[0063] In some embodiments, the perception system 402 receives data associated with at least one physical object in the environment (e.g., data used by the perception system 402 to detect at least one physical object), and classifies the at least one physical object. In some examples, the perception system 402 receives image data captured by at least one camera (e.g., camera 202a), the image being associated with one or more physical objects within the field of view of the at least one camera (e.g., representing the one or more physical objects). In such examples, the perception system 402 classifies the at least one physical object based on one or more groupings of physical objects (e.g., bicycles, vehicles, traffic signs, and / or pedestrians, etc.). In some embodiments, based on the classification of the physical objects by the perception system 402, the perception system 402 transmits data associated with the classification of the physical objects to the planning system 404.

[0064] In some embodiments, the planning system 404 receives data associated with a destination, and generates data associated with at least one route (e.g., route 106) along which a vehicle (e.g., vehicle 102) can travel towards the destination. In some embodiments, the planning system 404 periodically or continuously receives data from the perception system 402 (e.g., the data associated with the classification of the physical objects described above), and the planning system 404 updates at least one trajectory or generates at least one different trajectory based on the data generated by the perception system 402. In other words, the planning system 404 can perform tasks related to the tactical functions required to operate the vehicle 102 in road traffic. Tactical efforts involve maneuvering the vehicle in traffic during the journey, which includes but is not limited to deciding whether and when to overtake another vehicle, change lanes, or select an appropriate speed, acceleration, deceleration, etc. In some embodiments, the planning system 404 receives data associated with the updated position of the vehicle (e.g., vehicle 102) from the positioning system 406, and the planning system 404 updates at least one trajectory or generates at least one different trajectory based on the data generated by the positioning system 406.

[0065] In some embodiments, the positioning system 406 receives data associated with (e.g., representing) the location of a vehicle (e.g., vehicle 102) in an area. In some examples, the positioning system 406 receives LiDAR data associated with at least one point cloud generated by at least one LiDAR sensor (e.g., LiDAR sensor 202b). In certain examples, the positioning system 406 receives data associated with at least one point cloud from multiple LiDAR sensors, and the positioning system 406 generates a combined point cloud based on the respective point clouds. In these examples, the positioning system 406 compares the at least one point cloud or combined point cloud with two-dimensional (2D) and / or three-dimensional (3D) maps of the area stored in the database 410. Then, based on the positioning system 406 comparing the at least one point cloud or combined point cloud with the map, the positioning system 406 determines the position of the vehicle in the area. In some embodiments, the map includes a combined point cloud of the area generated prior to the navigation of the vehicle. In some embodiments, the map includes, but is not limited to, a high-precision map of the roadway geometry, a map describing the connection properties of the road network, a map describing the physical properties of the roadways (such as traffic speed, traffic flow, the number of vehicle and bicycle traffic lanes, lane width, lane traffic direction or the type and location of lane markings, or a combination thereof, etc.), and a map describing the spatial location of road features (such as crosswalks, traffic signs or various other types of driving signal lights, etc.). In some embodiments, the map is generated in real time based on the data received by the perception system.

[0066] In another example, the positioning system 406 receives global navigation satellite system (GNSS) data generated by a global positioning system (GPS) receiver. In some examples, the positioning system 406 receives GNSS data associated with the location of a vehicle in an area, and the positioning system 406 determines the latitude and longitude of the vehicle in the area. In such examples, the positioning system 406 determines the position of the vehicle in the area based on the latitude and longitude of the vehicle. In some embodiments, the positioning system 406 generates data associated with the position of the vehicle. In some examples, based on the positioning system 406 determining the position of the vehicle, the positioning system 406 generates data associated with the position of the vehicle. In such examples, the data associated with the position of the vehicle includes data associated with one or more semantic properties corresponding to the position of the vehicle.

[0067] In some embodiments, the control system 408 receives data associated with at least one trajectory from the planning system 404, and the control system 408 controls the operation of the vehicle. In some examples, the control system 408 receives data associated with at least one trajectory from the planning system 404, and the control system 408 controls the operation of the vehicle by generating and transmitting control signals to cause the powertrain control system (e.g., the DBW system 202h and / or the powertrain control system 204, etc.), the steering control system (e.g., the steering control system 206), and / or the braking system (e.g., the braking system 208) to operate. For example, the control system 408 is configured to perform operational functions such as lateral vehicle motion control or longitudinal vehicle motion control. Lateral vehicle motion control causes the activities required to regulate the y-axis component of the vehicle motion. Longitudinal vehicle motion control causes the activities required to regulate the x-axis component of the vehicle motion. In an example, in the case where the trajectory includes a left turn, the control system 408 transmits a control signal to cause the steering control system 206 to adjust the steering angle of the vehicle 200, thereby causing the vehicle 200 to turn left. Additionally or alternatively, the control system 408 generates and transmits control signals to cause other devices of the vehicle 200 (e.g., headlights, turn signals, door locks, and / or windshield wipers, etc.) to change states.

[0068] In some embodiments, the perception system 402, the planning system 404, the positioning system 406, and / or the control system 408 implement at least one machine learning model (e.g., at least one multi-layer perceptron (MLP), at least one convolutional neural network (CNN), at least one recurrent neural network (RNN), at least one autoencoder, and / or at least one transformer, etc.). In some examples, the perception system 402, the planning system 404, the positioning system 406, and / or the control system 408 implement at least one machine learning model individually or in combination with one or more of the above systems. In some examples, the perception system 402, the planning system 404, the positioning system 406, and / or the control system 408 implement at least one machine learning model as part of a pipeline (e.g., a pipeline for identifying one or more objects located in the environment, etc.).

[0069] The database 410 stores data transmitted to, received from, and / or updated by the perception system 402, the planning system 404, the positioning system 406, and / or the control system 408. In some examples, the database 410 includes a storage component (e.g., associated with Figure 3the same or similar storage components as the storage component 308). In some embodiments, the database 410 stores data associated with 2D and / or 3D maps of at least one area. In some examples, the database 410 stores data associated with 2D and / or 3D maps of a part of a city, multiple parts of multiple cities, multiple cities, counties, states, and / or countries (e.g., a country), etc. In such examples, a vehicle (e.g., a vehicle the same or similar to the vehicle 102 and / or the vehicle 200) can drive along one or more drivable areas (e.g., a single-lane road, a multi-lane road, a highway, a back road, and / or an off-road path, etc.), and cause at least one LiDAR sensor (e.g., a LiDAR sensor the same or similar to the LiDAR sensor 202b) to generate data associated with an image representing an object included in the field of view of the at least one LiDAR sensor.

[0070] In some embodiments, the database 410 can be implemented across multiple devices. In some examples, the database 410 is included in a vehicle (e.g., a vehicle the same or similar to the vehicle 102 and / or the vehicle 200), an autonomous vehicle system (e.g., an autonomous vehicle system the same or similar to the remote AV system 114), a queue management system (e.g., a queue management system Figure 1 the same or similar to the queue management system 116), and / or a V2I system (e.g., a V2I system Figure 1 the same or similar to the V2I system 118), etc.

[0071] Now refer to Figure 4B, an example block diagram illustrating an example of a Remote Vehicle Assist (RVA) system 450. In some embodiments, the Remote Vehicle Assist system 450 may be communicatively coupled to vehicles 102a - 102n and / or vehicle 200, objects 104a - 104n, routes 106a - 106n, regions 108, Vehicle-to-Infrastructure (V2I) devices 110, network 112, Remote Autonomous Vehicle (AV) system 114, queue management system 116, and / or V2I system 118. In some embodiments, the Remote Vehicle Assist system 450 may include vehicles 102a - 102n and / or vehicle 200, objects 104a - 104n, routes 106a - 106n, regions 108, Vehicle-to-Infrastructure (V2I) devices 110, network 112, Remote Autonomous Vehicle (AV) system 114, queue management system 116, and / or V2I system 118, form a part of vehicles 102a - 102n and / or vehicle 200, objects 104a - 104n, routes 106a - 106n, regions 108, Vehicle-to-Infrastructure (V2I) devices 110, network 112, Remote Autonomous Vehicle (AV) system 114, queue management system 116, and / or V2I system 118, be coupled to vehicles 102a - 102n and / or vehicle 200, objects 104a - 104n, routes 106a - 106n, regions 108, Vehicle-to-Infrastructure (V2I) devices 110, network 112, Remote Autonomous Vehicle (AV) system 114, queue management system 116, and / or V2I system 118, and / or use vehicles 102a - 102n and / or vehicle 200, objects 104a - 104n, routes 106a - 106n, regions 108, Vehicle-to-Infrastructure (V2I) devices 110, network 112, Remote Autonomous Vehicle (AV) system 114, queue management system 116, and / or V2I system 118.

[0072] Such as Figure 4BAs shown, the remote vehicle assistance system 450 may include a path planner 455 configured to obtain sensor data from one or more sensors 202 attached to the vehicle and process the sensor data. In some embodiments, the sensor data may be used to determine and provide a proposed trajectory 415 for the vehicle to navigate from a current location (such as a location where the vehicle is stranded and has requested remote vehicle assistance, etc.) to a target location that the vehicle is attempting to navigate to. In some cases, the proposed trajectory 415 may be generated based on one or more inputs received from a remote assistance operator via a user interface 467 of the path planner 455 (such as a graphical user interface (GUI), etc.). The operator may manipulate (e.g., change, redraw, add, or delete) one or more aspects of the proposed trajectory 415. The operator may adjust or change the proposed trajectory 415 to maximize the likelihood that the proposed trajectory 415 will be accepted by the vehicle's planning system 404, thereby providing continuous autonomous navigation from the stranded location. Then, the operator may select the proposed trajectory 415 to be provided to the vehicle to navigate through or away from the location where the vehicle is stranded (or has otherwise requested remote vehicle assistance). The operator may determine or otherwise select the proposed trajectory 415 based on the sensor data 461 and the costs and constraints that may be associated with one or more of the proposed trajectories 415 provided to the operator via the user interface 467.

[0073] In addition, as Figure 4B shown, the path planner 455 may include a Remote Vehicle Assistance (RVA) manager 465 that processes the sensor data 461 in response to a request for remote vehicle assistance received from the vehicle to generate the proposed trajectory 415. The Remote Vehicle Assistance manager 465 may be configured to generate the proposed trajectory 415 to minimize the likelihood that the proposed trajectory 415 will be rejected by the vehicle planning system 404. For example, in some cases, the Remote Vehicle Assistance manager 465 may ensure that the proposed trajectory 415 satisfies one or more constraints imposed by the planning system 404. Alternatively and / or additionally, the Remote Vehicle Assistance manager 465 may optimize the proposed trajectory 415 to minimize the cost associated with the proposed trajectory 415. The Remote Vehicle Assistance manager 465 may also provide the proposed trajectory 415 and the costs and constraints associated with the proposed trajectory 415 for visualization in the user interface 467. In this way, the remote assistance operator may adjust or modify the proposed trajectory 415 based on the costs and / or constraints associated with the proposed trajectory 415.

[0074] The path planner 455 may subscribe to or otherwise receive data (such as status updates or remote vehicle assistance (RVA) requests, etc.) from other systems included in the autonomous vehicle computing system 400. Additionally, the path planner 455 may read static configurations at startup and provide them as data feeds to the user interface 467. The path planner 455 may create visualizations of different possible trajectories including the proposed trajectory 415, which visualizations include costs and constraints associated with each trajectory. The visualizations may be provided via the user interface 467. The path planner 455 may also determine optimized alternative trajectories and may provide the alternative trajectories via the user interface 467.

[0075] In some embodiments, the proposed trajectory 415 may be modified by a remote assistance operator via the user interface 467. For example, when presenting a visualization including the proposed trajectory 415, the remote assistance operator may adjust the proposed trajectory 415 with respect to one or more constraints imposed by the planning system 404 (such as the location of curbs, lane boundaries, and / or objects being tracked, etc.) such that the adjustments made to the proposed trajectory 415 avoid violating one or more constraints. In some cases, the proposed trajectory 415 may be adjusted as a whole. Alternatively and / or additionally, the adjustments made to the proposed trajectory 415 may include adjustments to one or more individual segments of the proposed trajectory 415. Once the remote vehicle assistance manager 465 determines that the proposed trajectory 415 satisfies one or more constraints imposed by the vehicle's planning system 404, the remote assistance operator may then interact with the user interface 467 to select the proposed trajectory 415 for providing to the vehicle planning system 404. Although the proposed trajectory 415 may undergo further verification at the planning system 404, the verification performed by the remote vehicle assistance manager 465 maximizes the likelihood that the proposed trajectory 415 will be accepted by the planning system 404 and executed to navigate the vehicle along the selected path.

[0076] Now refer to Figure 5 , which depicts a sequence diagram illustrating an example of a process 500 for generating the proposed trajectory 415 at the remote vehicle assistance system 450. Figure 5 The example of the process 500 shown in depicts a scenario where the proposed trajectory 415 generated by the remote vehicle assistance (RVA) manager 465 fails to satisfy one or more constraints of the planning system 404 of the autonomous vehicle computing 400 at vehicle 102 or vehicle 200 and is thus rejected, for example, by a trajectory checker 504.

[0077] As Figure 5As shown, when receiving a remote vehicle assistance request from vehicle 102 or vehicle 200, the remote vehicle assistance manager 465 may generate the proposed trajectory 415 based on one or more user inputs received from the remote assistance operator 502 via the user interface 467. However, in some cases, the proposed trajectory 415 may include one or more errors (such as overlapping with a curb, leaving the drivable surface, and / or colliding with an object, etc.), which prevent the proposed trajectory 415 from being verified by the trajectory checker 504 at vehicle 102 or vehicle 200. That is, in the case of sending the proposed trajectory 415 to vehicle 102 or vehicle 200 without any pre-verification at the remote vehicle assistance system 450, when the trajectory checker 504 identifies one or more errors in the proposed trajectory 415, the proposed trajectory 415 may be rejected by the autonomous vehicle computing 400. Therefore, in response to rejecting the proposed trajectory 415, the remote vehicle assistance system 450 may redraw the proposed trajectory 415 based on one or more additional user inputs received from the remote assistance operator 502 via the user interface 467. The verification of the proposed trajectory 415 and the redrawing of the proposed trajectory 415 may continue at the trajectory checker 504 until the proposed trajectory 415 meets the constraints imposed by the trajectory checker 504, at which time the proposed trajectory 415 may be deployed to the control system 408 and used to navigate vehicle 102 or vehicle 200.

[0078] The rejection of the proposed trajectory 415 by the trajectory checker 504 and the redrawing thereof at the remote vehicle assistance system 450 may impose a significant delay (e.g., typically between one and three minutes). Therefore, in some example embodiments, the remote vehicle assistance manager 465 may verify the proposed trajectory 415 before the proposed trajectory 415 is sent to the autonomous vehicle computing 400 of vehicle 102 or vehicle 200. For example, when verifying the proposed trajectory 415, the remote vehicle assistance manager 465 may determine one or more constraints imposed by the trajectory checker 504 based on the data received from the autonomous vehicle computing 400. In addition, the remote vehicle assistance manager 465 may verify the proposed trajectory 415 to ensure that the proposed trajectory 415 meets one or more constraints imposed by the trajectory checker 504, such as, for example, the proposed trajectory 415 is within the drivable surface, does not collide with one or more objects existing in the surrounding environment, and / or remains within the lane boundaries, etc. Doing so can minimize the likelihood that the proposed trajectory 415 sent to the autonomous vehicle computing 400 of vehicle 102 or vehicle 200 is rejected by the trajectory checker 504.

[0079] As described above, the remote vehicle assistance manager 465 can determine one or more constraints imposed by the trajectory checker 504 based on data received from the autonomous vehicle computing 400. Examples of data received from the autonomous vehicle computing 400 can include one or more maps and objects tracked by the autonomous vehicle computing 400. In some cases, the remote vehicle assistance manager 465 can identify drivable surfaces based at least on lane and road segment annotations included in one or more maps received from the autonomous vehicle computing 400. Alternatively and / or additionally, the remote vehicle assistance manager 465 can determine location details of one or more tracked objects, which can include moving objects such as other vehicles, cyclists, and pedestrians, as well as stationary objects such as buildings, curbs, and street poles.

[0080] In some example embodiments, the remote vehicle assistance manager 465 can generate a user interface 467 to facilitate providing one or more user inputs that define the proposed trajectory 415. For example, in some cases, the user interface 467 can be generated to provide a visualization of one or more constraints imposed by the trajectory checker 504 to increase the likelihood that one or more user inputs received from the remote assistance operator 502 will define the proposed trajectory 415 as consistent with one or more constraints imposed by the trajectory checker 504.

[0081] Figures 6A to 6D A screenshot depicting an example of the user interface 467 is now referenced Figure 6A , an example of the user interface 467 is illustrated that provides one or more explicit visual indicators, the color and / or shape of the one or more explicit visual indicators indicating the cost associated with one or more corresponding segments of the proposed trajectory 415. In some cases, the one or more explicit visual indicators can identify segments of the proposed trajectory 415 that have a cost above a threshold due to violating one or more constraints imposed by the trajectory checker 504. In Figure 6A the example shown, the user interface 467 can be drawn to have a first visual indicator 602 and a second visual indicator 604, the first visual indicator 602 identifying a first segment of the proposed trajectory 415 that has a cost above the threshold due to proximity to a curb, and the second visual indicator 604 identifying a second segment of the proposed trajectory 415 that has a cost above the threshold due to proximity to another object (e.g., another vehicle in this example).

[0082] Now reference Figure 6B, Another example of a user interface 467 that provides one or more implicit visual indicators is illustrated, where the color and / or shape of the one or more implicit visual indicators indicate the cost associated with one or more corresponding segments of the proposed trajectory 415. In Figure 6B the example shown, the segments of the proposed trajectory 415 can be drawn in a color corresponding to the cost associated with that segment of the proposed trajectory 415. In some cases, if a segment of the proposed trajectory 415 does not violate any of the constraints imposed by the trajectory checker 504 and is associated with a cost below a threshold, that segment of the proposed trajectory 415 can be drawn in a first color. If the same segment of the proposed trajectory 415 violates one or more of the constraints imposed by the trajectory checker 504 and is associated with a cost above the threshold, that same segment of the proposed trajectory 415 can be drawn in a second color. Alternatively and / or additionally, the segments of the proposed trajectory 415 can be drawn in a color corresponding to the constraint(s) violated by the segment of the proposed trajectory 415. In Figure 6B the example shown, the first segment 606 of the proposed trajectory 415 can be drawn in a first color (blue) to indicate that the first segment 606 violates the constraint of having an overly tight turning radius, the second segment 608 and the third segment 610 of the proposed trajectory 415 can be drawn in a second color (amber) to indicate that the second segment 608 and the third segment 610 violate the constraint of crossing a lane boundary, the fourth segment 612 of the proposed trajectory 415 can be drawn in a third color (e.g., purple) to indicate that the fourth segment 612 violates the constraint of overlapping a curb, and the fifth segment 614 of the proposed trajectory 415 can be drawn in a fourth color (e.g., red) to indicate that the fifth segment 614 violates the constraint of being too close to another object (e.g., another vehicle in this example).

[0083] Now referring to Figure 6C , Another example of the user interface 467 is illustrated, where the user interface 467 provides a visual indicator 650 that demarcates an area in which a trajectory consistent with one or more of the constraints imposed by the trajectory checker 504 will be drawn. In some example embodiments, the remote vehicle assistance manager 465 can determine the area of the valid trajectory based on the extreme values of one or more of the constraints imposed by the trajectory checker 504. For example, in some cases, the remote vehicle assistance manager 465 can determine the area (e.g., the envelope of the valid trajectory) of the valid trajectory based on the minimum turning radius of the vehicle 102 or the vehicle 200 and the minimum clearance from one or more curbs, lane boundaries, and objects (e.g., other vehicles, cyclists, pedestrians, buildings, and / or street poles, etc.). As Figure 6CAs shown, one or more user inputs received from the remote assist operator 502 can define the proposed trajectory 415 by setting at least one or more nodes and segments of the proposed trajectory 415 within the area indicated by the visual indicator 650.

[0084] Figure 6D FIG. depicts another example of the user interface 467 that provides various visual indicators 662, 664, 666, 668, and 670 indicating where the trajectory 680 violates one or more constraints imposed by the trajectory checker 504. In Figure 6D the example shown, the visual indicators 662, 664, 666, and 668 indicate segments of the trajectory 680 that leave the drivable surface. At the same time, the visual indicator 670 indicates the segment of the trajectory 680 that collides with an object.

[0085] Figure 7 FIG. depicts a flowchart illustrating an example of a process 700 for generating a trajectory for a vehicle at a remote vehicle assist system. Referring to Figures 1 to 7 , the process 700 can be performed by the remote vehicle assist system 450 (e.g., by the remote vehicle assist manager 465 of the path planner 455).

[0086] At 702, the remote vehicle assist manager 465 can receive one or more user inputs that define a proposed trajectory for the vehicle. For example, the remote vehicle assist manager 465 can receive one or more user inputs that define the proposed trajectory 415 via the user interface 467. In some example embodiments, the remote vehicle assist manager 465 can generate the user interface 467 to facilitate providing one or more user inputs that define the proposed trajectory 415. For example, the user interface 467 can be generated to provide a visualization of one or more constraints imposed by the trajectory checker 504 of the vehicle 102 or the vehicle 200. The visualization can include, for example, one or more visual indicators indicating the cost associated with various segments of the proposed trajectory 415. In Figures 6A to 6BIn the example shown, these visual indicators can identify segments of the proposed trajectory 415 that fail to meet the constraints imposed by the trajectory checker 504 and are thus associated with a cost above a threshold. Additionally, in some cases, these visual indicators can identify specific constraints violated by segments of the proposed trajectory 415. Instead of and / or in addition to the visual indicators of cost, the remote vehicle assistance manager 465 can generate a user interface 467 to provide visual indicators of a delineated area in which the proposed trajectory 415 that is consistent with one or more constraints imposed by the trajectory checker 504 is to be drawn. Thus, one or more user inputs received via the user interface 467 can set one or more nodes and segments of the proposed trajectory 415 within the area indicated by the visual indicators.

[0087] At 704, the remote vehicle assistance manager 465 can determine one or more constraints imposed by the motion planner of the vehicle. In some example embodiments, the remote vehicle assistance manager 465 can determine one or more constraints imposed by the trajectory checker 504 at least based on data received from the autonomous vehicle computing 400 of vehicle 102 or vehicle 200. In some cases, the data received from the autonomous vehicle computing 400 can include one or more maps, and the remote vehicle assistance manager 465 can identify drivable surfaces at least based on lane and road segment annotations included in one or more maps received from the autonomous vehicle computing 400. Alternatively and / or additionally, the data received from the autonomous vehicle computing 400 can include one or more objects tracked by the autonomous vehicle computing 400, in which case the remote vehicle assistance manager 465 can determine location details of one or more of the tracked objects.

[0088] At 706, the remote vehicle assistance manager 465 can perform a verification of the proposed trajectory to determine whether the proposed trajectory meets one or more constraints. For example, in some example embodiments, the remote vehicle assistance manager 465 can verify the proposed trajectory 415 to ensure that the proposed trajectory 415 does not include one or more errors (such as overlapping with a curb, leaving a drivable surface, and / or colliding with an object, etc.) that prevent the proposed trajectory 415 from being verified by the trajectory checker 504 at vehicle 102 or vehicle 200.

[0089] At 708, the remote vehicle assistance manager 465 may prevent sending the proposed trajectory to the vehicle's motion planner in response to an unsuccessful verification of the proposed trajectory. In some example embodiments, if the proposed trajectory 415 is determined to include one or more errors (such as overlapping with a curb, leaving the drivable surface, and / or colliding with an object, etc.) that prevent the proposed trajectory 415 from being verified by the trajectory checker 504 at the vehicle 102 or the vehicle 200, the remote vehicle assistance manager 465 may avoid sending the proposed trajectory 415 to the autonomous vehicle computing 400 of the vehicle 102 or the vehicle 200. In an instance where the remote vehicle assistance manager 465 fails to verify the proposed trajectory 415, the remote vehicle assistance manager 465 may generate a notification for redrawing the proposed trajectory. Thus, in some instances, the remote vehicle assistance manager 465 may receive, via the user interface 467, one or more user inputs for redrawing the proposed trajectory 415 to better comply with one or more constraints imposed by the trajectory checker 504. Additionally, if the redrawn trajectory is determined to comply with one or more constraints imposed by the trajectory checker 504, the remote vehicle assistance manager 465 may verify the redrawn trajectory and send the redrawn trajectory to the autonomous vehicle computing 400 of the vehicle 102 or the vehicle 200.

[0090] At 710, the remote vehicle assistance manager 465 may send the proposed trajectory to the vehicle's motion planner in response to a successful verification of the proposed trajectory. In some example embodiments, if the proposed trajectory 415 is determined to comply with one or more constraints of the trajectory checker 504, the remote vehicle assistance manager 465 may send the proposed trajectory 415 to the autonomous vehicle computing 400 at the vehicle 102 or the vehicle 200.

[0091] According to some non - limiting embodiments or examples, a method is provided, including: receiving, by at least one data processor, one or more user inputs defining a proposed trajectory for a vehicle; determining, by the at least one data processor, one or more constraints imposed by the motion planner of the vehicle; verifying, by the at least one data processor, the proposed trajectory to determine whether the proposed trajectory satisfies the one or more constraints; and preventing, by the at least one data processor, sending the proposed trajectory to the motion planner of the vehicle in response to an unsuccessful verification of the proposed trajectory.

[0092] According to some non - limiting embodiments or examples, at least one non - transitory computer - readable medium is provided, which includes one or more instructions that, when executed by at least one processor, cause the at least one processor to: receive one or more user inputs defining a proposed trajectory for a vehicle; determine one or more constraints imposed by a motion planner of the vehicle; perform verification of the proposed trajectory to determine whether the proposed trajectory satisfies the one or more constraints; and in response to unsuccessful verification of the proposed trajectory, prevent the proposed trajectory from being sent to the motion planner of the vehicle.

[0093] According to some non - limiting embodiments or examples, a system is provided, including: at least one data processor; and at least one memory storing instructions, wherein the instructions, when executed by the at least one data processor, result in operations including: receiving one or more user inputs defining a proposed trajectory for a vehicle; determining one or more constraints imposed by a motion planner of the vehicle; performing verification of the proposed trajectory to determine whether the proposed trajectory satisfies the one or more constraints; and in response to unsuccessful verification of the proposed trajectory, preventing the proposed trajectory from being sent to the motion planner of the vehicle.

[0094] Further non - limiting aspects or embodiments are set forth in the numbered clauses below:

[0095] Clause 1: A method includes: receiving, by at least one data processor, one or more user inputs defining a proposed trajectory for a vehicle; determining, by the at least one data processor, one or more constraints imposed by a motion planner of the vehicle; performing, by the at least one data processor, verification of the proposed trajectory to determine whether the proposed trajectory satisfies the one or more constraints; and in response to unsuccessful verification of the proposed trajectory, preventing, by the at least one data processor, the proposed trajectory from being sent to the motion planner of the vehicle.

[0096] Clause 2: The method according to Clause 1 further includes: generating a user interface configured to receive the one or more user inputs defining the proposed trajectory for the vehicle.

[0097] Clause 3: The method according to Clause 2, wherein the user interface is generated to include one or more indicators corresponding to the one or more constraints imposed by the motion planner of the vehicle.

[0098] Clause 4: The method according to Clause 3, wherein the one or more indicators include a visual indication of one or more zones in which the valid path is to be drawn.

[0099] Clause 5: The method according to Clause 4, wherein the one or more zones in which the valid path is to be drawn are determined based on one or more maximum or minimum values associated with the one or more constraints imposed by the motion planner of the vehicle.

[0100] Clause 6: The method according to any one of Clauses 4 to 5, wherein the one or more zones in which the valid path is to be drawn are determined based on at least one of the following: the minimum turning radius of the vehicle, the minimum clearance from one or more objects in the surrounding environment of the vehicle, and / or the minimum clearance from lane boundaries.

[0101] Clause 7: The method according to any one of Clauses 3 to 6, wherein the one or more indicators include a visual indication of the cost associated with the proposed trajectory, and wherein the cost of the proposed trajectory corresponds to the magnitude of the deviation from the one or more constraints imposed by the motion planner of the vehicle.

[0102] Clause 8: The method according to Clause 7, wherein the visual indication identifies one or more portions of the proposed trajectory in which the proposed trajectory is at least one of the following: within a threshold distance of an object, deviating from the drivable surface, and crossing a lane boundary.

[0103] Clause 9: The method according to any one of Clauses 1 to 8, wherein the one or more user inputs define the proposed trajectory by at least specifying one or more nodes or segments that form the proposed trajectory.

[0104] Clause 10: The method according to any one of Clauses 1 to 9, wherein the one or more constraints include that the proposed trajectory is within the drivable surface.

[0105] Clause 11: The method according to any one of Clauses 1 to 10, wherein the one or more constraints include that the proposed trajectory does not collide with one or more objects present in the surrounding environment of the vehicle.

[0106] Clause 12: The method according to any one of Clauses 1 to 11, wherein the one or more constraints imposed by the motion planner of the vehicle are determined based on a map received from the vehicle.

[0107] Clause 13: The method according to any one of Clauses 1 to 12, wherein one or more constraints imposed by the motion planner of the vehicle are determined based on the location of one or more objects tracked by the vehicle.

[0108] Clause 14: The method according to any one of Clauses 1 to 13, further comprising: in response to successful verification of the proposed trajectory, sending, by the at least one data processor, the proposed trajectory to the motion planner of the vehicle.

[0109] Clause 15: The method according to any one of Clauses 1 to 14, further comprising: in response to unsuccessful verification of the proposed trajectory, generating, by the at least one data processor, a notification for redrawing the proposed trajectory.

[0110] Clause 16: The method according to Clause 15, further comprising: receiving, by at least one data processor, one or more user inputs for redrawing the proposed trajectory for the vehicle; verifying, by the at least one data processor, the redrawn trajectory to determine whether the redrawn trajectory satisfies the one or more constraints; and in response to successful verification of the redrawn trajectory, sending, by the at least one data processor, the redrawn trajectory to the motion planner of the vehicle.

[0111] Clause 17: A system, comprising: at least one data processor; and at least one memory storing instructions, wherein the instructions, when executed by the at least one data processor, perform operations including any one of Clauses 1 to 16.

[0112] Clause 18: A non-transitory computer-readable medium storing instructions, wherein the instructions, when executed by at least one data processor, perform operations including any one of Clauses 1 to 16.

[0113] In the foregoing description, aspects and embodiments of the present disclosure have been described with reference to numerous specific details, which may vary depending on the implementation. Therefore, the specification and drawings should be regarded as illustrative rather than restrictive in a limiting sense. The sole and exclusive indication of the scope of the invention, and what the applicant desires to be the scope of the invention, is the literal and equivalent scope of the claims as issued from this application in the specific form of the issued claims, including any subsequent amendments. Any definition of terms expressly set forth herein for inclusion in such claims shall be construed in the sense such terms are used in the claims. Additionally, when the term "further comprising" is used in the foregoing specification or the appended claims, the text following that phrase may be additional steps or entities, or sub-steps / sub-entities of the previously recited steps or entities.

Claims

1. A method, comprising: Receiving, by at least one data processor, one or more user inputs that define a proposed trajectory for a vehicle; Determining, by the at least one data processor, one or more constraints imposed by a motion planner of the vehicle; Validating, by the at least one data processor, the proposed trajectory to determine whether the proposed trajectory satisfies the one or more constraints; and In response to the validation of the proposed trajectory being unsuccessful, preventing, by the at least one data processor, the proposed trajectory from being sent to the motion planner of the vehicle.

2. The method according to claim 1, further comprising: Generating a user interface configured to receive the one or more user inputs that define the proposed trajectory for the vehicle.

3. The method according to claim 2, wherein, The user interface is generated to include one or more indicators corresponding to the one or more constraints imposed by the motion planner of the vehicle.

4. The method according to claim 3, wherein The one or more indicators include a visual indication of one or more areas in which a valid path is to be drawn.

5. The method according to claim 4, wherein, The one or more areas in which the valid path is to be drawn are determined based on one or more maximum or minimum values associated with the one or more constraints imposed by the motion planner of the vehicle.

6. The method according to claim 4, wherein The one or more areas in which the valid path is to be drawn are determined based on at least one of: a minimum turning radius of the vehicle, a minimum clearance from one or more objects in the surrounding environment of the vehicle, and a minimum clearance from lane boundaries.

7. The method according to claim 3, wherein, The one or more indicators include a visual indication of a cost associated with the proposed trajectory, and wherein the cost of the proposed trajectory corresponds to a magnitude of a deviation from the one or more constraints imposed by the motion planner of the vehicle.

8. The method according to claim 7, wherein, The visual indication identifies one or more portions of the proposed trajectory in which the proposed trajectory is at least one of: within a threshold distance of an object, off a drivable surface, and crossing a lane boundary.

9. The method according to claim 1, wherein The one or more user inputs define the proposed trajectory by at least specifying one or more nodes or segments that form the proposed trajectory.

10. The method according to claim 1, wherein, The one or more constraints include that the proposed trajectory is within a drivable surface.

11. The method according to claim 1, wherein The one or more constraints include that the proposed trajectory does not collide with one or more objects present in the surrounding environment of the vehicle.

12. The method according to claim 1, wherein, Determining the one or more constraints imposed by the motion planner of the vehicle based on a map received from the vehicle.

13. The method according to claim 1, wherein Determining the one or more constraints imposed by the motion planner of the vehicle based on locations of one or more objects tracked by the vehicle.

14. The method according to claim 1, further comprising: In response to successful verification of the proposed trajectory, the proposed trajectory is sent to the motion planner of the vehicle by the at least one data processor.

15. The method according to claim 1, further comprising: In response to unsuccessful verification of the proposed trajectory, a notification for redrawing the proposed trajectory is generated by the at least one data processor.

16. The method according to claim 15, further comprising: Receiving, by the at least one data processor, one or more user inputs for redrawing the proposed trajectory for the vehicle; Verifying, by the at least one data processor, the redrawn trajectory to determine whether the redrawn trajectory satisfies the one or more constraints; and In response to successful verification of the redrawn trajectory, sending the redrawn trajectory to the motion planner of the vehicle by the at least one data processor.

17. A system, comprising: At least one data processor; And At least one memory storing instructions, wherein the instructions, when executed by the at least one data processor, result in operations that include: Receiving one or more user inputs defining a proposed trajectory for a vehicle; Determining one or more constraints imposed by the motion planner of the vehicle; Verifying the proposed trajectory to determine whether the proposed trajectory satisfies the one or more constraints; and In response to unsuccessful verification of the proposed trajectory, preventing the proposed trajectory from being sent to the motion planner of the vehicle.

18. The method according to claim 1, wherein The one or more constraints include that the proposed trajectory is at least one of the following: within a drivable surface and not colliding with one or more objects existing in the surrounding environment of the vehicle.

19. The method according to claim 1, further comprising: In response to unsuccessful verification of the proposed trajectory, generating, by the at least one data processor, a notification for redrawing the proposed trajectory; Receiving, by the at least one data processor, one or more user inputs for redrawing the proposed trajectory for the vehicle; Verifying, by the at least one data processor, the redrawn trajectory to determine whether the redrawn trajectory satisfies the one or more constraints; and In response to successful verification of the redrawn trajectory, sending the redrawn trajectory to the motion planner of the vehicle by the at least one data processor.

20. A non-transitory computer-readable medium stores instructions, wherein, The instructions, when executed by at least one data processor, result in operations that include: Receiving one or more user inputs defining a proposed trajectory for a vehicle; Determining one or more constraints imposed by the motion planner of the vehicle; Verifying the proposed trajectory to determine whether the proposed trajectory satisfies the one or more constraints; and In response to the verification of the proposed trajectory being unsuccessful, prevent the proposed trajectory from being sent to the motion planner of the vehicle.

Images