Path generation based on predicted actions

Abstract

Methods and systems are provided for improving predicted semantic behavior filtering. A method for operating an autonomous vehicle is provided. The method includes obtaining, by at least one processor, semantic image data associated with an environment in which the autonomous vehicle is operating. The method includes determining, by at least one processor, a set of agents in the environment based on the semantic image data. The method includes determining a set of predicted actions of at least one of the set of agents. The method includes determining a set of secondary predicted actions for the at least one primary agent from the set of predicted actions using semantic data. The method includes determining a set of primary predicted actions other than secondary predicted actions from the set of predicted actions. The method includes generating a path of the autonomous vehicle based on the set of primary predicted actions.

Description

Background Art

[0001] Autonomous vehicles can use a variety of methods and systems for determining the trajectory of autonomous vehicles. However, these methods and systems may require high computing power, which may lead to low computational efficiency. In addition, the methods and systems may slow down the reaction time of the autonomous vehicle, which may lead to complications in the real world. BRIEF DESCRIPTION OF THE DRAWINGS

[0002] Figure 1 is an example environment in which a vehicle including one or more components of an autonomous system may be implemented;

[0003] Figure 2 is a diagram of one or more systems of a vehicle including an autonomous system;

[0004] Figure 3 yes Figure 1 and Figure 2 a diagram of one or more devices and / or components of one or more systems;

[0005] Figure 4A is a diagram of some components of an autonomous system;

[0006] Figure 4B is a graph of the implementation of a neural network;

[0007] Figure 4C and Figure 4D is a diagram illustrating an example operation of a CNN;

[0008] Figure 5A and Figure 5B is an operational flow chart illustrating an example filtering operation of a planning system;

[0009] Figure 6 is a flow chart illustrating an example of a routine implemented by a planning system for controlling a vehicle based on a path generated using a primary agent;

[0010] Figure 7 is a flow chart illustrating an example of a routine implemented by a planning system for controlling a vehicle based on a path generated using primary actions;

[0011] Figure 8 is a representation of position over time for an implementation of the process of improving predicted semantic behavior filtering in case of unavoidable collisions;

[0012] Fig. 9 is a flow chart illustrating an example of a routine implemented by a planning system for improving predicted semantic behavior filtering in the event of an unavoidable collision. DETAILED DESCRIPTION

[0013] In the following description, for the purpose of explanation, many specific details are set forth in order to provide a thorough understanding of the present disclosure. However, it will be apparent that the embodiments described in the present disclosure can be implemented without these specific details. In some instances, well-known configurations and devices are illustrated in block diagram form to avoid unnecessarily obscuring aspects of the present disclosure.

[0014] In the accompanying drawings, for ease of description, the specific arrangement or order of schematic elements (such as those representing systems, devices, modules, instruction blocks and / or data elements, etc.) is illustrated. However, those skilled in the art will understand that, unless explicitly described, the specific order or arrangement of schematic elements in the accompanying drawings is not intended to mean that a specific processing order or sequence, or separation of processing is required. In addition, unless explicitly described, the inclusion of schematic elements in the accompanying drawings is not intended to mean that such elements are required in all embodiments, nor is it intended to mean that the features represented by such elements cannot be included in some embodiments or cannot be combined with other elements in some embodiments.

[0015] In addition, in the accompanying drawings, connecting elements (such as solid or dotted lines or arrows, etc.) are used to illustrate the connection, relationship or association between or among two or more other schematic elements, and the absence of any such connecting elements is not intended to mean that there can be no connection, relationship or association. In other words, some connections, relationships or associations between elements are not illustrated in the accompanying drawings so as not to obscure the present disclosure. In addition, for ease of illustration, a single connecting element can be used to represent multiple connections, relationships or associations between elements. For example, if the connecting element represents the communication of a signal, data or instruction (e.g., "software instruction"), it will be understood by those skilled in the art that such an element can represent one or more than one signal path (e.g., bus) that may be needed to affect the communication.

[0016] Although the terms "first", "second" and / or "third", etc. are used to describe various elements, these elements should not be limited by these terms. The terms "first", "second" and / or "third" are only used to distinguish one element from another. For example, a first contact may be referred to as a second contact, and similarly, a second contact may be referred to as a first contact without departing from the scope of the described embodiments. Both the first contact and the second contact are contacts, but they are not the same contacts.

[0017] The terms used in the specification of the various embodiments described herein are included only for the purpose of describing specific embodiments and are not intended to be limiting. As used in the specification of the various embodiments described and the appended claims, the singular forms "a", "an" and "the" are also intended to include plural forms and can be used interchangeably with "one or more than one" or "at least one", unless the context clearly indicates otherwise. It will also be understood that the term "and / or" as used herein refers to and includes any and all possible combinations of one or more than one of the associated listed items. It will also be understood that when the terms "include", "comprise", "have" and / or "have" are used in this specification, the stated features, integers, steps, operations, elements and / or components are specifically stated, but the presence or addition of one or more than one other features, integers, steps, operations, elements, components and / or groups thereof are not excluded.

[0018] As used herein, the terms "communication" and "communicating" refer to at least one of receiving, receiving, transmitting, transmitting and / or providing information (or information represented by, for example, data, signals, messages, instructions and / or commands, etc.). For a unit (e.g., a device, a system, a component of a device or system, and / or a combination thereof) to communicate with another unit, this means that the unit is able to directly or indirectly receive information from the other unit and / or send (e.g., transmit) information to the other unit. This can refer to a direct or indirect connection that is wired and / or wireless in nature. In addition, even if the transmitted information can be modified, processed, relayed and / or routed between the first unit and the second unit, the two units can communicate with each other. For example, even if the first unit passively receives information and does not actively transmit information to the second unit, the first unit can communicate with the second unit. As another example, if at least one intermediary unit (e.g., a third unit located between the first unit and the second unit) processes the information received from the first unit and transmits the processed information to the second unit, the first unit can communicate with the second unit. In some embodiments, a message may refer to a network packet (eg, a data packet, etc.) that includes data.

[0019] As used herein, the term "if" is optionally interpreted to mean "when," "at," "in response to being determined to be," and / or "in response to being detected," etc., depending on the context. Similarly, the phrases "if it is determined" or "if [the stated condition or event] is detected" are optionally interpreted to mean "when determining," "in response to being determined to be" or "when [the stated condition or event] is detected," and / or "in response to being detected," etc., depending on the context. In addition, as used herein, the terms "have," "have," or "possess," etc. are intended to be open-ended terms. Furthermore, unless expressly stated otherwise, the phrase "based on" is intended to mean "based at least in part on."

[0020] Reference will now be made in detail to embodiments, examples of which are illustrated in the accompanying drawings. In the following detailed description, numerous specific details are set forth in order to provide a thorough understanding of the various embodiments described. However, it will be apparent to one of ordinary skill in the art that the various embodiments described may be implemented without these specific details. In other cases, well-known methods, processes, components, circuits, and networks have not yet been described in detail in order not to unnecessarily obscure aspects of the embodiments.

[0021] General Overview

[0022] In some aspects and / or embodiments, the systems, methods, and computer program products described herein include and / or implement semantic behavior filtering for improving predictions. A prediction engine can be used to predict the actions of an agent in a scenario (or vehicle scenario). However, in some cases, the prediction engine may be prone to errors and / or may generate low-quality predictions or generate unlikely predictions for a specific vehicle scenario. For example, a prediction engine may spend computing resources to generate predictions for an agent that is unlikely to affect an autonomous vehicle, and / or may predict unrealistic agent actions. Such a prediction engine can predict based on behaviors that are likely, rather than predicting based on behaviors that are likely. Predictions based on behaviors that are likely but unlikely may result in inefficient or unsafe scenarios in a given vehicle scenario.

[0023] To address these issues, the system can remove or filter out low-quality predictions. In some cases, the system can filter agents based on contextual information before requesting the prediction engine to generate predictions for agents in the scene. For example, the system can filter out vehicles that are behind a stop light or stop sign so that the prediction engine does not expend resources predicting what these vehicles will do. Filtering out agents before determining predictions can reduce the computational resources used by the vehicle to determine its path through the scene. In some cases, the system can filter agents after requesting predictions from the prediction engine to take the predicted actions into account when filtering.

[0024] Additionally, the system can filter some or all predictions for a particular agent based on contextual information in the scene. For example, the system can filter out predictions that a pedestrian walking parallel to a street will suddenly change direction and run into the street. Filtering out low-quality predictions can reduce the computational resources used by the vehicle to determine its path through the scene and / or reduce the likelihood that the vehicle will take unsafe actions (e.g., sudden braking) based on low-quality predictions.

[0025] In addition, the system can process the scene to determine that a collision is likely to occur. This can include determining that a collision with other objects is likely to occur regardless of whether the vehicle is accelerating or decelerating. In some such cases, the system can take measures to mitigate losses based on determining that a collision is likely to occur. In some cases, the measures may include acceleration. For example, the vehicle can be accelerated to adjust the position of the vehicle where a collision is likely to occur. With the implementation of the systems, methods, and computer program products described herein, techniques for improving predicted semantic behavior filtering can be accomplished. Some advantages of these techniques include: reducing the computing resources used to determine the path, being able to determine the path in less time (i.e., better responsiveness) and with fewer computing resources, being able to spend computing resources on higher quality predictions to achieve a more efficient path, and being able to reduce losses during a collision.

[0026] Reference now Figure 1, illustrates an example environment 100 in which vehicles including autonomous systems and vehicles not including autonomous systems operate. As illustrated, the environment 100 includes vehicles 102a-102n, objects 104a-104n, routes 106a-106n, areas 108, vehicle-to-infrastructure (V2I) devices 110, a network 112, a remote autonomous vehicle (AV) system 114, a fleet management system 116, and a V2I system 118. The vehicles 102a-102n, the vehicle-to-infrastructure (V2I) devices 110, the network 112, the autonomous vehicle (AV) system 114, the fleet management system 116, and the V2I system 118 are interconnected (e.g., establish connections for communication, etc.) via wired connections, wireless connections, or a combination of wired or wireless connections. In some embodiments, objects 104a-104n are interconnected with at least one of vehicles 102a-102n, vehicle-to-infrastructure (V2I) devices 110, networks 112, autonomous vehicle (AV) systems 114, fleet management systems 116, and V2I systems 118 via wired connections, wireless connections, or a combination of wired or wireless connections.

[0027] Vehicles 102a-102n (individually referred to as vehicles 102 and collectively referred to as vehicles 102) include at least one device configured to transport goods and / or people. In some embodiments, vehicles 102 are configured to communicate with V2I devices 110, remote AV systems 114, fleet management systems 116, and / or V2I systems 118 via network 112. In some embodiments, vehicles 102 include cars, buses, trucks, and / or trains, etc. In some embodiments, vehicles 102 are similar to vehicles 200 described herein (see Figure 2 ). In some embodiments, vehicles 200 in the set of vehicles 200 are associated with an autonomous queue manager. In some embodiments, vehicles 102 travel along respective routes 106a-106n (individually referred to as routes 106 and collectively referred to as routes 106) as described herein. In some embodiments, one or more vehicles 102 include an autonomous system (e.g., an autonomous system that is the same as or similar to autonomous system 202).

[0028] Objects 104a-104n (individually referred to as objects 104 and collectively referred to as objects 104) include, for example, at least one vehicle, at least one pedestrian, at least one cyclist, and / or at least one structure (e.g., a building, a sign, a fire hydrant, etc.), etc. Each object 104 is stationary (e.g., located at a fixed location and over a period of time) or moves (e.g., has a speed and is associated with at least one trajectory). In some embodiments, objects 104 are associated with corresponding locations in area 108.

[0029] Routes 106a-106n (individually referred to as routes 106 and collectively referred to as routes 106) are each associated with (e.g., specify a series of actions (also referred to as trajectories) connecting states along which the AV can navigate. Each route 106 begins at an initial state (e.g., a state corresponding to a first spatiotemporal location and / or speed, etc.) and ends at a final target state (e.g., a state corresponding to a second spatiotemporal location different from the first spatiotemporal location) or a target zone (e.g., a subspace of acceptable states (e.g., terminal states)). In some embodiments, the first state includes a location where one or more individuals will board the AV, and the second state or zone includes one or more locations where one or more individuals boarding the AV will disembark. In some embodiments, routes 106 include multiple acceptable state sequences (e.g., multiple spatiotemporal location sequences) that are associated with (e.g., define multiple trajectories). In an example, routes 106 include only high-level actions or imprecise state locations, such as a series of connecting roads indicating a change of direction at a roadway intersection, etc. Additionally or alternatively, the route 106 may include more precise actions or states, such as, for example, specific target lanes or precise locations within lane regions and target speeds at those locations, etc. In an example, the route 106 includes a plurality of precise state sequences along at least one high-level action with a limited look-ahead horizon to an intermediate target, wherein a combination of consecutive iterations of the limited horizon state sequences cumulatively correspond to a plurality of trajectories that collectively form a high-level route terminating at a final target state or region.

[0030] The area 108 includes a physical area (e.g., a geographic region) in which the vehicle 102 can navigate. In an example, the area 108 includes at least one state (e.g., a country, a province, a separate state of a plurality of states included in a country, etc.), at least a portion of a state, at least one city, at least a portion of a city, etc. In some embodiments, the area 108 includes at least one named thoroughfare (referred to herein as a "road"), such as a highway, an interstate highway, a parkway, a city street, etc. Additionally or alternatively, in some examples, the area 108 includes at least one unnamed road, such as a driveway, a section of a parking lot, a section of an open space and / or undeveloped area, a dirt road, etc. In some embodiments, the road includes at least one lane (e.g., a portion of the road that the vehicle 102 can traverse). In an example, the road includes at least one lane associated with (e.g., identified based on) at least one lane marking line.

[0031] The vehicle-to-infrastructure (V2I) device 110 (sometimes referred to as a vehicle-to-infrastructure or vehicle-to-everything (V2X) device) includes at least one device configured to communicate with the vehicle 102 and / or the V2I system 118. In some embodiments, the V2I device 110 is configured to communicate with the vehicle 102, the remote AV system 114, the fleet management system 116, and / or the V2I system 118 via the network 112. In some embodiments, the V2I device 110 includes a radio frequency identification (RFID) device, a sign, a camera (e.g., a two-dimensional (2D) and / or three-dimensional (3D) camera), lane markings, street lights, parking meters, etc. In some embodiments, the V2I device 110 is configured to communicate directly with the vehicle 102. Additionally or alternatively, in some embodiments, the V2I device 110 is configured to communicate with the vehicle 102, the remote AV system 114, and / or the fleet management system 116 via the V2I system 118. In some embodiments, the V2I device 110 is configured to communicate with the V2I system 118 via the network 112.

[0032] The network 112 includes one or more wired and / or wireless networks. In an example, the network 112 includes a cellular network (e.g., a long-term evolution (LTE) network, a third generation (3G) network, a fourth generation (4G) network, a fifth generation (5G) network, a code division multiple access (CDMA) network, etc.), a public land mobile network (PLMN), a local area network (LAN), a wide area network (WAN), a metropolitan area network (MAN), a telephone network (e.g., a public switched telephone network (PSTN)), a private network, an ad hoc network, an intranet, the Internet, a fiber-based network, a cloud computing network, etc., and / or a combination of some or all of these networks, etc.

[0033] The remote AV system 114 includes at least one device configured to communicate with the vehicle 102, the V2I device 110, the network 112, the fleet management system 116, and / or the V2I system 118 via the network 112. In an example, the remote AV system 114 includes a server, a server group, and / or other similar devices. In some embodiments, the remote AV system 114 is co-located with the fleet management system 116. In some embodiments, the remote AV system 114 participates in the installation of some or all of the components of the vehicle (including autonomous systems, autonomous vehicle computing, and / or software implemented by autonomous vehicle computing, etc.). In some embodiments, the remote AV system 114 maintains (e.g., updates and / or replaces) these components and / or software during the life of the vehicle.

[0034] The queue management system 116 includes at least one device configured to communicate with the vehicles 102, the V2I devices 110, the remote AV system 114, and / or the V2I system 118. In an example, the queue management system 116 includes a server, a server group, and / or other similar devices. In some embodiments, the queue management system 116 is associated with a ridesharing company (e.g., an organization for controlling the operation of multiple vehicles (e.g., vehicles including autonomous systems and / or vehicles not including autonomous systems)).

[0035] In some embodiments, the V2I system 118 includes at least one device configured to communicate with the vehicle 102, the V2I device 110, the remote AV system 114, and / or the fleet management system 116 via the network 112. In some examples, the V2I system 118 is configured to communicate with the V2I device 110 via a connection other than the network 112. In some embodiments, the V2I system 118 includes a server, a server group, and / or other similar devices. In some embodiments, the V2I system 118 is associated with a municipality or a private agency (e.g., a private agency for maintaining the V2I device 110, etc.).

[0036] supply Figure 1 The number and arrangement of elements illustrated are examples. Figure 1 There may be additional elements, fewer elements, different elements, and / or differently arranged elements than those illustrated. Additionally or alternatively, at least one element of environment 100 may be described as being Figure 1 Additionally or alternatively, at least one set of elements of environment 100 may perform one or more functions described as being performed by at least one different set of elements of environment 100.

[0037] Reference now Figure 2 , vehicle 200 (which is Figure 1 102) includes, or is associated with, autonomous system 202, powertrain control system 204, steering control system 206, and braking system 208. In some embodiments, vehicle 200 is similar to vehicle 102 (see Figure 1 ) is the same or similar. In some embodiments, the autonomous system 202 is configured to give the vehicle 200 autonomous driving capabilities (e.g., implementing at least one of the following driving automatic or maneuver-based functions, features and / or devices, etc., which enable the vehicle 200 to operate partially or completely without human intervention, including but not limited to fully autonomous vehicles (e.g., vehicles that abandon reliance on human intervention, such as Level 5 ADS-operated vehicles, etc.), highly autonomous vehicles (e.g., vehicles that abandon reliance on human intervention in certain situations, such as Level 4 ADS-operated vehicles, etc.), and / or conditionally autonomous vehicles (e.g., vehicles that abandon reliance on human intervention in limited situations, such as Level 3 ADS-operated vehicles, etc.), etc.). In one embodiment, the autonomous system 202 includes the operational or tactical functions required to enable the vehicle 200 to operate in road traffic and continuously perform part or all of a dynamic driving task (DDT). In another embodiment, the autonomous system 202 includes an advanced driver assistance system (ADAS) including driver support features. The autonomous system 202 supports various levels of driving automation ranging from no driving automation (e.g., level 0) to full driving automation (e.g., level 5). For a detailed description of fully autonomous vehicles and highly autonomous vehicles, reference may be made to SAE International's standard J3016: Taxonomy and Definitions for Terms Related to On-Road Motor Vehicle Automated Driving Systems, the entire contents of which are incorporated by reference. In some embodiments, the vehicle 200 is associated with an autonomous queue manager and / or a ridesharing company.

[0038] Autonomous system 202 includes a sensor suite that includes one or more devices such as camera 202a, LiDAR sensor 202b, Radar sensor 202c, and microphone 202d. In some embodiments, autonomous system 202 may include more or fewer devices and / or different devices (e.g., ultrasonic sensors, inertial sensors, GPS receivers (discussed below), and / or odometer sensors for generating data associated with an indication of the distance that vehicle 200 has traveled, etc.). In some embodiments, autonomous system 202 uses one or more devices included in autonomous system 202 to generate data associated with environment 100 described herein. Data generated by one or more devices of autonomous system 202 can be used by one or more systems described herein to observe the environment (e.g., environment 100) in which vehicle 200 is located. In some embodiments, autonomous system 202 includes communication device 202e, autonomous vehicle computing 202f, drive-by-wire (DBW) system 202h, and safety controller 202g.

[0039] The camera 202a includes a communication device 202e, an autonomous vehicle computer 202f, and / or a safety controller 202g configured to communicate with the communication device 202e via a bus (e.g., Figure 3 The camera 202a includes at least one device for communicating with the autonomous vehicle computing 202f (e.g., an image processing unit 202a, a bus ... Figure 1In some embodiments, the autonomous vehicle computing 202f determines a depth to one or more objects in a field of view of at least two of the plurality of cameras based on image data from the at least two cameras. In some embodiments, the camera 202a is configured to capture images of objects within a distance relative to the camera 202a (e.g., up to 100 meters and / or up to 1 kilometer, etc.). Thus, the camera 202a includes features such as sensors and lenses that are optimized for sensing objects at one or more distances relative to the camera 202a.

[0040] In an embodiment, the camera 202a includes at least one camera configured to capture one or more images associated with one or more traffic lights, street signs, and / or other physical objects that provide visual navigation information. In some embodiments, the camera 202a generates traffic light data associated with the one or more images. In some examples, the camera 202a generates TLD (traffic light detection) data associated with one or more images including a format (e.g., RAW, JPEG, and / or PNG, etc.). In some embodiments, the camera 202a that generates TLD data differs from other systems incorporating cameras described herein in that the camera 202a may include one or more cameras with a wide field of view (e.g., a wide-angle lens, a fisheye lens, and / or a lens with a viewing angle of approximately 120 degrees or greater, etc.) to generate images related to as many physical objects as possible.

[0041] The laser detection and ranging (LiDAR) sensor 202b includes a communication device 202e, an autonomous vehicle computing device 202f, and / or a safety controller 202g configured to communicate with the communication device 202e via a bus (e.g., Figure 3The LiDAR sensor 202b includes at least one device that communicates with a bus (the same or similar bus as the bus 302 of the embodiment of the present invention). The LiDAR sensor 202b includes a system configured to emit light from a light emitter (e.g., a laser emitter). The light emitted by the LiDAR sensor 202b includes light outside the visible spectrum (e.g., infrared light, etc.). In some embodiments, during operation, the light emitted by the LiDAR sensor 202b encounters a physical object (e.g., a vehicle) and is reflected back to the LiDAR sensor 202b. In some embodiments, the light emitted by the LiDAR sensor 202b does not penetrate the physical object encountered by the light. The LiDAR sensor 202b also includes at least one light detector that detects the light emitted from the light emitter after encountering the physical object. In some embodiments, at least one data processing system associated with the LiDAR sensor 202b generates an image (e.g., a point cloud and / or a combined point cloud, etc.) representing objects included in the field of view of the LiDAR sensor 202b. In some examples, at least one data processing system associated with the LiDAR sensor 202b generates an image representing the boundaries of the physical object and / or the surface of the physical object (e.g., the topology of the surface), etc. In such examples, the image is used to determine the boundaries of the physical object in the field of view of the LiDAR sensor 202b.

[0042] The radio detection and ranging (Radar) sensor 202c includes a sensor configured to communicate with the communication device 202e, the autonomous vehicle computing device 202f, and / or the safety controller 202g via a bus (e.g., Figure 3 At least one device for communicating with a bus (same or similar bus as bus 302 of the embodiment of the present invention). Radar sensor 202c includes a system configured to transmit (pulsed or continuous) radio waves. The radio waves transmitted by Radar sensor 202c include radio waves within a predetermined spectrum. In some embodiments, during operation, the radio waves transmitted by Radar sensor 202c encounter physical objects and are reflected back to Radar sensor 202c. In some embodiments, the radio waves transmitted by Radar sensor 202c are not reflected by some objects. In some embodiments, at least one data processing system associated with Radar sensor 202c generates a signal representing an object included in the field of view of Radar sensor 202c. For example, at least one data processing system associated with Radar sensor 202c generates an image representing the boundary of a physical object and / or the surface of a physical object (e.g., the topology of the surface), etc. In some examples, the image is used to determine the boundary of a physical object in the field of view of Radar sensor 202c.

[0043] The microphone 202d includes a microphone configured to communicate with the communication device 202e, the autonomous vehicle computing device 202f, and / or the safety controller 202g via a bus (e.g., Figure 3 At least one device for communicating with the vehicle 200 (the same or similar bus as bus 302 of FIG. 1 ). Microphone 202d includes one or more microphones (e.g., an array microphone and / or an external microphone, etc.) that capture an audio signal and generate data associated with (e.g., representing) the audio signal. In some examples, microphone 202d includes a transducer device and / or the like. In some embodiments, one or more systems described herein can receive the data generated by microphone 202d and determine the position (e.g., distance, etc.) of an object relative to vehicle 200 based on an audio signal associated with the data.

[0044] The communication device 202e includes at least one device configured to communicate with the camera 202a, the LiDAR sensor 202b, the Radar sensor 202c, the microphone 202d, the autonomous vehicle computing 202f, the safety controller 202g, and / or the drive-by-wire (DBW) system 202h. For example, the communication device 202e may include at least one device configured to communicate with the camera 202a, the LiDAR sensor 202b, the Radar sensor 202c, the microphone 202d, the autonomous vehicle computing 202f, the safety controller 202g, and / or the drive-by-wire (DBW) system 202h. Figure 3 The communication device 202e may be a device that is the same as or similar to the communication interface 314 of the vehicle. In some embodiments, the communication device 202e includes a vehicle-to-vehicle (V2V) communication device (eg, a device for enabling wireless communication of data between vehicles).

[0045] Autonomous vehicle computing 202f includes at least one device configured to communicate with camera 202a, LiDAR sensor 202b, Radar sensor 202c, microphone 202d, communication device 202e, safety controller 202g, and / or DBW system 202h. In some examples, autonomous vehicle computing 202f includes devices such as client devices, mobile devices (e.g., cellular phones and / or tablet computers, etc.), and / or servers (e.g., computing devices including one or more central processing units and / or graphics processing units, etc.). In some embodiments, autonomous vehicle computing 202f is the same or similar to autonomous vehicle computing 400 described herein. Additionally or alternatively, in some embodiments, autonomous vehicle computing 202f is configured to communicate with an autonomous vehicle system (e.g., with Figure 1 remote AV system 114 of the same or similar autonomous vehicle system), a fleet management system (e.g., Figure 1 of the same or similar queue management system as the queue management system 116), V2I devices (e.g., Figure 1 V2I device 110 that is the same as or similar to V2I device 110) and / or V2I system (e.g., Figure 1The V2I system 118 may communicate with the same or similar V2I system.

[0046] Safety controller 202g includes at least one device configured to communicate with camera 202a, LiDAR sensor 202b, Radar sensor 202c, microphone 202d, communication device 202e, autonomous vehicle computing 202f, and / or DBW system 202h. In some examples, safety controller 202g includes one or more controllers (electrical controllers and / or electromechanical controllers, etc.) configured to generate and / or transmit control signals to operate one or more devices of vehicle 200 (e.g., powertrain control system 204, steering control system 206, and / or braking system 208, etc.). In some embodiments, safety controller 202g is configured to generate control signals that take precedence over (e.g., override) control signals generated and / or transmitted by autonomous vehicle computing 202f.

[0047] The DBW system 202h includes at least one device configured to communicate with the communication device 202e and / or the autonomous vehicle computing 202f. In some examples, the DBW system 202h includes one or more controllers (e.g., electrical controllers and / or electromechanical controllers, etc.) configured to generate and / or transmit control signals to operate one or more devices of the vehicle 200 (e.g., powertrain control system 204, steering control system 206, and / or braking system 208, etc.). Additionally or alternatively, one or more controllers of the DBW system 202h are configured to generate and / or transmit control signals to operate at least one different device of the vehicle 200 (e.g., turn signals, headlights, door locks, and / or windshield wipers, etc.).

[0048] The powertrain control system 204 includes at least one device configured to communicate with the DBW system 202h. In some examples, the powertrain control system 204 includes at least one controller and / or actuator, etc. In some embodiments, the powertrain control system 204 receives control signals from the DBW system 202h, and the powertrain control system 204 causes the vehicle 200 to perform longitudinal vehicle movement such as starting forward movement, stopping forward movement, starting backward movement, stopping backward movement, accelerating in a certain direction, decelerating in a certain direction, etc., or lateral vehicle movement such as turning left and / or turning right. In an example, the powertrain control system 204 increases, maintains the same, or decreases the energy (e.g., fuel and / or electricity, etc.) provided to the motor of the vehicle, thereby rotating or not rotating at least one wheel of the vehicle 200.

[0049] The steering control system 206 includes at least one device configured to rotate one or more wheels of the vehicle 200. In some examples, the steering control system 206 includes at least one controller and / or actuator, etc. In some embodiments, the steering control system 206 rotates the two front wheels and / or the two rear wheels of the vehicle 200 to the left or right to turn the vehicle 200 left or right. In other words, the steering control system 206 causes the activity required to adjust the y-axis component of the vehicle's motion.

[0050] Braking system 208 includes at least one device configured to actuate one or more brakes to decelerate and / or hold vehicle 200 stationary. In some examples, braking system 208 includes at least one controller and / or actuator configured to cause one or more calipers associated with one or more wheels of vehicle 200 to close on respective rotors of vehicle 200. Additionally or alternatively, in some examples, braking system 208 includes an automatic emergency braking (AEB) system and / or a regenerative braking system, etc.

[0051] In some embodiments, vehicle 200 includes at least one platform sensor (not explicitly illustrated) for measuring or inferring a property of a state or condition of vehicle 200. In some examples, vehicle 200 includes platform sensors such as a global positioning system (GPS) receiver, an inertial measurement unit (IMU), wheel rate sensors, wheel brake pressure sensors, wheel torque sensors, engine torque sensors, and / or steering angle sensors. Although in Figure 2 Braking system 208 is shown located on the proximal side of vehicle 200 , but braking system 208 may be located anywhere in vehicle 200 .

[0052] Reference now Figure 3 , a schematic diagram illustrating an apparatus 300. As illustrated, the apparatus 300 includes a processor 304, a memory 306, a storage component 308, an input interface 310, an output interface 312, a communication interface 314, and a bus 302. In some embodiments, the apparatus 300 corresponds to: at least one apparatus of the vehicle 102 (e.g., at least one apparatus of a system of the vehicle 102); [continued in a similar manner Figure 1-3 and / or one or more devices of network 112 (e.g., one or more devices of a system of network 112). In some embodiments, one or more devices of vehicle 102 (e.g., one or more devices of a system of vehicle 102), [continued in a similar manner Figure 1-3All devices contemplated in the present invention], and / or one or more devices of the network 112 (e.g., one or more devices of the system of the network 112) include at least one device 300 and / or at least one component of the device 300. Figure 3 As shown, apparatus 300 includes a bus 302 , a processor 304 , a memory 306 , a storage component 308 , an input interface 310 , an output interface 312 , and a communication interface 314 .

[0053] The bus 302 includes components that permit communication between components of the device 300. In some embodiments, the processor 304 includes a processor (e.g., a central processing unit (CPU), a graphics processing unit (GPU), and / or an accelerated processing unit (APU), etc.), a microphone, a digital signal processor (DSP), and / or any processing component that can be programmed to perform at least one function (e.g., a field programmable gate array (FPGA) and / or an application specific integrated circuit (ASIC), etc.). The memory 306 includes a random access memory (RAM), a read-only memory (ROM), and / or another type of dynamic and / or static storage device (e.g., flash memory, magnetic memory, and / or optical memory, etc.) that stores data and / or instructions for use by the processor 304.

[0054] Storage component 308 stores data and / or software related to the operation and use of device 300. In some examples, storage component 308 includes a hard disk (e.g., a magnetic disk, an optical disk, a magneto-optical disk, and / or a solid-state disk, etc.), a compact disk (CD), a digital versatile disk (DVD), a floppy disk, a cassette, a tape, a CD-ROM, a RAM, a PROM, an EPROM, a FLASH-EPROM, an NV-RAM, and / or another type of computer-readable medium, and a corresponding drive.

[0055] The input interface 310 includes components that permit the device 300 to receive information, such as via user input (e.g., a touch screen display, a keyboard, a keypad, a mouse, a button, a switch, a microphone, and / or a camera, etc.). Additionally or alternatively, in some embodiments, the input interface 310 includes a sensor for sensing information (e.g., a global positioning system (GPS) receiver, an accelerometer, a gyroscope, and / or an actuator, etc.). The output interface 312 includes components for providing output information from the device 300 (e.g., a display, a speaker, and / or one or more light emitting diodes (LEDs), etc.).

[0056] In some embodiments, communication interface 314 includes a transceiver-like component (e.g., a transceiver and / or a separate receiver and transmitter, etc.) that permits device 300 to communicate with other devices via a wired connection, a wireless connection, or a combination of a wired connection and a wireless connection. In some examples, communication interface 314 permits device 300 to receive information from another device and / or provide information to another device. In some examples, communication interface 314 includes an Ethernet interface, an optical interface, a coaxial interface, an infrared interface, a radio frequency (RF) interface, a universal serial bus (USB) interface, interface and / or cellular network interface, etc.

[0057] In some embodiments, the device 300 performs one or more processes described herein. The device 300 performs these processes based on the processor 304 executing software instructions stored by a computer-readable medium such as a memory 306 and / or a storage component 308. Computer-readable media (e.g., non-transitory computer-readable media) are defined herein as non-transitory memory devices. Non-transitory memory devices include storage space located within a single physical storage device or storage space distributed across multiple physical storage devices.

[0058] In some embodiments, the software instructions are read into the memory 306 and / or storage component 308 from another computer-readable medium or from another device via the communication interface 314. The software instructions stored in the memory 306 and / or storage component 308, when executed, cause the processor 304 to perform one or more processes described herein. Additionally or alternatively, hardwired circuitry is used in place of or in combination with the software instructions to perform one or more processes described herein. Therefore, unless expressly stated otherwise, the embodiments described herein are not limited to any specific combination of hardware circuitry and software.

[0059] The memory 306 and / or the storage component 308 include a data storage unit or at least one data structure (e.g., a database, etc.). The device 300 can receive information from the data storage unit or at least one data structure in the memory 306 or the storage component 308, store information in the data storage unit or at least one data structure, communicate information to the data storage unit or at least one data structure, or search for information stored in the data storage unit or at least one data structure. In some examples, the information includes network data, input data, output data, or any combination thereof.

[0060] In some embodiments, the device 300 is configured to execute software instructions stored in the memory 306 and / or the memory of another device (e.g., another device that is the same as or similar to the device 300). As used herein, the term "module" refers to at least one instruction stored in the memory 306 and / or the memory of another device, which, when executed by the processor 304 and / or the processor of another device (e.g., another device that is the same as or similar to the device 300), causes the device 300 (e.g., at least one component of the device 300) to perform one or more processes described herein. In some embodiments, the module is implemented in software, firmware, and / or hardware, etc.

[0061] supply Figure 3 The number and arrangement of components illustrated are examples. Figure 3 The device 300 may include additional components, fewer components, different components, or differently arranged components than those illustrated. Additionally or alternatively, a set of components (e.g., one or more components) of the device 300 may perform one or more functions described as being performed by another component or set of components of the device 300.

[0062] Reference now Figure 4A, illustrates an example block diagram of an autonomous vehicle computing 400 (sometimes referred to as an "AV stack"). As illustrated, the autonomous vehicle computing 400 includes a perception system 402 (sometimes referred to as a perception module), a planning system 404 (sometimes referred to as a planning module), a positioning system 406 (sometimes referred to as a positioning module), a control system 408 (sometimes referred to as a control module), and a database 410. In some embodiments, the perception system 402, the planning system 404, the positioning system 406, the control system 408, and the database 410 are included in and / or implemented in an automatic navigation system of a vehicle (e.g., the autonomous vehicle computing 202f of the vehicle 200). Additionally or alternatively, in some embodiments, the perception system 402, the planning system 404, the positioning system 406, the control system 408, and the database 410 are included in one or more independent systems (e.g., one or more systems that are the same or similar to the autonomous vehicle computing 400, etc.). In some examples, the perception system 402, planning system 404, positioning system 406, control system 408, and database 410 are included in one or more independent systems located in the vehicle and / or at least one remote system as described herein. In some embodiments, any and / or all of the systems included in the autonomous vehicle computing 400 are implemented in software (e.g., software instructions stored in a memory), computer hardware (e.g., by a microprocessor, microcontroller, application specific integrated circuit (ASIC) and / or field programmable gate array (FPGA), etc.), or a combination of computer software and computer hardware. It will also be understood that in some embodiments, the autonomous vehicle computing 400 is configured to communicate with a remote system (e.g., an autonomous vehicle system that is the same or similar to the remote AV system 114, a fleet management system 116 that is the same or similar to the fleet management system 116, and / or a V2I system that is the same or similar to the V2I system 118, etc.).

[0063] In some embodiments, the perception system 402 receives data associated with at least one physical object in the environment (e.g., data used by the perception system 402 to detect at least one physical object) and classifies the at least one physical object. In some examples, the perception system 402 receives image data captured by at least one camera (e.g., camera 202a), the image being associated with one or more physical objects within the field of view of the at least one camera (e.g., representing the one or more physical objects). In such examples, the perception system 402 classifies the at least one physical object based on one or more groups of physical objects (e.g., bicycles, vehicles, traffic signs, and / or pedestrians, etc.). In some cases, the perception system 402 classifies the at least one physical object using at least one neural network, which is a neural network trained to generate semantic image data from one or more image sensors. In some embodiments, based on the perception system 402 classifying the physical object, the perception system 402 transmits data associated with the classification of the physical object (e.g., semantic image or semantic image data) to the planning system 404.

[0064] In some embodiments, the planning system 404 receives data associated with a destination and generates data associated with at least one route (e.g., route 106) along which a vehicle (e.g., vehicle 102) can travel toward the destination. In some embodiments, the planning system 404 periodically or continuously receives data (e.g., data associated with the classification of physical objects described above) from the perception system 402, and the planning system 404 updates at least one trajectory or generates at least one different trajectory based on the data generated by the perception system 402. In other words, the planning system 404 can perform tasks related to the tactical functions required to operate the vehicle 101 in traffic on the road. Tactical efforts involve maneuvering the vehicle in traffic during the journey, which includes but is not limited to deciding whether and when to overtake another vehicle, change lanes, or select an appropriate rate, acceleration, deceleration, etc. In some embodiments, the planning system 404 receives data associated with an updated position of the vehicle (e.g., vehicle 102) from the positioning system 406, and the planning system 404 updates at least one trajectory or generates at least one different trajectory based on the data generated by the positioning system 406.

[0065] In some embodiments, the positioning system 406 receives data associated with (e.g., representing) a location of a vehicle (e.g., vehicle 102) in an area. In some examples, the positioning system 406 receives LiDAR data associated with at least one point cloud generated by at least one LiDAR sensor (e.g., LiDAR sensor 202b). In some examples, the positioning system 406 receives data associated with at least one point cloud from multiple LiDAR sensors, and the positioning system 406 generates a combined point cloud based on each point cloud. In these examples, the positioning system 406 compares the at least one point cloud or the combined point cloud with a two-dimensional (2D) and / or three-dimensional (3D) map of the area stored in the database 410. Then, based on the positioning system 406 comparing the at least one point cloud or the combined point cloud with the map, the positioning system 406 determines the position of the vehicle in the area. In some embodiments, the map includes a combined point cloud of the area generated before the navigation of the vehicle. In some embodiments, the map includes, but is not limited to, a high-precision map of roadway geometry, a map describing the road network connectivity, a map describing the physical properties of the roadway (such as traffic speed, traffic volume, the number of vehicle and bicycle traffic lanes, lane width, lane traffic direction or the type and location of lane markings, or a combination thereof, etc.), and a map describing the spatial location of road features (such as crosswalks, traffic signs, or various types of other driving lights, etc.). In some embodiments, the map is generated in real time based on data received by the perception system.

[0066] In another example, positioning system 406 receives global navigation satellite system (GNSS) data generated by a global positioning system (GPS) receiver. In some examples, positioning system 406 receives GNSS data associated with a location of a vehicle in an area, and positioning system 406 determines the latitude and longitude of the vehicle in the area. In such an example, positioning system 406 determines the position of the vehicle in the area based on the latitude and longitude of the vehicle. In some embodiments, positioning system 406 generates data associated with the position of the vehicle. In some examples, based on the location of the vehicle determined by positioning system 406, positioning system 406 generates data associated with the position of the vehicle. In such an example, the data associated with the position of the vehicle include data associated with one or more semantic properties corresponding to the position of the vehicle.

[0067] In some embodiments, the control system 408 receives data associated with at least one trajectory from the planning system 404, and the control system 408 controls the operation of the vehicle. In some examples, the control system 408 receives data associated with at least one trajectory from the planning system 404, and the control system 408 controls the operation of the vehicle by generating and transmitting control signals to operate the powertrain control system (e.g., DBW system 202h and / or powertrain control system 204, etc.), the steering control system (e.g., steering control system 206) and / or the braking system (e.g., braking system 208). For example, the control system 408 is configured to perform operational functions such as lateral vehicle motion control or longitudinal vehicle motion control. Lateral vehicle motion control causes the activity required to adjust the y-axis component of the vehicle motion. Longitudinal vehicle motion control causes the activity required to adjust the x-axis component of the vehicle motion. In the example, in the case where the trajectory includes a left turn, the control system 408 transmits a control signal to cause the steering control system 206 to adjust the steering angle of the vehicle 200, thereby turning the vehicle 200 left. Additionally or alternatively, the control system 408 generates and transmits control signals to cause other devices of the vehicle 200 (eg, headlights, turn signals, door locks, and / or windshield wipers, etc.) to change states.

[0068] In some embodiments, perception system 402, planning system 404, positioning system 406, and / or control system 408 implement at least one machine learning model (e.g., at least one multi-layer perceptron (MLP), at least one convolutional neural network (CNN), at least one recurrent neural network (RNN), at least one autoencoder, and / or at least one transformer, etc.). In some examples, perception system 402, planning system 404, positioning system 406, and / or control system 408 implement at least one machine learning model alone or in combination with one or more of the above systems. In some examples, perception system 402, planning system 404, positioning system 406, and / or control system 408 implement at least one machine learning model as part of a pipeline (e.g., a pipeline for identifying one or more objects located in an environment, etc.). The following is about FIG. 4B to FIG. 4D Includes examples of implementations of machine learning models.

[0069] Database 410 stores data transmitted to, received from, and / or updated by perception system 402, planning system 404, positioning system 406, and / or control system 408. In some examples, database 410 includes a storage component (e.g., a storage component) for storing data and / or software related to operations and using at least one system of autonomous vehicle computing 400. Figure 3In some embodiments, database 410 stores data associated with a 2D and / or 3D map of at least one area. In some examples, database 410 stores data associated with a 2D and / or 3D map of a portion of a city, portions of multiple cities, multiple cities, counties, states, and / or countries (State) (e.g., a country), etc. In such an example, a vehicle (e.g., a vehicle that is the same or similar to vehicle 102 and / or vehicle 200) can be driven along one or more drivable areas (e.g., single-lane roads, multi-lane roads, highways, remote roads, and / or off-road roads, etc.) and cause at least one LiDAR sensor (e.g., a LiDAR sensor that is the same or similar to LiDAR sensor 202b) to generate data associated with an image representing an object included in the field of view of the at least one LiDAR sensor.

[0070] In some embodiments, database 410 can be implemented across multiple devices. In some examples, database 410 includes a vehicle (e.g., a vehicle that is the same or similar to vehicle 102 and / or vehicle 200), an autonomous vehicle system (e.g., an autonomous vehicle system that is the same or similar to remote AV system 114), a fleet management system (e.g., a vehicle that is the same or similar to remote AV system 114), and a fleet management system (e.g., a vehicle that is the same or similar to remote AV system 114). Figure 1 The same or similar queue management system as the queue management system 116 of FIG. 1 and / or the V2I system (e.g., Figure 1 The V2I system 118 is the same as or similar to the V2I system) and the like.

[0071] Reference now Figure 4B , a diagram illustrating an implementation of a machine learning model. More specifically, a diagram illustrating an implementation of a convolutional neural network (CNN) 420. For purposes of illustration, the following description of CNN 420 will be with respect to implementing CNN 420 by perception system 402. However, it will be understood that in some examples, CNN 420 (e.g., one or more components of CNN 420) is implemented by other systems (such as planning system 404, positioning system 406, and / or control system 408, etc.) other than or in addition to perception system 402. Although CNN 420 includes certain features as described herein, these features are provided for purposes of illustration and are not intended to limit the present disclosure.

[0072] CNN 420 includes a plurality of convolutional layers including a first convolutional layer 422, a second convolutional layer 424, and a convolutional layer 426. In some embodiments, CNN 420 includes a subsampling layer 428 (sometimes referred to as a pooling layer). In some embodiments, subsampling layer 428 and / or other subsampling layers have a dimension that is smaller than the dimension of the upstream system (i.e., the number of nodes). With the subsampling layer 428 having a dimension that is smaller than the dimension of the upstream layer, CNN 420 merges the amount of data associated with the initial input and / or output of the upstream layer, thereby reducing the amount of computation required for CNN 420 to perform downstream convolution operations. Additionally or alternatively, with the subsampling layer 428 being associated with (e.g., configured to perform) at least one subsampling function (as described below with respect to Figure 4C and Figure 4D As described above, CNN 420 incorporates the amount of data associated with the initial input.

[0073] The perception system 402 performs the convolution operation based on the perception system 402 providing respective inputs and / or outputs associated with each of the first convolution layer 422, the second convolution layer 424, and the convolution layer 426 to generate respective outputs. In some examples, the perception system 402 implements the CNN 420 based on the perception system 402 providing data as input to the first convolution layer 422, the second convolution layer 424, and the convolution layer 426. In such examples, the perception system 402 provides data as input to the first convolution layer 422, the second convolution layer 424, and the convolution layer 426 based on the perception system 402 receiving data from one or more different systems (e.g., one or more systems of a vehicle that is the same or similar to the vehicle 102, a remote AV system that is the same or similar to the remote AV system 114, a queue management system that is the same or similar to the queue management system 116, and / or a V2I system that is the same or similar to the V2I system 118, etc.). The following is about Figure 4C Includes a detailed description of the convolution operation.

[0074] In some embodiments, the perception system 402 provides data associated with the input (referred to as the initial input) to the first convolutional layer 422, and the perception system 402 generates data associated with the output using the first convolutional layer 422. In some embodiments, the perception system 402 provides the output generated by the convolutional layer as input to a different convolutional layer. For example, the perception system 402 provides the output of the first convolutional layer 422 as input to the subsampling layer 428, the second convolutional layer 424, and / or the convolutional layer 426. In such an example, the first convolutional layer 422 is referred to as an upstream layer, and the subsampling layer 428, the second convolutional layer 424, and / or the convolutional layer 426 are referred to as downstream layers. Similarly, in some embodiments, the perception system 402 provides the output of the subsampling layer 428 to the second convolutional layer 424 and / or the convolutional layer 426, and in this example, the subsampling layer 428 will be referred to as the upstream layer, and the second convolutional layer 424 and / or the convolutional layer 426 will be referred to as the downstream layer.

[0075] In some embodiments, before the perception system 402 provides the input to the CNN 420, the perception system 402 processes the data associated with the input provided to the CNN 420. For example, the perception system 402 processes the data associated with the input provided to the CNN 420 based on the perception system 402 normalizing the sensor data (e.g., image data, LiDAR data, and / or Radar data, etc.).

[0076] In some embodiments, CNN 420 generates an output based on perception system 402 performing convolution operations associated with each convolution layer. In some examples, CNN 420 generates an output based on perception system 402 performing convolution operations associated with each convolution layer and the initial input. In some embodiments, perception system 402 generates an output and provides the output to fully connected layer 430. In some examples, perception system 402 provides the output of convolution layer 426 to fully connected layer 430, wherein fully connected layer 430 includes data associated with multiple feature values ​​referred to as F1, F2, ..., FN. In this example, the output of convolution layer 426 includes data associated with multiple output feature values ​​representing predictions.

[0077] In some embodiments, perception system 402 identifies a prediction from the plurality of predictions based on perception system 402 identifying a feature value associated with a highest likelihood of being a correct prediction from the plurality of predictions. For example, where fully connected layer 430 includes feature values ​​F1, F2, ..., FN and F1 is the largest feature value, perception system 402 identifies the prediction associated with F1 as the correct prediction from the plurality of predictions. In some embodiments, perception system 402 trains CNN 420 to generate the predictions. In some examples, perception system 402 trains CNN 420 to generate the predictions based on perception system 402 providing training data associated with the predictions to CNN 420.

[0078] Reference now Figure 4C and Figure 4D , a diagram illustrating an example operation of CNN 440 utilizing perception system 402. In some embodiments, CNN 440 (e.g., one or more components of CNN 440) is coupled to CNN 420 (e.g., one or more components of CNN 420) (see Figure 4B ) are the same or similar.

[0079] At step 450, the perception system 402 provides data associated with the image as input to the CNN 440 (step 450). For example, as illustrated, the perception system 402 provides data associated with the image to the CNN 440, where the image is a grayscale image represented as values ​​stored in a two-dimensional (2D) array. In some embodiments, the data associated with the image may include data associated with a color image represented as values ​​stored in a three-dimensional (3D) array. Additionally or alternatively, the data associated with the image may include data associated with an infrared image and / or a Radar image, etc.

[0080] At step 455, CNN 440 performs a first convolution function. For example, CNN 440 performs a first convolution function based on CNN 440 providing a value representing an image as an input to one or more neurons (not explicitly illustrated) included in first convolution layer 442. In this example, the value representing the image may correspond to a value of a region (sometimes referred to as a receptive field) representing the image. In some embodiments, each neuron is associated with a filter (not explicitly illustrated). The filter (sometimes referred to as a kernel) may be represented as an array of values ​​corresponding in size to the value provided as input to the neuron. In one example, the filter may be configured to identify edges (e.g., horizontal lines, vertical lines, and / or straight lines, etc.). In successive convolution layers, the filters associated with the neurons may be configured to continuously identify more complex patterns (e.g., arcs and / or objects, etc.).

[0081] In some embodiments, CNN 440 performs a first convolution function based on CNN 440 multiplying the values ​​of each neuron provided as input to one or more neurons included in the first convolution layer 442 by the values ​​of the filters corresponding to each neuron in the same or more neurons. For example, CNN 440 may multiply the values ​​of each neuron provided as input to one or more neurons included in the first convolution layer 442 by the values ​​of the filters corresponding to each neuron in the one or more neurons to generate a single value or an array of values ​​as output. In some embodiments, the collective output of the neurons of the first convolution layer 442 is referred to as a convolution output. In some embodiments, when each neuron has the same filter, the convolution output is referred to as a feature map.

[0082] In some embodiments, CNN 440 provides the output of each neuron of the first convolutional layer 442 to the neurons of the downstream layer. For clarity, the upstream layer may be a layer that transmits data to a different layer (referred to as the downstream layer). For example, CNN 440 may provide the output of each neuron of the first convolutional layer 442 to the corresponding neurons of the subsampling layer. In the example, CNN 440 provides the output of each neuron of the first convolutional layer 442 to the corresponding neurons of the first subsampling layer 444. In some embodiments, CNN 440 adds a bias value to the aggregate set of all values ​​provided to each neuron of the downstream layer. For example, CNN 440 adds a bias value to the aggregate set of all values ​​provided to each neuron of the first subsampling layer 444. In such an example, CNN 440 determines the final value to be provided to each neuron of the first subsampling layer 444 based on the aggregate set of all values ​​provided to each neuron and the activation function associated with each neuron of the first subsampling layer 444.

[0083] At step 460, CNN 440 performs a first subsampling function. For example, based on CNN 440 providing the values ​​output by first convolutional layer 442 to the corresponding neurons of first subsampling layer 444, CNN 440 may perform the first subsampling function. In some embodiments, CNN 440 performs the first subsampling function based on an aggregation function. In an example, CNN 440 performs the first subsampling function based on CNN 440 determining the maximum input (referred to as a maximum pooling function) among the values ​​provided to a given neuron. In another example, CNN 440 performs the first subsampling function based on CNN 440 determining the average input (referred to as an average pooling function) among the values ​​provided to a given neuron. In some embodiments, based on CNN 440 providing values ​​to the respective neurons of first subsampling layer 444, CNN 440 generates an output, which is sometimes referred to as a subsampled convolution output.

[0084] At step 465, CNN 440 performs a second convolution function. In some embodiments, CNN 440 performs the second convolution function in a manner similar to how CNN 440 performs the first convolution function described above. In some embodiments, CNN 440 performs the second convolution function based on CNN 440 providing the value output by first subsampling layer 444 as input to one or more neurons (not explicitly illustrated) included in second convolution layer 446. In some embodiments, as described above, each neuron of second convolution layer 446 is associated with a filter. As described above, the filter (one or more) associated with second convolution layer 446 can be configured to recognize more complex patterns than the filter associated with first convolution layer 442.

[0085] In some embodiments, the CNN 440 performs a second convolution function based on the CNN 440 multiplying the value of each neuron provided as input to the one or more neurons included in the second convolution layer 446 by the value of the filter corresponding to each neuron of the one or more neurons. For example, the CNN 440 may multiply the value of each neuron provided as input to the one or more neurons included in the second convolution layer 446 by the value of the filter corresponding to each neuron of the one or more neurons to generate a single value or a value array as an output.

[0086] In some embodiments, the CNN 440 provides the output of each neuron of the second convolutional layer 446 to the neurons of the downstream layer. For example, the CNN 440 may provide the output of each neuron of the first convolutional layer 442 to the corresponding neurons of the subsampling layer. In an example, the CNN 440 provides the output of each neuron of the first convolutional layer 442 to the corresponding neurons of the second subsampling layer 448. In some embodiments, the CNN 440 adds a bias value to the aggregate set of all values ​​provided to each neuron of the downstream layer. For example, the CNN 440 adds a bias value to the aggregate set of all values ​​provided to each neuron of the second subsampling layer 448. In such an example, the CNN 440 determines the final value provided to each neuron of the second subsampling layer 448 based on the aggregate set of all values ​​provided to each neuron and the activation function associated with each neuron of the second subsampling layer 448.

[0087] At step 470, CNN 440 performs a second subsampling function. For example, based on CNN 440 providing the values ​​output by second convolutional layer 446 to corresponding neurons of second subsampling layer 448, CNN 440 may perform a second subsampling function. In some embodiments, based on CNN 440 using an aggregation function, CNN 440 performs a second subsampling function. In an example, as described above, based on CNN 440 determining the maximum input or average input among the values ​​provided to a given neuron, CNN 440 performs a first subsampling function. In some embodiments, based on CNN 440 providing values ​​to respective neurons of second subsampling layer 448, CNN 440 generates an output.

[0088] At step 475, CNN 440 provides the output of each neuron of the second subsampling layer 448 to the fully connected layer 449. For example, CNN 440 provides the output of each neuron of the second subsampling layer 448 to the fully connected layer 449 so that the fully connected layer 449 generates an output. In some embodiments, the fully connected layer 449 is configured to generate an output associated with a prediction (sometimes referred to as a classification). The prediction may include an indication that the objects included in the image provided as input to CNN 440 include objects and / or sets of objects, etc. In some embodiments, the perception system 402 performs one or more operations and / or provides data associated with the prediction to the various systems described herein.

[0089] Prediction Filtering

[0090] As described herein, planning system 404 may plan a path or trajectory for vehicle 200 based on objects in the vehicle's environment detected by perception system 402. Some of the objects may be able to move or act (also referred to herein as agents) independently of vehicle 200. For example, an agent may be another vehicle, a pedestrian, or a bicycle.

[0091] In some cases, the planning system 404 may use a prediction engine (not shown) to predict the path / trajectory of the agents in the scene. The prediction engine may generate a predicted path / trajectory for some or all agents in the scene. The predicted path / trajectory may enable the planning system 404 to better understand the scene and determine the possible path / trajectory of the vehicle 200. The prediction engine may be configured to predict the path / trajectory of the agent based on the type of agent and / or the movement of the agent. For example, the prediction engine may determine based on different types of agents that move differently (such as vehicles that may move faster than other agents; pedestrians that may be more unpredictable than other agents; and bicycles that may change their speed based on height changes, etc.). The prediction engine may also make determinations related to the agent based on the trajectories of the surrounding agents. For example, the prediction engine may make determinations about the path of the agent based on additional traffic entering the lane in front of the agent.

[0092] In some examples, the prediction engine may be implemented using one or more neural networks, such as CNN 440, etc. For example, one trained neural network may be configured to generate predictions for vehicles and a second trained neural network may be configured to generate predictions for pedestrians, etc., or one trained neural network may be configured to generate predictions for vehicles, pedestrians, and / or bicycles, etc.

[0093] Generating predictions for agents may be time and / or computationally intensive for the planning system 404. Furthermore, the prediction engine may generate predictions for agents that have a low probability of moving based on the context of the scene or for agents that should not move (e.g., a vehicle behind a traffic light). Furthermore, some predictions for specific agents may have a low probability of occurring (e.g., a person suddenly changing direction and jumping into the street).

[0094] Low probability or low quality predictions may use computing resources and increase the time it takes for the planning system to determine its path / trajectory. Low quality predictions may also cause the vehicle 200 to take unnecessary or unsafe actions. For example, a low quality prediction may indicate that a pedestrian may suddenly leave the sidewalk and enter the intersection to enter oncoming traffic. Such low quality predictions related to pedestrians may cause the vehicle 200 to quickly slow down and / or stop in the middle of the intersection.

[0095] The planning system 404 can reduce the time and computing resources spent on low-probability predictions by identifying / classifying primary agents and secondary agents based on the context of the scene and / or semantic data associated with the scene. The planning system 404 can request a prediction for the primary agent from the prediction engine. The prediction engine can further generate a prediction for the primary agent. The planning system 404 can process the predictions to identify primary predictions and secondary predictions, and use the primary predictions to generate one or more trajectories or paths. In addition, the planning system 404 can relay instructions to the control system 408 to cause the vehicle 200 to travel along a path determined based on the primary prediction.

[0096] Figure 5A and Figure 5B is an operational flow diagram illustrating an example filtering operation of the planning system 404. At block 550, the planning system 404 receives semantic image data associated with a semantic image of a scene. The semantic image data may be generated by the perception system 402.

[0097] A semantic image may include rows of pixels. Some or all of the pixels in a semantic image may include semantic data such as one or more feature embeddings. In some cases, feature embeddings may involve one or more object attributes such as, but not limited to, object classification or a class label of a classification of an identified object (sometimes referred to as a class of an object) (non-limiting examples: vehicles, pedestrians, bicycles, obstacles, traffic cones, drivable surfaces, or background, etc.). Object classification may also be referred to as pixel class probability or semantic segmentation score. In some cases, the object classification of pixels of an image may be used as a compact summary feature of the image. For example, an object classification may include a probability value indicating the probability of correctly predicting the identified object classification of a pixel.

[0098] In some cases, feature embeddings may include one or more than one n-dimensional feature vectors. In some such cases, individual feature vectors may not correspond to object attributes, but the combination of multiple n-dimensional feature vectors may contain information about the attributes of the object (such as, but not limited to, the object's classification, width, length, height, etc.). In some cases, feature embeddings may include one or more than one floating point number, which can assist downstream models in their detection / segmentation / prediction tasks.

[0099] Graphic 500a is a representation of at least a portion of the semantic image data received by planning system 404. In the illustrated example, the scene includes self-vehicle 502, which is an autonomous vehicle (as described herein with at least reference to FIG. 1 ) traveling through intersection 510 along with various objects identified. Figure 2200 ), wherein the various objects include vehicles 504a, 504b, 504c, 504d, 504e, 504f (individually or collectively referred to as (one or more) vehicles 504), pedestrians 506, and traffic lights 508a, 508b, 508c, 508d (individually or collectively referred to as (one or more) traffic lights 508). As described herein with reference to semantic images, the above-mentioned objects can be identified based on one or more feature embeddings in the semantic image that indicate the probability that a particular pixel forms part of a particular object and / or the identification is accurate.

[0100] At block 552, the self-vehicle 502 identifies the primary agent and the secondary agents. As described herein, agents may include objects that can move or act independently of the self-vehicle 502, such as other vehicles, pedestrians, and / or bicycles. In the example shown, the planning system 404 identifies the vehicle 504 and the pedestrian 506 as agents.

[0101] The planning system 404 can classify the identified agents as primary agents or secondary agents. In some cases, a primary agent may refer to an agent that may interact with the self-carrier 502 based on contextual data associated with the scene. A secondary agent may refer to an agent that is less likely to interact with the self-carrier 502 based on contextual data associated with the scene. For example, a secondary agent may be an agent behind a traffic light or traffic sign indicating that the agent should not move. A secondary agent may also be an agent behind the self-carrier 502, an agent that is too far away from the self-carrier 502 to interact with the self-carrier 502 within a relevant time period, or an agent that cannot interact with the self-carrier 502 due to mobility characteristics (e.g., a nearby stationary vehicle).

[0102] The planning system 404 can determine the primary agent and the secondary agent based on the semantic context, where the semantic context includes but is not limited to the distance or location relative to the self-vehicle 502, the position relative to the self-vehicle 502 and other objects in the scene (for example, on the other side of the cone or traffic light relative to the self-vehicle 502), the posture relative to the self-vehicle (facing or facing away from the self-vehicle 502 and / or the trajectory of the self-vehicle 502), the speed, etc.

[0103] In some cases, the planning system 404 determines the primary agent and / or the secondary agent based on the location data associated with the agent. For example, an agent with a distance greater than a threshold distance can be identified as a secondary agent, and an agent with a distance less than a threshold distance can be identified as a primary agent. The threshold distance can be based on the speed of the agent and / or the self-carrier 502. For example, the threshold distance in the case where the agent is not moving can be less than the threshold distance in the case where the agent is moving. Similarly, the threshold distance for an agent traveling at 50mph can be greater than the threshold distance for an agent traveling at 20mph.

[0104] In some cases, the planning system 404 can determine the primary agent and / or secondary agent based on the location of the agent relative to other objects and / or relative to the self-vehicle 502 (or the trajectory of the self-vehicle 502). For example, if there is another object (e.g., a pedestrian, a crosswalk, an obstacle, and / or a traffic signal such as a traffic sign or a traffic light) between the agent and the self-vehicle 502 or the trajectory of the self-vehicle 502, the planning system 404 can determine that the agent is a secondary agent. Conversely, if there is no object between the agent and the self-vehicle 502, the planning system 404 can determine that the agent is a primary agent. As another example, if the agent (e.g., another vehicle or a bicycle) is behind or to the side of the self-vehicle 502 (and traveling in the same direction), the planning system 404 can determine that the agent is a secondary agent. If the agent is in front of the self-vehicle 502 (and traveling in the same or opposite direction), the planning system 404 can determine that the agent is the primary agent.

[0105] In some cases, the planning system 404 can determine the primary agent and / or secondary agent based on the agent's posture relative to the self-carrier 502. For example, if the agent is facing away from the self-carrier 502 and / or the trajectory of the self-carrier 502, the planning system 404 can determine that the agent is a secondary agent. If the agent is facing the self-carrier 502 and / or the trajectory of the self-carrier 502, the planning system 404 can determine that the agent is a primary agent. In some cases, the planning system 404 can consider the type, state and / or posture of the relevant object when determining whether the agent is a primary agent or a secondary agent. For example, if the object is a traffic light type facing the agent and has a red state, the planning system 404 can identify the agent as a secondary agent, and if the traffic light has a green state, the planning system 404 can identify the agent as a primary agent.

[0106] It will be appreciated that any one or any combination of the above criteria may be used to identify an agent as a primary agent or a secondary agent. In some cases, the planning system 404 may first identify secondary agents within a vehicle scene, and then identify any remaining agents as primary agents (or vice versa).

[0107] As shown in figure 500b, in the illustrated example, self-carrier 502 identifies vehicles 504a, 504b, 504c and pedestrian 506 as primary agents, and identifies vehicles 504d, 504e, 504f as secondary agents (these secondary agents may be collectively referred to as a secondary agent set). As described herein, self-carrier 502 may identify 504d, 504f (e.g., vehicles 504d and 504f) as secondary agents based on the locations of 504d and 504f being behind (red) traffic lights 508c and 508d. Self-carrier 502 may identify vehicle 504e as a secondary agent based on the location of vehicle 504e being behind self-carrier 502. However, it will be appreciated that other contexts may be used to identify primary / secondary agents. In some cases, the ego vehicle 502 can identify secondary agents based on semantic context. In the example shown, the ego vehicle 502 determines that the remaining vehicles 504a, 504b, 504c (agents not identified as secondary agents) are primary agents.

[0108] refer to Figure 5B At block 556, planning system 404 determines a predicted trajectory for the primary agent. In some cases, planning system 404 does not perform a determination of a predicted trajectory for a secondary agent.

[0109] In some cases, planning system 404 determines the predicted trajectory by communicating with the prediction engine. For example, planning system 404 can request the prediction engine to return one or more predicted trajectories of the primary agent.

[0110] As described herein, the prediction engine can be implemented separately from or as part of planning system 404 and can generate one or more trajectories for each of the requested agents. Graph 500c depicts an example prediction made by the prediction engine based on data received from perception system 402.

[0111] In the illustrated example, planning system 404 determines two predictions related to vehicle 504a (e.g., vehicle 504a will go straight or turn toward ego vehicle 502), three predictions related to pedestrian 506 (e.g., pedestrian 506 will make a U-turn, turn right and exit the intersection, or turn left and enter the intersection), one prediction related to vehicle 504b (e.g., vehicle 504b will go straight and forward), two predictions related to vehicle 504c (e.g., vehicle 504c will go straight and forward, or straight and backward), and no predictions related to each of vehicles 504d-504f because these agents are identified as secondary agents. Figure 5B In 500c, the prediction engine makes seven predictions. It will be appreciated that the planning system 404 can determine fewer predictions or more predictions based on the desired number of predictions and the capabilities of the prediction engine being used (e.g., memory, processor speed, etc.).

[0112] At block 558, the planning system filters the predictions or actions. As described herein, the planning system 404 may determine multiple predictions or actions for a single agent. Some of these predictions may have a high probability of occurrence, while other predictions do not. In addition, predictions with a low probability of occurrence may cause the self-vehicle 502 to react in an unsafe manner. Therefore, the planning system 404 can identify the predictions as primary predictions (also referred to herein as primary actions) or secondary predictions (also referred to herein as secondary actions).

[0113] The planning system 404 can identify primary actions and secondary actions in various ways. In some cases, the primary action can be an action (or prediction) with a high probability of occurrence, and the secondary action can be an action with a low probability of occurrence. In some examples, the high probability and the low probability are determined relative to each other. Therefore, the primary action can be a specific number of actions that are more likely to occur than a set number of secondary actions. For example, the top two or three predictions for each agent can be identified as primary actions, and the remaining actions for the corresponding agent can be identified as secondary actions. In some cases, the probability of an action occurring can be based on the probability assigned to the action by the prediction engine or the planning system 404 and / or based on the ranking assigned to the action by the prediction engine and / or the planning system 404. For example, when the planning system receives a prediction from the prediction engine, the prediction engine can include the probability that a specific prediction (or action) will occur and / or include the ranking of different predictions for a specific agent. The planning system 404 can use probability and / or ranking to determine the primary action and secondary action.

[0114] In some examples, the planning system can determine the primary action and the secondary action based on the probability threshold. For example, the primary action can be an action with a probability of occurrence that meets the probability threshold. The secondary action can be an action with a probability of occurrence that does not meet the probability threshold. For example, the primary action can be an action with a probability of occurrence greater than 10%. The secondary action can be an action with a probability of occurrence less than or equal to 10%.

[0115] In some cases, the planning system can use one or more than one threshold value or standard to determine whether the action is a primary action or a secondary action. In some cases, the planning system 404 can use a change of direction threshold to determine whether the action is a primary action or a secondary action. For example, the action indicating that the agent may move backward and / or change the direction by more than a threshold amount (e.g., 30 degrees) can be identified as a secondary action. In some cases, the threshold turning degree can change based on the speed of the agent. Therefore, the planning system 404 can use the position, orientation and / or speed of the agent to determine whether the action is a primary action or a secondary action. For example, if the agent is stopped, the backward action can not be identified as a secondary action, and if the agent is moving forward, the backward action can be identified as a secondary action. Similarly, the higher the speed of the agent, the lower the threshold turning degree can be, so that two actions of different agents with the same turning degree can be classified as primary actions or secondary actions according to the speed of the corresponding agent.

[0116] In some cases, the planning system 404 may determine primary and secondary actions based on the type of agent. In some cases, the planning system 404 may classify primary / secondary actions for vehicles using criteria different from those for pedestrians or bicycles. For example, the planning system 404 may determine that for pedestrians, the action of moving backwards may be a primary action, but for bicycles or vehicles, any action of moving backwards is a secondary action. Similarly, the planning system 404 may determine that the action of moving sideways may be a primary action for pedestrians or bicycles, but a secondary action for vehicles. Similarly, if the planning system 404 uses a threshold turning degree to identify an action as a primary action or a secondary action, the planning system 404 may use different threshold turning degrees for pedestrians, bicycles, and / or vehicles.

[0117] In some cases, planning system 404 may determine primary and secondary actions based on the position and / or orientation of the agent relative to self-vehicle 502. For example, if an action indicates that the agent will change its direction to collide with vehicle 502 and / or interfere with the path of the vehicle, planning system 404 may identify the action as a secondary action. For example, if a pedestrian is identified as walking in the same direction as self-vehicle 502, an action indicating that the pedestrian is likely to suddenly change direction and enter the street may be identified as a secondary action. Similarly, an action indicating that vehicle 504a will change direction (turn) to collide with self-vehicle 502 may be identified as a secondary action.

[0118] In some cases, planning system 404 may ignore secondary actions and plan a path based on the primary actions of the primary agent. In some examples, planning system 404 may classify agents predicted to have secondary actions as secondary agents and ignore these agents entirely. In other examples, the planning system may consider the primary actions of certain primary agents and ignore the secondary actions of these primary agents.

[0119] Graph 500d depicts the five primary actions of the primary agent identified in the scene after filtering out the secondary actions. As shown in graph 500d, the number of predictions has been reduced relative to the number of predictions shown in graph 500c. In the illustrated example of 500d, planning system 404 identifies five actions as primary actions: one primary action (going straight) for vehicle 504a, two primary actions (U-turn and right turn) for pedestrian 506, and one primary action (going straight) for each of vehicles 504b and 504c. It will be understood that planning system 404 may have fewer predictions or more predictions based on semantic data.

[0120] At block 560, planning system 404 generates a path for self-propelled vehicle 502. As described herein, planning system 404 can generate a path based on the primary actions of the primary agent. In some cases, planning system 404 can use one or more neural networks to generate a path through a scene based on the primary actions of the primary agent.

[0121] Figure 6 is a flow chart illustrating an example of a routine 600 implemented by the planning system 404 for controlling a vehicle based on a path generated using a primary agent. Figure 6 The flowchart shown is provided for illustration purposes only. It will be understood that the Figure 6, or the order of the steps may be changed. In addition, to illustrate a clear example, one or more specific system components (e.g., planning system 404) are described in the context of performing various operations during various data flow stages. However, other system configurations and distribution of processing steps across system components may be used.

[0122] At block 602, planning system 404 receives / obtains semantic image data from perception system 402. In some examples, planning system 404 receives data from perception system 402, where the data has been associated and classified at perception system 402 with at least one physical object in the environment surrounding vehicle 200. For example, Figure 5A and Figure 5B As shown, planning system 404 receives semantic image data associated with a semantic image of a scene from perception system 402, wherein perception system 402 generates the semantic image from sensor data associated with at least one sensor (such as camera 202a, LiDAR sensor 202b, Radar sensor 202c, etc.). Figure 5A and Figure 5B As described, the semantic image data may include data associated with at least one vehicle, pedestrian, road feature, and / or traffic signal, among others.

[0123] At block 604, planning system 404 identifies agents in the environment based on the semantic image data received from planning system 404. For example, as described herein, the semantic image data may indicate different types of objects within the semantic image. In some cases, planning system 404 may identify agents based on the type of corresponding object identified. For example, planning system 404 may identify objects that can move independently of a vehicle (such as other vehicles, pedestrians, and / or bicycles, etc.) as agents.

[0124] At block 606, planning system 404 classifies primary agents and secondary agents. As described herein, planning system 404 can identify agents based on one or more criteria, such as, but not limited to, distance or location relative to the vehicle, position relative to the vehicle and other objects in the scene (e.g., on the other side of a cone or traffic light from ego vehicle 502), pose relative to the ego vehicle (facing or facing away from the vehicle and / or the trajectory of the vehicle), velocity of the agent, and the like.

[0125] For example, the criteria may be a determination of whether an agent is likely to physically interact with vehicle 200 based on the agent's location, trajectory, and mobility at a given time. If the agent is likely to interact with vehicle 200, planning system 404 may identify the agent as a primary agent. Thus, the set of primary agents may be a subset of the agents identified in the semantic image.

[0126] Similar to the determination of the primary agent, the processor 304 may identify the secondary agent based on a set of criteria. For example, if an agent cannot interact with the vehicle (e.g., within a certain time period), the planning system 404 may identify the agent as a secondary agent. Thus, the set of secondary agents may be a subset of the agents identified in the semantic image. In some cases, the set of secondary agents may be mutually exclusive with the set of primary agents.

[0127] At block 608, the planning system 404 generates a path for the vehicle based on the main agent. In some cases, the planning system 404 generates one or more trajectories, predictions, and / or actions for the main agent indicating where the main agent may move, and generates a path based on the generated predictions or actions. The generated path may include maintaining a specific heading and rate, decelerating, accelerating, turning left or right, etc. In some cases, the planning system may generate a path within a specific time period (such as but not limited to three seconds or six seconds, etc.). For example, if the prediction generated for the main agent is proven to be accurate (or occurs), the planning system 404 can plan the expected path for six consecutive seconds of the vehicle. If the generated prediction is proven to be inaccurate or additional information is obtained, the planning system 404 can generate an alternative path. In addition, it will be understood that the planning system 404 can generate multiple paths multiple times within one second, and select different paths according to the dynamically changing environment (the vehicle operates in this environment).

[0128] At block 610, planning system 404 transmits instructions to control at least one of autonomous system 202, powertrain control system 204, steering control system 206, and braking system 208 to cause vehicle 200 to follow the path. For example, planning system 404 may send instructions to powertrain control system 204 to accelerate vehicle 200 and avoid collision with the agent. Planning system 404 may send instructions to steering control system 206 to cause vehicle 200 to navigate around the agent. Planning system 404 may also send instructions to braking system 208 to stop vehicle 200 before colliding with the agent.

[0129] It will be understood that fewer, more, or different blocks may be used in routine 600. In addition, one or more blocks may be rearranged, or performed simultaneously or in parallel. In some cases, one or more blocks in the routine may be repeated multiple times.

[0130] Figure 7 is a flow chart illustrating an example of a routine 700 implemented by the planning system 404 for controlling a vehicle based on a path generated using a primary agent and primary actions. Figure 7 The flowchart shown is provided for illustration purposes only. It will be understood that the Figure 6 , or the order of the steps may be changed. In addition, to illustrate a clear example, one or more specific system components (e.g., planning system 404) are described in the context of performing various operations during various data flow stages. However, other system configurations and distribution of processing steps across system components may be used.

[0131] At block 702, as described herein at least with respect to Figure 6 As depicted, planning system 404 receives / obtains semantic image data from perception system 402 .

[0132] At block 704, as described herein at least with respect to Figure 6 As described, planning system 404 identifies agents in the environment based on semantic image data obtained by perception system 402.

[0133] At block 706, the planning system 404 obtains / receives predictions for agents based on the environment surrounding the vehicle 200 and based on agents identified in the environment. In some cases, the prediction may be a prediction for an agent identified as a primary agent (rather than an agent identified as a secondary agent). In some examples, the prediction may be made by a prediction engine based on the trajectory and type of the agent identified at block 704. In some examples, the prediction may be made within the planning system 404 as described above. In some examples, the prediction may be made by a processor or a trained neural network that is remote from the planning system 404, within the vehicle 200, or remote from the vehicle 200. In some cases, the planning system 404 may obtain / receive multiple predictions for a specific (primary) agent, multiple (primary) agents, or each (primary) agent.

[0134] At block 708, the planning system 404 categorizes the primary actions and the secondary actions. Figure 5A and Figure 5BAs described, planning system 404 may classify, determine, and / or identify primary and secondary actions in various ways. In some cases, the planning system may classify actions based on any one or any combination of the type of agent, the position and / or orientation of the agent relative to planning system 404, the speed of the agent, the probability that the action will occur, the probability that the action will occur relative to other actions, and the like.

[0135] In some cases, a primary action may be an action (or prediction) with a high likelihood of occurring, and a secondary action may be an action with a low likelihood of occurring. In some examples, the high likelihood and the low likelihood are determined relative to each other. Thus, a primary action may be a certain number of actions that are more likely to occur than a set number of secondary actions. For example, the top two or three predictions for an agent may be identified as primary actions, and the remaining actions for the corresponding agent may be identified as secondary actions.

[0136] In some cases, the likelihood of an action occurring can be based on the probability assigned to the action by the prediction engine or planning system 404 and / or based on the ranking assigned to the action by the prediction engine and / or planning system 404. For example, when the planning system receives a prediction from the prediction engine, the prediction engine can include the probability that a particular prediction (or action) will occur and / or include a ranking of different predictions for a particular agent. The planning system 404 can use the probabilities and / or rankings to determine primary and secondary actions.

[0137] In some examples, planning system 404 can determine primary actions and secondary actions based on probability thresholds. For example, a primary action can be an action with a probability of occurrence that meets a probability threshold. A secondary action can be an action with a probability of occurrence that does not meet the probability threshold. For example, a primary action can be an action with a probability of occurrence greater than 10%. A secondary action can be an action with a probability of occurrence less than or equal to 10%.

[0138] Thus, semantic data and semantic behavioral data related to the vehicle scenario can be used to remove low-quality predictions from consideration during path planning. In some examples, the planning system 404 can transmit actions identified as secondary actions to a processing engine or neural network to inform future predictions. Thus, data generated based on the classification or determination of primary and secondary actions can be used to intervene in subsequent determinations.

[0139] At block 710, the planning system 404 generates a path for the vehicle 200 based on the primary action. In some cases, the planning system 404 uses the primary action to predict where the corresponding agent will move within the vehicle scene within a time period. Based on the movement of various agents, the planning system can generate a path for the vehicle 200 to pass through the vehicle scene. In some cases, the planning system 404 generates a path corresponding to one or more path planning strategies or a path that meets one or more thresholds. For example, the planning system 404 can generate a path that meets a comfort threshold (e.g., no acceleration / deceleration greater than a threshold amount, and / or no path that causes a centripetal force greater than a threshold amount) and / or a safety threshold (e.g., no collision, no leaving the road or drivable area, etc.). In some cases, the generated path can include maintaining a specific heading and speed, decelerating, accelerating, turning left or right, etc. In some cases, the planning system can generate a path within a specific time period (such as, but not limited to, three seconds or six seconds, etc.). For example, if the (primary) agent's primary action proves to be accurate (or occurs), then planning system 404 may plan the vehicle's expected path for six consecutive seconds. If the generated prediction proves to be inaccurate or additional information is obtained, then planning system 404 may generate an alternative path. Furthermore, it will be appreciated that planning system 404 may generate multiple paths multiple times within a second, and select different paths depending on the dynamically changing environment in which vehicle 200 is operating.

[0140] In some cases, the path is a planned path that avoids collisions with agents in the vehicle scene and operates based on a desired set of priorities. For example, a path that minimizes the amount of travel time or a path that minimizes safety risks can be planned for vehicle 200. For example, a path can be generated based at least in part on information about agents that are predicted to move into the path of vehicle 200.

[0141] At block 712, as described herein at least with respect to Figure 6 As depicted, planning system 404 transmits instructions to control at least one of autonomous system 202 , powertrain control system 204 , steering control system 206 , and braking system 208 to cause vehicle 200 to follow a path.

[0142] It will be understood that fewer, more, or different blocks may be used in routine 700. In addition, one or more blocks may be rearranged, or performed simultaneously or in parallel. In some cases, one or more blocks in the routine may be repeated multiple times.

[0143] Major collisions

[0144] As described herein, the perception system 402 can detect various objects or obstacles in the vehicle scene. In some scenarios, the planning system 404 can predict a collision between the vehicle 200 and the object. In response, the vehicle 200 can simulate one or more actions that the vehicle 200 can take to avoid the collision. In some cases, the planning system 404 can determine that the predicted collision is a major collision. In some cases, if the planning system 404 cannot determine a path that will avoid the predicted collision, the planning system 404 can determine that the predicted collision is a major collision. For example, the perception system 402 can detect other vehicles entering the road from an invisible lane, or animals or pedestrians that unexpectedly rush into the street directly in front of the vehicle 200. In response, the planning system can simulate one or more actions that the vehicle can take to avoid colliding with vehicles, animals or pedestrians that unexpectedly enter the path of the vehicle 200. However, in some cases, planning system 404 may determine that vehicle 200 cannot brake, accelerate, or steer quickly enough to avoid a collision, given the momentum, braking capabilities, and certain driving characteristics of vehicle 200. Therefore, planning system 404 may identify the collision as a major collision and take appropriate action.

[0145] Based on the determination that the collision is a major collision and / or unavoidable, the planning system 404 can simulate one or more actions that the vehicle 200 can take to reduce damage to the vehicle 200 or the object and / or improve safety. Based on the simulated actions, the planning system 404 can cause the vehicle 200 to take actions corresponding to the collision mitigation strategy.

[0146] Figure 8 An example graph 800 is shown illustrating the position (y-axis) of a vehicle 802 and an obstacle 804 over time (x-axis). In the example shown, the graph 800 shows the simulated position of the vehicle 802 over time while performing two collision avoidance actions to avoid the obstacle 804, however it will be appreciated that fewer or more collision avoidance actions may be simulated and / or taken. In some examples, the obstacle 804 may be a vehicle, a pedestrian, a bicycle, or other obstacle that the vehicle may collide with during operation.

[0147] Line 806 illustrates a simulation of the position of vehicle 802 over time if vehicle 802 attempts to travel behind obstacle 804. Traveling behind obstacle 804 can include at least one of maintaining the speed of vehicle 802, decelerating, or turning. Line 808 illustrates a simulation of the position of vehicle 802 over time if vehicle 802 attempts to travel in front of obstacle 804. Traveling in front of obstacle 804 can include maintaining the speed, accelerating, or turning. In the example shown, neither attempting to travel in front of obstacle 804 nor attempting to travel behind obstacle 804 will cause vehicle 802 to avoid colliding with obstacle 804. Therefore, planning system 404 can identify the predicted collision between vehicle 802 and obstacle 804 as a major collision.

[0148] Based on the result of identifying the collision as a major collision, the planning system 404 can analyze different scenarios of collision with the obstacle 804. For example, the planning system 404 can determine the predicted location of the collision impact on the vehicle 200 (e.g., near the front, middle, or rear of the vehicle 200), and calculate the rate and force of the collision and additional factors such as the predicted collision angle. Using the impact location, rate, and / or force, the planning system 404 can determine the loss prediction, injury prediction, or other related prediction that the collision at different locations on the vehicle 200 will result in.

[0149] Based on the scenario of collision with obstacle 804, planning system 404 can select the action to be performed based on collision mitigation strategy. Collision mitigation strategy can indicate how to evaluate / prioritize the possibility of collision. For example, collision mitigation strategy can indicate planning system 404 to prioritize minimum rate collision (e.g., reduce rate as much as possible), hit the obstacle with the front or rear of the car (e.g., accelerate or decelerate to collide with the object at a specific position on vehicle 200), or hit an unoccupied vehicle or obstacle instead of an occupied vehicle or pedestrian. Planning system 404 can consider collision mitigation strategy and select collision mitigation action to be performed based on loss prediction, injury prediction or other relevant prediction. For example, if the collision mitigation strategy is to hit the object with the front of vehicle 200, the planning system can take the relevant action of hitting the object with the front of vehicle 200 instead of the side or rear of vehicle 200. In some examples, collision mitigation actions can include but are not limited to deceleration, acceleration, stopping, maintaining rate, turning, or various other actions that can affect the path / trajectory of the vehicle and the nature of the collision.

[0150] In some examples, planning system 404 may use the position graph over time to simulate scenarios where vehicle 200 attempts to avoid more than one obstacle. The simulation may determine that a collision with one obstacle is unavoidable in order to avoid a collision with another obstacle. For example, vehicle 200 may determine that vehicle 200 should slow down to travel behind an upcoming obstacle, but slowing down will cause an agent traveling closely behind vehicle 200 to collide with vehicle 200. Therefore, planning system 404 may analyze different simulation scenarios based on the above-mentioned alternative paths. Planning system 404 may select actions to be performed based on scenarios and collision mitigation strategies. In the case of considering multiple collision options, the collision mitigation strategy may instruct planning system 404 to determine actions based on collisions with the least consequences. For example, based on similar speeds of agents and vehicle 200, a collision with an agent behind vehicle 200 may have a lower energy impact. This lower energy impact may be desirable compared to the higher energy impact of an oncoming vehicle. In another example, planning system 404 may determine an action based at least in part on local regulations, which may favor occupants of a vehicle that receives a collision from the rear over occupants of a vehicle that receives a collision from the front.

[0151] Fig. 9 is a flow chart illustrating an example of a routine 900 implemented by the planning system 404 for controlling a vehicle based on a path generated using a primary agent. Fig. 9 The flowchart shown is provided for illustration purposes only. It will be understood that the Fig. 9 , or the order of the steps may be changed. In addition, to illustrate a clear example, one or more specific system components (e.g., planning system 404) are described in the context of performing various operations during various data flow stages. However, other system configurations and distribution of processing steps across system components may be used.

[0152] At block 902, as described herein at least with respect to Figure 6 As depicted, planning system 404 obtains / receives data from perception system 402 .

[0153] At block 904 , the planning system 404 identifies agents in the environment based on the semantic image data received from the perception system 402 , as described herein.

[0154] At block 906, planning system 404 determines at least one predicted action for an agent in the environment. As described herein, in some cases, at least one predicted action may be based on a possible trajectory or path of the agent. The predicted action may also be based at least in part on semantic behavior data. As described above, semantic behavior data may be based on considerations such as predicting how the agent will behave, safety thresholds, and regulatory considerations. In some cases, planning system 404 may determine one or more predicted actions for an agent identified as a primary agent. In some cases, as described herein, planning system 404 may determine a primary action from a set of predicted actions.

[0155] At block 908, the planning system 404 determines a predicted path for the agent based on the predicted actions of the agent. In some cases, the planning system 404 determines the predicted path for the agent based on one or more primary actions. For example, the planning system can identify a primary action from a set of generated actions and use one of the primary actions to generate a predicted path for the agent. Figure 8 As described, the planning system 404 can plot the predicted path of the agent over time in the position graph over time. For example, the planning system 404 can plot the predicted path of the agent in the environment based on the surrounding environment, other agents, semantic data, and semantic behavior data mapping.

[0156] At block 910, the planning system 404 determines a path for the vehicle 200. As described herein, the path for the vehicle 200 can be a planned path based on a set of desired system priorities. For example, as described above, a path can be planned to minimize travel distance or a path to maximize safety of the vehicle 200. In this example, the planning system 404 determines a path that is intended to avoid collisions.

[0157] At block 912, planning system 404 predicts that a collision will occur based on the predicted path of the agent and the determined path of vehicle 200. For example, planning system 404 may determine that the agent is entering the determined path of the vehicle from a side road at a speed that will place the agent's position at the same place as vehicle 200 at a given time. In another example, planning system 404 may determine that a pedestrian has entered the determined path of vehicle 200 at a speed that will place the pedestrian's position at the same place as vehicle 200 at a given time.

[0158] At block 914, the planning system 404 simulates the avoidance maneuvers of the vehicle 200 to avoid collision with the obstacle. For example, the planning system 404 may simulate the avoidance maneuvers of the vehicle 200 to avoid collision with the obstacle. Figure 8The planning system 404 may also plot a predicted path of the vehicle 200 over time based at least in part on the semantic data, including potential paths for evasive actions of the vehicle 200 in the environment. In some cases, the evasive actions may include, but are not limited to, accelerating or decelerating, turning left or right, etc.

[0159] At block 916, the planning system 404 classifies the collision as a major collision. Figure 8 As described, in a major collision scenario, the planning system 404 determines that despite the avoidance action, the vehicle 200 is likely to collide. For example, the planning system 404 may determine that a collision is likely to occur regardless of whether the vehicle is accelerating or decelerating, turning left or right, etc. Therefore, the planning system 404 determines that the collision is a major collision. In some examples, a major collision may correspond to an unavoidable collision due to physical constraints (e.g., based on the speed of the vehicle and the agent, etc.). In some examples, a major collision is a collision that, if avoided, would increase the risk of harming the passengers of the vehicle 200. In some examples, a major collision is a collision that would result in a more severe or less desirable collision than an alternative collision (e.g., rear-end, hitting a pedestrian, a cyclist, etc.).

[0160] At block 918, the planning system 404 transmits vehicle control instructions based on the primary collision determination and path planning. In some cases, the planning system can control the vehicle according to the collision mitigation strategy. As described herein, the collision mitigation strategy can indicate the priority of the collision of the vehicle. In some cases, the collision mitigation strategy can indicate the priority position, priority speed or priority relative speed of the vehicle collision, etc. For example, the collision mitigation strategy can indicate that the vehicle should be positioned so that the collision occurs at the front, middle or rear of the vehicle. As another example, the collision mitigation strategy can indicate that the vehicle should have a low speed or a low speed relative to the speed of the intelligent body it will collide with, etc. Therefore, based on the collision mitigation strategy, the planning system 404 can cause the vehicle to accelerate, decelerate, turn left / right, etc. For example, the planning system 404 can draw a path for the intelligent body to collide with the vehicle 200 at a specific point on the vehicle 200, which can minimize the results of the collision, etc.

[0161] As this article at least Figure 6 As depicted, the instructions cause one of autonomous system 202 , powertrain control system 204 , steering control system 206 , and braking system 208 to cause vehicle 200 to follow the path.

[0162] It will be understood that fewer, more, or different blocks may be used in routine 900. In addition, one or more blocks may be rearranged, or performed simultaneously or in parallel. In some cases, one or more blocks in the routine may be repeated multiple times.

[0163] FIG. 5A to FIG. 7 and Fig. 9 The flowchart shown is provided for illustration purposes only. It will be understood that the FIG. 5A to FIG. 7 and Fig. 9 One or more steps in the routines shown may be used, or the order of the steps may be changed. In addition, it will be understood that the steps from FIG. 5A to FIG. 7 and Fig. 9 For example, planning system 404 may obtain semantic image data (602), identify agents in the environment (604), classify primary agents / secondary agents (606), receive predictions for the agents (706), classify primary actions / secondary actions (708), generate a vehicle path (608 / 710), and control a vehicle based on the vehicle path (610 / 712). Furthermore, although described in some cases in terms of generating a path based on an agent, it will be understood that planning system 404 may perform similar steps for multiple agents in the environment and may repeatedly generate vehicle paths based on the dynamic nature of the environment.

[0164] Furthermore, to illustrate clear examples, one or more specific system components are described in the context of performing various operations during various data flow stages. However, other system configurations and distribution of processing steps across system components may be used.

[0165] In some embodiments, one or more steps described with respect to processes 500, 600, 700, and 900 are performed by the planning system 404 or other system of the vehicle 200 (e.g., completely and / or partially, etc.). Additionally or alternatively, in some embodiments, one or more steps described with respect to processes 500, 600, 700, and 900 are performed by other devices or groups of devices that are separate from or include the planning system 404 or other system of the vehicle 200 (such as a processor that is remote from the planning system 404 or other system of the vehicle 200, etc.) (e.g., completely and / or partially, etc.).

[0166] In the previous description, aspects and embodiments of the present disclosure have been described with reference to many specific details, which may vary depending on the implementation. Therefore, the description and the accompanying drawings should be regarded as illustrative, not restrictive. The only and exclusive indication of the scope of the invention, and the applicant's expectation that the content of the scope of the invention is the literal and equivalent scope of the claims issued from this application in the specific form of the claims, including any subsequent amendments. Any definition of the terms used to be included in such claims that are clearly set forth herein should be based on the meaning of such terms as used in the claims. In addition, when the term "also includes" is used in the previous description or the attached claims, the following of the phrase may be an additional step or entity, or a sub-step / sub-entity of the previously described step or entity.

[0167] Example

[0168] Various example embodiments of the present disclosure may be described by the following terms:

[0169] Item 1. A method for operating an autonomous vehicle, the method comprising: obtaining, by at least one processor, semantic image data associated with an environment in which the autonomous vehicle is operating; determining, by the at least one processor, a set of agents in the environment based on the semantic image data; determining a set of predicted actions for at least one primary agent in the set of agents; determining, based on the set of predicted actions, a set of secondary predicted actions for the at least one primary agent, wherein the set of secondary predicted actions is determined for the at least one primary agent based on a location of the at least one primary agent and based on agent semantic behavior data associated with the at least one primary agent, wherein the agent semantic behavior data includes logic-based rules and exceptions for the predicted agent actions; determining, based on the set of predicted actions, a set of primary predicted actions excluding secondary predicted actions; and generating a path for the autonomous vehicle based on the set of primary predicted actions.

[0170] Clause 2. The method of clause 1, further comprising operating the autonomous vehicle along a path of the autonomous vehicle.

[0171] Clause 3. A method according to clause 1 or 2, wherein the collection of agents includes objects configured to move.

[0172] Clause 4. A method according to any one of clauses 1-3, wherein the set of agents also includes a set of secondary agents, wherein the set of secondary agents includes agents that are unlikely to interact with the autonomous vehicle.

[0173] Clause 5. A method according to clause 4, wherein the set of secondary agents includes agents that will inevitably interact with the autonomous vehicle.

[0174] Clause 6. The method according to clauses 4-5 also includes transmitting the set of secondary agents to a processing engine.

[0175] Clause 7. The method of any of clauses 4-6, further comprising determining a second set of interaction parameters based at least in part on the set of secondary agents.

[0176] Clause 8. A method according to any one of clauses 1-7, wherein the at least one primary agent is a pedestrian on a sidewalk.

[0177] Clause 9. The method of clause 8, wherein the pedestrian is following a predetermined path.

[0178] Clause 10. The method of clause 9, wherein the pedestrian is determined to be approaching an intersection between the pedestrian's predetermined path and the path of the autonomous vehicle.

[0179] Clause 11. The method of clause 10, wherein the pedestrian is determined to have collided with the autonomous vehicle.

[0180] Clause 12. The method of any one of clauses 1-11, wherein generating a path for the autonomous vehicle further comprises determining a plurality of alternative paths for the autonomous vehicle.

[0181] Clause 13. The method of clause 12, wherein the alternate paths include an increased speed during at least a portion of at least one of the alternate paths.

[0182] Clause 14. The method of clause 12 or 13, wherein the alternative path comprises a reduced speed during at least a portion of at least one of the alternative paths.

[0183] Item 15. A system comprising: at least one processor; and at least one non-volatile storage medium storing instructions that, when executed by the at least one processor, cause the at least one processor to: obtain, by the at least one processor, semantic image data associated with an environment in which an autonomous vehicle is operating; determine, by the at least one processor, a set of agents in the environment based on the semantic image data; determine a set of predicted actions for at least one primary agent in the set of agents; determine, based on the set of predicted actions, a set of secondary predicted actions for the at least one primary agent, wherein the set of secondary predicted actions is determined for the at least one primary agent based on a location of the at least one primary agent and based on agent semantic behavior data associated with the at least one primary agent, wherein the agent semantic behavior data includes logic-based rules and exceptions for the predicted agent actions; determine, based on the set of predicted actions, a set of primary predicted actions other than secondary predicted actions; and generate a path for the autonomous vehicle based on the set of primary predicted actions.

[0184] Clause 16. The system of clause 15, further comprising operating the autonomous vehicle along a path of the autonomous vehicle.

[0185] Clause 17. A system according to clause 15 or 16, wherein the set of agents also includes a set of secondary agents, and wherein the set of secondary agents includes agents that are unlikely to interact with the autonomous vehicle.

[0186] Item 18. At least one non-volatile storage medium storing instructions that, when executed by the at least one processor, cause the at least one processor to: obtain, by the at least one processor, semantic image data associated with an environment in which an autonomous vehicle is operating; determine, by the at least one processor, a set of agents in the environment based on the semantic image data; determine a set of predicted actions for at least one primary agent in the set of agents; determine, based on the set of predicted actions, a set of secondary predicted actions for the at least one primary agent, wherein the set of secondary predicted actions is determined for the at least one primary agent based on a location of the at least one primary agent and based on agent semantic behavior data associated with the at least one primary agent, wherein the agent semantic behavior data includes logic-based rules and exceptions for the predicted agent actions; determine, based on the set of predicted actions, a set of primary predicted actions other than secondary predicted actions; and generate a path for the autonomous vehicle based on the set of primary predicted actions.

[0187] Clause 19. The at least one non-volatile storage medium of Clause 18, further comprising causing the autonomous vehicle to operate along a path of the autonomous vehicle.

[0188] Clause 20. At least one non-volatile storage medium according to clause 18 or 19, wherein the set of agents further comprises a set of secondary agents, and wherein the set of secondary agents comprises agents that are less likely to interact with the autonomous vehicle.

[0189] Item 21. A method for operating an autonomous vehicle, the method comprising: obtaining, by at least one processor, semantic image data associated with an environment in which the autonomous vehicle is operating; determining, by the at least one processor, a set of agents in the environment based on the semantic image data; determining, by the at least one processor, a set of secondary agents from the set of agents based on the relative locations of the corresponding secondary agents to the corresponding objects and a set of object semantic behavior data associated with the corresponding objects, wherein the object semantic behavior data includes logic-based rules and exceptions for safe autonomous operation; determining a set of primary agents excluding secondary agents based on the set of agents; and generating a path for the autonomous vehicle based on the set of primary agents.

[0190] Clause 22. The method of clause 21, further comprising operating the autonomous vehicle along a path of the autonomous vehicle.

[0191] Clause 23. The method of clause 21 or 22, wherein the collection of agents includes objects configured to move.

[0192] Clause 24. A method according to any of clauses 21-23, wherein the set of secondary agents includes agents that are unlikely to interact with the autonomous vehicle.

[0193] Clause 25. A method according to any of clauses 21-24, wherein the set of secondary agents includes agents that will inevitably interact with the autonomous vehicle.

[0194] Clause 26. The method of any of clauses 21-25, wherein the set of secondary agents includes agents disposed on an opposite side of a traffic signal from the autonomous vehicle.

[0195] Clause 27. A method according to any of clauses 21-26, wherein the agents in the set of secondary agents are vehicles that are approaching a traffic signal relatively close to the autonomous vehicle.

[0196] Item 28. A system comprising: at least one processor, and at least one non-volatile storage medium storing instructions that, when executed by the at least one processor, cause the at least one processor to: obtain, through the at least one processor, semantic image data associated with an environment in which an autonomous vehicle is operating; determine, through the at least one processor, a set of agents in the environment based on the semantic image data; determine, through the at least one processor, a set of secondary agents based on the set of agents based on the relative locations of the corresponding secondary agents to the corresponding objects and a set of object semantic behavior data associated with the corresponding objects, wherein the object semantic behavior data includes logic-based rules and exceptions for safe autonomous operations; determine, based on the set of agents, a set of primary agents excluding secondary agents; and generate a path for the autonomous vehicle based on the set of primary agents.

[0197] Clause 29. The system of clause 28, further comprising operating the autonomous vehicle along a path of the autonomous vehicle.

[0198] Clause 30. A system according to clause 28 or 29, wherein the collection of agents includes objects configured to move.

[0199] Clause 31. A system according to any of clauses 28-30, wherein the set of secondary agents includes agents that are unlikely to interact with the autonomous vehicle.

[0200] Clause 32. A system according to any of clauses 28-31, wherein the set of secondary agents includes agents that will inevitably interact with the autonomous vehicle.

[0201] Clause 33. The system of any of clauses 28-32, wherein the set of secondary agents includes agents disposed on an opposite side of a traffic signal from the autonomous vehicle.

[0202] Clause 34. A system according to any of clauses 28-33, wherein the agents in the set of secondary agents are vehicles that are approaching a traffic signal relatively close to the autonomous vehicle.

[0203] Item 35. At least one non-volatile storage medium storing instructions that, when executed by at least one processor, cause the at least one processor to: obtain, by the at least one processor, semantic image data associated with an environment in which an autonomous vehicle is operating; determine, by the at least one processor, a set of agents in the environment based on the semantic image data; determine, by the at least one processor, a set of secondary agents based on the set of agents based on the relative locations of the corresponding secondary agents to the corresponding objects and a set of object semantic behavior data associated with the corresponding objects, wherein the object semantic behavior data includes logic-based rules and exceptions for safe autonomous operation; determine, by the at least one processor, a set of primary agents excluding secondary agents based on the set of agents; and generate a path for the autonomous vehicle based on the set of primary agents.

[0204] Clause 36. The at least one non-volatile storage medium of Clause 35, further comprising causing the autonomous vehicle to operate along a path of the autonomous vehicle.

[0205] Clause 37. At least one non-volatile storage medium according to clause 35 or 36, wherein the collection of agents includes objects configured to move.

[0206] Clause 38. At least one non-volatile storage medium as recited in any one of clauses 35-37, wherein the set of secondary agents comprises agents that are unlikely to interact with the autonomous vehicle.

[0207] Clause 39. At least one non-volatile storage medium according to any one of clauses 35-38, wherein the set of secondary agents includes agents that will inevitably interact with the autonomous vehicle.

[0208] Clause 40. At least one non-volatile storage medium as recited in any one of Clauses 35-39, wherein the set of secondary agents includes agents disposed on an opposite side of a traffic signal from the autonomous vehicle.

[0209] Item 41. A method for operating an autonomous vehicle, the method comprising: obtaining, by at least one processor, semantic image data associated with an environment in which the autonomous vehicle is operating; determining, by the at least one processor, at least one agent in the environment based on the semantic image data; determining a predicted action for the at least one agent; determining an agent predicted path for the at least one agent based on the determined predicted action; determining a vehicle path for the autonomous vehicle; determining a predicted collision between the at least one agent and the autonomous vehicle based on the agent predicted path of the at least one agent and based on the vehicle path of the autonomous vehicle; simulating actions to avoid the predicted collision; classifying the predicted collision as a major predicted collision based on the simulated actions; and transmitting operating instructions based on classifying the predicted collision as the major predicted collision.

[0210] Clause 42. The method of clause 41, wherein the primary predicted collision is a predicted collision that the autonomous vehicle cannot avoid when simulated.

[0211] Clause 43. The method of clause 41 or 42, wherein taking action is also based on a predicted collision mitigation strategy.

[0212] Clause 44. The method of any one of clauses 41-43, wherein taking action comprises accelerating the autonomous vehicle.

[0213] Clause 45. The method of any of clauses 41-44, further comprising transforming the agent predicted path and the vehicle path into a vehicle path progression and time graph.

[0214] Clause 46. The method of clause 45, wherein determining the predicted collision is based at least in part on the vehicle path progression and time graph.

[0215] Clause 47. The method according to any one of clauses 41-46, wherein simulating an action to avoid the predicted collision further comprises at least one of acceleration, deceleration, left turn, and right turn.

[0216] Item 48. A system comprising: at least one processor, and at least one non-volatile storage medium storing instructions that, when executed by the at least one processor, cause the at least one processor to: obtain, by the at least one processor, semantic image data associated with an environment in which an autonomous vehicle is operating; determine, by the at least one processor, at least one agent in the environment based on the semantic image data; determine a predicted action for the at least one agent; determine an agent predicted path for the at least one agent based on the determined predicted action; determine a vehicle path for the autonomous vehicle; determine a predicted collision between the at least one agent and the autonomous vehicle based on the agent predicted path of the at least one agent and based on the vehicle path of the autonomous vehicle; simulate an action to avoid the predicted collision; classify the predicted collision as a major predicted collision based on the simulated action; and transmit an operating instruction based on classifying the predicted collision as the major predicted collision.

[0217] Clause 49. The system of clause 48, wherein the primary predicted collision is a predicted collision that the autonomous vehicle cannot avoid when simulated.

[0218] Clause 50. The system of clause 48 or 49, wherein taking action is also based on a predicted collision mitigation strategy.

[0219] Clause 51. The system of any of clauses 48-50, wherein taking action comprises accelerating the autonomous vehicle.

[0220] Clause 52. The system of any one of clauses 48-51, further comprising transforming the agent predicted path and the vehicle path into a vehicle path progression and time graph.

[0221] Clause 53. The system of clause 52, wherein determining the predicted collision is based at least in part on the vehicle path progression and time graph.

[0222] Clause 54. The system of clauses 48-53, wherein simulating the action to avoid the predicted collision further comprises at least one of acceleration, deceleration, left turn, and right turn.

[0223] Item 55. At least one non-volatile storage medium storing instructions that, when executed by at least one processor, cause the at least one processor to: obtain, by the at least one processor, semantic image data associated with an environment in which an autonomous vehicle is operating; determine, by the at least one processor, at least one agent in the environment based on the semantic image data; determine a predicted action for the at least one agent; determine an agent-predicted path for the at least one agent based on the determined predicted action; determine a vehicle path for the autonomous vehicle; determine a predicted collision between the at least one agent and the autonomous vehicle based on the agent-predicted path of the at least one agent and based on the vehicle path of the autonomous vehicle; simulate actions to avoid the predicted collision; classify the predicted collision as a major predicted collision based on the simulated actions; and transmit operating instructions based on classifying the predicted collision as the major predicted collision.

[0224] Clause 56. At least one non-volatile storage medium as recited in Clause 55, wherein the primary predicted collision is a predicted collision that the autonomous vehicle cannot avoid when simulated.

[0225] Clause 57. At least one non-volatile storage medium as described in Clause 55 or 56, wherein taking action is further based on a predicted collision mitigation strategy.

[0226] Clause 58. At least one non-volatile storage medium as recited in any one of clauses 55-57, wherein taking an action comprises accelerating the autonomous vehicle.

[0227] Clause 59. The at least one non-volatile storage medium of any one of clauses 55-58, further comprising transforming the agent predicted path and the vehicle path into a vehicle path progression and time graph.

[0228] Clause 60. The at least one non-volatile storage medium of Clause 59, wherein determining the predicted collision is based at least in part on the vehicle path progression and time graph.

Claims

1. A method for operating an autonomous vehicle, the method comprising: obtaining, by at least one processor, semantic image data associated with an environment in which the autonomous vehicle is operating; determining, by the at least one processor, a set of agents in the environment based on the semantic image data; determining a set of predicted actions for at least one primary agent in the set of agents; Determining a set of secondary predicted actions for the at least one primary agent based on the set of predicted actions, wherein the set of secondary predicted actions is determined for the at least one primary agent based on a location of the at least one primary agent and based on agent semantic behavior data associated with the at least one primary agent, wherein the agent semantic behavior data includes logic-based rules and exceptions for the predicted agent actions; Determining a set of primary prediction actions excluding secondary prediction actions according to the set of prediction actions; and A path for the autonomous vehicle is generated based on the set of primary predicted actions.

2. The method of claim 1 further comprising causing the autonomous vehicle to operate along the path of the autonomous vehicle.

3. A method according to claim 1 or 2, wherein the set of agents includes objects configured to move.

4. A method according to any one of claims 1-3, wherein the set of agents also includes a set of secondary agents, wherein the set of secondary agents includes agents that are unlikely to interact with the autonomous vehicle.

5. A method according to claim 4, wherein the set of secondary agents includes agents that will inevitably interact with the autonomous vehicle.

6. The method according to any one of claims 4-5 also includes transmitting the set of secondary agents to a processing engine.

7. The method according to any one of claims 4-6 also includes determining a second set of interaction parameters based at least in part on the set of secondary agents.

8. The method according to any one of claims 1-7, wherein the at least one primary agent is a pedestrian on a sidewalk.

9. The method of claim 8, wherein the pedestrian is following a predetermined path.

10. The method of claim 9, wherein the pedestrian is determined to be approaching an intersection between the predetermined path of the pedestrian and the path of the autonomous vehicle.

11. The method of claim 10, wherein the pedestrian is determined to be in collision with the autonomous vehicle.

12. The method of any one of claims 1-11, wherein generating a path for the autonomous vehicle further comprises determining a plurality of alternative paths for the autonomous vehicle.

13. The method of claim 12, wherein the alternate paths include an increased speed during at least a portion of at least one of the alternate paths.

14. The method of claim 12 or 13, wherein the alternative paths include a reduced speed during at least a portion of at least one of the alternative paths.

15. A system comprising: at least one processor; as well as at least one non-volatile storage medium storing instructions that, when executed by the at least one processor, cause the at least one processor to: obtaining, by at least one processor, semantic image data associated with an environment in which the autonomous vehicle is operating; determining, by the at least one processor, a set of agents in the environment based on the semantic image data; determining a set of predicted actions for at least one primary agent in the set of agents; Determining a set of secondary predicted actions for the at least one primary agent based on the set of predicted actions, wherein the set of secondary predicted actions is determined for the at least one primary agent based on a location of the at least one primary agent and based on agent semantic behavior data associated with the at least one primary agent, wherein the agent semantic behavior data includes logic-based rules and exceptions for the predicted agent actions; Determining a set of primary prediction actions excluding secondary prediction actions according to the set of prediction actions; and A path for the autonomous vehicle is generated based on the set of primary predicted actions.

16. The system of claim 15, further comprising operating the autonomous vehicle along the path of the autonomous vehicle.

17. A system according to claim 15 or 16, wherein the set of agents also includes a set of secondary agents, and wherein the set of secondary agents includes agents that are unlikely to interact with the autonomous vehicle.

18. At least one non-volatile storage medium storing instructions that, when executed by at least one processor, cause the at least one processor to: obtaining, by at least one processor, semantic image data associated with an environment in which the autonomous vehicle is operating; determining, by the at least one processor, a set of agents in the environment based on the semantic image data; determining a set of predicted actions for at least one primary agent in the set of agents; A set of secondary predicted actions of the at least one primary agent is determined based on the set of predicted actions, wherein: The set of secondary predicted actions is determined for the at least one primary agent based on the location of the at least one primary agent and based on agent semantic behavior data associated with the at least one primary agent, wherein the agent semantic behavior data includes logic-based rules and exceptions for the predicted agent actions; Determining a set of primary prediction actions excluding secondary prediction actions according to the set of prediction actions; and A path for the autonomous vehicle is generated based on the set of primary predicted actions.

19. The at least one non-volatile storage medium of claim 18, further comprising causing the autonomous vehicle to operate along the path of the autonomous vehicle.

20. At least one non-volatile storage medium according to claim 18 or 19, wherein the set of agents further comprises a set of secondary agents, and wherein the set of secondary agents comprises agents that are less likely to interact with the autonomous vehicle.

Images