Implicit Occupancy for Autonomous Systems

Abstract

Implicit occupation for an autonomous system includes receiving a request for point attributes at a query point that matches a geographic location and obtaining a query point feature vector from a feature map. The feature map encodes a geographic region that includes the geographic location. A first set of multilayer perceptrons in a decoder model processes the query point feature vector to generate an offset. The offset feature vector is obtained from the feature map for the offset. A second set of multilayer perceptrons in the decoder model processes the offset feature vector and the query point feature vector to generate point attributes. The operations further include responding to the request with the point attributes.

Description

[Technical Field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application is a nonprovisional application of and claims the benefit of U.S. Patent Application No. 63 / 424,864, filed November 11, 2022. U.S. Patent Application No. 63 / 424,864 is incorporated herein by reference in its entirety. [Background technology]

[0002] background An autonomous system is an automated driving mode of transportation that does not require a human pilot or human driver to navigate and react to a real-world environment. Rather, an autonomous system includes a virtual driver, which is the decision-making portion of the autonomous system. Specifically, the virtual driver controls the operation of the autonomous system. The virtual driver is an artificial intelligence system that learns how to interact in the real world and then executes those interactions when in the real world. Summary of the Invention [Problem to be solved by the invention]

[0003] Part of real-world interaction is collision avoidance with other parties and objects in the environment. To navigate real-world environments autonomously and safely, predictions not only need to be accurate and generalize across many scenarios, but also be made in a timely manner so that the autonomous system can react appropriately.

[0004] Most autonomous systems are object-based. In object-based systems, objects of interest are first detected within a region. To do so, the object detector uses a threshold confidence score for each object to determine which objects are present in the scene, where the threshold is essentially a trade-off between accuracy and cancellation. Then, for each object, an object-based motion estimation method is performed to predict only a few sample trajectories or parametric distributions for each object.

[0005] Recently, object-free methods have been used. Object-free methods do not detect individual objects. Rather, they predict the occupancy probability and motion for each cell in a space-time grid directly from sensor data. More specifically, the space-time grid is a three-dimensional dense grid with two spatial dimensions representing the bird's-eye view and a temporal dimension extending from the current observation time to the optimal future range. All dimensions of the three-dimensional grid are quantized at regular intervals. Therefore, object-free methods can be computationally expensive. However, detection confidence thresholding is not used, and the distribution for future motion is much more expressive, allowing downstream motion planners to plan for low-probability objects and futures. [Means for solving the problem]

[0006] overview In general, in one aspect, one or more embodiments relate to a method that includes receiving a request for point attributes at a query point that matches a geographic location and obtaining a query point feature vector from a feature map. The feature map encodes a geographic region that includes the geographic location. A first set of multilayer perceptrons in a decoder model processes the query point feature vector to generate an offset. The offset feature vector is obtained from the feature map for the offset. A second set of multilayer perceptrons in the decoder model processes the offset feature vector and the query point feature vector to generate point attributes. The method further includes responding to the request with the point attributes.

[0007] In general, in one aspect, one or more embodiments relate to a system including a computer processor and a non-transitory computer-readable medium for causing the computer processor to perform operations. The operations include receiving a request for point attributes at a query point that matches a geographic location and obtaining a query point feature vector from a feature map. The feature map encodes a geographic region that includes the geographic location. A first set of multilayer perceptrons in a decoder model processes the query point feature vector to generate an offset. The offset feature vector is obtained from the feature map for the offset. A second set of multilayer perceptrons in the decoder model processes the offset feature vector and the query point feature vector to generate point attributes. The operations further include responding to the request with the point attributes.

[0008] In general, in one aspect, one or more embodiments relate to a non-transitory computer-readable medium including computer-readable program code for causing a computer system to perform operations. The operations include receiving a request for point attributes at a query point that matches a geographic location and obtaining a query point feature vector from a feature map. The feature map encodes a geographic region that includes the geographic location. A first set of multilayer perceptrons in a decoder model processes the query point feature vector to generate an offset. The offset feature vector is obtained from the feature map for the offset. A second set of multilayer perceptrons in the decoder model processes the offset feature vector and the query point feature vector to generate point attributes. The operations further include responding to the request with the point attributes.

[0009] Other aspects of the present invention will become apparent from the following description and appended claims. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 1 illustrates an autonomous system with a virtual driver, according to one or more embodiments. [Figure 2] FIG. 1 illustrates a simulation environment for training a virtual driver of an autonomous system, in accordance with one or more embodiments of the present invention. [Figure 3] FIG. 2 is a diagram of components of a virtual driver in accordance with one or more embodiments of the present invention. [Figure 4] FIG. 1 is a diagram of a feature map in accordance with one or more embodiments of the present invention. [Figure 5] FIG. 10 is a diagram of components of a virtual driver with a close-up view of an encoder model in accordance with one or more embodiments of the present invention. [Figure 6] FIG. 10 is a diagram of components of a virtual driver with an expanded view of an implicit decoder model in accordance with one or more embodiments of the present invention. [Figure 7]1 is a flowchart for implicit occupancy determination in accordance with one or more embodiments of the present invention. [Figure 8] FIG. 1 is an exemplary diagram of explicit and implicit occupancy, according to one or more embodiments of the present invention. [Figure 9] FIG. 1 illustrates an exemplary implementation according to one or more embodiments. [Figure 10A] FIG. 1 illustrates a computing system in accordance with one or more embodiments of the present invention. [Figure 10B] FIG. 1 illustrates a computing system in accordance with one or more embodiments of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0011] Similar elements in the various figures are designated by similar reference numerals for consistency. Detailed Description Generally, embodiments relate to implicit occupancy of a geographic region for an autonomous system. In particular, the geographic region includes agents, physical objects, and various map elements. Agents are participants within the geographic region capable of independent decision-making and movement, and physical objects can be fixed or temporary items that may or may not move. Map elements are physical portions of the geographic region that can be reflected in a map of the geographic region. Agents and physical objects may be located at various geographic locations within the geographic region. Whether an agent or physical object is located at a geographic location is an occupancy of the geographic location. That is, occupancy of a geographic location is a binary matter of whether the geographic location is occupied or not at a particular time. Determining occupancy is important for an autonomous system because a collision occurs when an autonomous system moves into an occupied geographic location.

[0012] Rather than identifying individual agents or objects and corresponding trajectories, one or more embodiments predict whether a particular geographic location within a geographic region will be occupied without considering the particular agent or physical object performing the occupancy. In determining whether it is safe for an autonomous system to travel to a particular location, embodiments effectively combine the identification of the agent and physical object, the corresponding trajectory, and whether the corresponding trajectory includes the geographic location into a single prediction of whether the geographic location will be occupied.

[0013] Moreover, one or more embodiments perform predictions for each query point. The query points are used as inputs to various machine learning models that determine implicit occupancy. In one or more embodiments, rather than constructing an occupancy grid and performing a search within the occupancy grid, occupancy is determined for only a subset of geographic locations. By not constructing the entire occupancy grid, computing resources may be saved. Furthermore, while the occupancy grid has a fixed resolution, the query points are not limited to a fixed location and size in one or more embodiments.

[0014] In one or more embodiments, execution of implicit occupancy proceeds as follows: A feature map is obtained for a geographic region. The feature map has features related to the current and past state of the geographic region regarding which geographic region was occupied, among other possible information (e.g., motion features, purpose such as lane change areas, geometry, object type, etc.). A request for point attributes is received at a query point. The query point is for a particular geographic location. The query point is used to obtain a query point feature vector from the feature map. The query point feature vector is passed through a first set of multi-layer perceptrons in the decoder model to obtain a set of offsets corresponding to the offset location within the geographic region. An offset feature vector for the offset location is obtained from the feature map. The query point feature vector and the offset feature vector are processed by a second set of multi-layer perceptrons in the decoder model to generate point attributes. The point attributes are returned.

[0015] Referring to the figures, Figures 1 and 2 show exemplary diagrams of an autonomous system and a virtual driver. Referring to Figure 1, the autonomous system (116) is an automated driving mode of transportation that does not require a human pilot or human driver to navigate and react to a real-world environment. The autonomous system (116) can be fully autonomous or semi-autonomous. As a mode of transportation, the autonomous system (116) is contained within a housing configured to navigate through a real-world environment. Examples of autonomous systems include autonomous vehicles (e.g., autonomous trucks and cars), drones, airplanes, robots, etc.

[0016] The autonomous system (116) includes a virtual driver (102), which is the decision-making portion of the autonomous system (116). The virtual driver (102) is an artificial intelligence system that learns how to interact in the real world and interacts accordingly. The virtual driver (102) is software running on a processor that makes decisions and causes the autonomous system (116) to interact with the real world, including moving, signaling, and stopping, or maintaining a current state. Specifically, the virtual driver (102) is decision-making software running on hardware (not shown). The hardware may include a hardware processor, memory or other storage devices, and one or more interfaces. The hardware processor is any hardware processing unit configured to process computer-readable program code and perform the operations described in the computer-readable program code.

[0017] The real-world environment is the portion of the real world in which the autonomous system (116) is designed to navigate when trained. Thus, the real-world environment may include concrete and land, buildings, and other objects within a geographic area along with the agent. Agents are other agents within the real-world environment that can navigate within the real-world environment. Agents may have independent decision-making capabilities. The agent's independent decision-making capabilities can dictate how the agent navigates within the environment and may be based on visual or tactile cues from the real-world environment. For example, agents may include other autonomous and non-autonomous traffic systems (e.g., other vehicles, bicyclists, robots), pedestrians, animals, etc.

[0018] In the real world, a geographic region is an actual region in the real world that surrounds an autonomous system. That is, from the perspective of a virtual driver, a geographic region is the region through which the autonomous system travels. A geographic region includes agents and map elements located in the real world. That is, agents and map elements each have a physical location within the geographic region that indicates where the corresponding agent or map element is located. Map elements are stationary within a geographic region, while agents may or may not be stationary within a geographic region. Map elements are elements that are shown on a map (e.g., a road map, a traffic map, etc.) or are derived from a map of a geographic region.

[0019] The real-world environment changes as the autonomous system (116) moves through it, for example, geographic regions may change and agents may move location, including new agents being added and existing agents leaving.

[0020] To interact with the real-world environment, the autonomous system (116) includes various types of sensors (104), such as LiDAR sensors, among other types, used to obtain measurements of the real-world environment, and cameras that capture images from the real-world environment. The autonomous system (116) may also include other types of sensors. The sensors (104) provide input to the virtual driver (102).

[0021] In addition to the sensors (104), the autonomous system (116) includes one or more actuators (108). An actuator is hardware and / or software configured to control one or more physical parts of the autonomous system based on a control signal from the virtual driver (102). In one or more embodiments, the control signal specifies an action for the autonomous system (e.g., turn on turn signals, brake a specified amount, accelerate a specified amount, rotate the steering wheel or tires a specified amount, etc.). The actuators (108) are configured to implement the action. In one or more embodiments, the control signal may specify a new state for the autonomous system, and the actuators may be configured to implement the new state to bring the autonomous system into the new state. For example, the control signal may specify that the autonomous system should turn a specific amount while accelerating at a predetermined speed, and the actuators determine the movement of the wheels and the amount of acceleration for the accelerator, causing them to achieve the specific amount of rotation and acceleration.

[0022] Testing and training a virtual driver (102) of an autonomous system in a real-world environment is unsafe due to the potential for accidents caused by an untrained virtual driver. Therefore, as shown in FIG. 2, a simulator (200) is configured to train and test a virtual driver (102) of an autonomous system. For example, the simulator can be a unified, modular, mixed reality, closed-loop simulator for an autonomous system. The simulator (200) is a configurable simulation framework that enables evaluation of different autonomous components of a single virtual driver (102) as well as evaluation as a complete system in a closed-loop manner. The simulator automatically reconstructs a "digital twin" of a real-world scenario, enabling accurate evaluation of virtual drivers at scale. The simulator (200) creates a simulated environment (204), a virtual world in which the virtual driver (102) is a player in the virtual world. The simulated environment (204) is a simulation of a real-world environment, which may or may not actually exist, in which the autonomous system is designed to navigate. As such, the simulated environment (204) includes a simulation of objects (i.e., simulated objects or agents) and backgrounds in the real world, including natural objects, architecture, buildings and roads, obstacles, and other autonomous and non-autonomous objects. The simulated environment simulates the environmental conditions in which an autonomous system may be deployed. The simulated objects may include both stationary and non-stationary objects. Non-stationary objects are agents in the real-world environment.

[0023] In a simulated environment, a geographic region is a realistic representation of a real-world region that may or may not actually exist. That is, from the perspective of a virtual driver, a geographic region appears as if the geographic region existed when it does not actually exist, or appears the same as an actual geographic region that exists in the real world. A geographic region in a simulated environment includes virtual agents and virtual map elements that would be actual agents and actual map elements in the real world. That is, the virtual agents and virtual map elements each have a physical location within the geographic region that indicates the exact spot or place where the corresponding agent or map element is located. The map elements are stationary within the geographic region, but the agents may or may not be stationary within the geographic region. As in the real world, there is a map of the geographic region that specifies the physical locations of the map elements.

[0024] The simulator (200) includes an autonomous system model (216), a sensor simulation model (214), and an agent model (218). The autonomous system model (216) is a detailed model of the autonomous system in which the virtual driver (102) executes. The autonomous system model (216) includes the model, geometry, physical parameters (e.g., mass distribution, significant points), engine parameters, sensor locations and types, sensor firing patterns, information about the hardware on which the virtual driver executes (e.g., processor power, amount of memory, and other hardware information), and other information about the autonomous system. Various parameters of the autonomous system model may be configurable by a user or another system.

[0025] The autonomous system model (216) includes an autonomous system dynamic model. The autonomous system dynamic model is used in the dynamics simulation to obtain the virtual driver's operational actions (e.g., steering angle, desired acceleration) and execute the operational actions on the autonomous system in the simulated environment to update the state of the simulated environment and the autonomous system. The interface between the virtual driver (102) and the simulator (200) may match the interface between the virtual driver (102) and the autonomous system in the real world. Thus, for the virtual driver (102), the simulator simulates the virtual driver's experience in the autonomous system in the real world.

[0026] In one or more embodiments, the sensor simulation model (214) models active and passive sensor inputs in the simulated environment. The sensor simulation model (114) is configured to simulate sensor observations of a surrounding scene in the simulated environment (204) at each time step according to a sensor configuration on the vehicle platform. The passive sensor inputs capture the visual appearance of the simulated environment, including stationary and non-stationary simulated objects, from the perspective of one or more cameras based on the simulated positions of the cameras in the simulated environment. Examples of passive sensor inputs include inertial measurement units (IMUs) and thermal inputs. Active sensor inputs are inputs to the autonomous system's virtual driver from active sensors, such as LiDAR, RADAR, global positioning system (GPS), and ultrasonic. That is, the active sensor inputs include measurements obtained by sensors, which are simulated based on the simulated environment based on the simulated positions of the sensors in the simulated environment.

[0027] The agent model (218) represents an agent in a scenario. An agent is a sentient entity with an independent decision-making process. That is, in the real world, an agent may be a living being (e.g., a person or an animal) that makes decisions based on the environment. An agent may be active rather than, or in addition to, passive. An instance of an agent model, or participant model, may exist for each agent in a scenario. The agent model is a model of the agent. If the agent is in a transportation mode, the agent model includes a model of the transportation mode in which the agent is located. For example, participant models may represent pedestrians, children, vehicles driven by drivers, pets, bicycles, and other types of participants.

[0028] Figure 3 shows a schematic diagram of a virtual driver (102) with an implicit occupancy system according to one or more embodiments. As shown in Figure 3, the virtual driver (102) is connected to a map data repository (302) and sensors (300). The map data repository (302) is a storage repository for map data (304). The map data (304) is a map of a geographic region with the map elements described above located at respective geographic locations within the map. For example, the map data may include lanes and road centerlines at corresponding locations on the map.

[0029] The sensor (300) may be a virtual sensor (e.g., the sensor simulation model (214) shown in FIG. 2) or a physical sensor (e.g., the sensor (104) shown in FIG. 1). The sensor provides sensor data (306). The sensor data (306) may include a LiDAR sweep, a LiDAR point cloud, a camera image, or other types of sensor data. A LiDAR sweep provides a set of LiDAR points radiating outward from the LiDAR sensor. A LiDAR point cloud is a set of LiDAR points at corresponding positions.

[0030] Continuing to refer to Figure 3, the virtual driver (102) includes an encoder model (308), an implicit decoder model (310), an autonomous system route selector (312), and an autonomous system controller (314). Each of these components is described below.

[0031] The encoder model (308) is a machine learning model configured to acquire sensor data (306) from the sensor (300) and map data (304) from the map data repository (302) and generate a feature map (316) of a geographic region. The encoder model is a machine learning model or collection of machine learning models that encodes the sensor data (306) and map data (304) into the feature map (316). Specifically, the encoder model is designed to learn vector embeddings for the sensor data (306) and map data (304) that are used to predict point attributes at various yet-to-be-specified points in time. The feature map (316) is a map of a geographic region having at least one axis with feature vectors for corresponding locations within the geographic region. The feature vectors are vector embeddings. The feature map (316) is an encoding of the current and past states of the geographic region. In one or more embodiments, the feature map (316) does not include future occupancy information. An example of the feature map (316) is shown in FIG. 4.

[0032] Continuing with reference to FIG. 3 , the implicit decoder model (310) is a machine learning model configured to obtain a set of one or more query points (318) and output a set of one or more point attributes (320) for each of the query points (318). The implicit decoder model (310) is a neural network model configured to obtain and decode feature vectors from feature maps for the query points. The query points may include an identification of a geographic location and a time value. The time value is a future time at which the set of point attributes (320) is predicted. Furthermore, the time may be specified relative to the current time. For example, the time value may be a number of seconds in the future.

[0033] The point attributes (320) are attributes of the geographic location at a specified time. For example, the point attributes (320) may include a binary value of occupancy or non-occupancy, a probability value of occupancy or non-occupancy, a backflow vector specifying where the object or agent occupying the geographic location came from, an agent type identifier, an object type identifier, or other attributes of the geographic point. The agent type identifier may be an identifier of the type of agent performing the occupancy without identifying the agent or the trajectory of the agent itself. In the case of an autonomous system that is a vehicle, the agent type identifier may be pedestrian, truck, car, bicyclist, etc.

[0034] The autonomous system route selector (312) is configured to select a route for the autonomous system using the map data (304). The autonomous system route selector (312) can select a route using routing information, current sensor data (306), point attributes (320), and other inputs. The route includes a trajectory and acceleration or speed. For example, the route may include slowing down, turning, accelerating or decelerating, waiting, or performing another action on the same trajectory.

[0035] The autonomous system controller (314) is a software process configured to send control signals to actuators of the autonomous system and to determine actions for the autonomous system to execute the route of the autonomous system route selector.

[0036] Figures 4-6 show exemplary enlarged views of various components of Figure 3, according to one or more embodiments. Figure 4 shows a diagram of a three-dimensional feature map (400), according to one or more embodiments of the present invention. The feature map of Figure 4 may correspond to the feature map (316) shown in Figure 3.

[0037] In the feature map (400) of FIG. 4 , the first and second dimensions correspond to a bird's-eye view of a geographic region. A bird's-eye view may also be referred to as an aerial perspective or top-down view of a geographic region. Specifically, the first axis of the feature map (400) is the first axis of the bird's-eye view (402), and the second axis of the feature map is the second axis of the bird's-eye view (404). For example, the first axis may correspond to the east-west axis of the geographic region, and the second axis may correspond to the north-south axis of the geographic region. The first and second axes may correspond to different axes of the geographic region. Thus, the plane formed by the first and second axes may coincide with a road or traffic map of the geographic region.

[0038] Furthermore, the feature map is a multidimensional grid. A grid is a division of a region into cells. In the three-dimensional feature map of FIG. 4, two dimensions of the three-dimensional grid, corresponding to the first and second axes, divide the geographic region into individual grid cells. Thus, each geographic location within the geographic region is within a specific grid cell. The third dimension corresponds to the third axis of the feature map, which is the feature vector axis (406). The feature vector axis has an individual corresponding feature vector for each grid cell in the first and second axes. There may be a one-to-one mapping between the feature map and the grid cells in the other dimensions. A feature vector is a vector of feature values. In one or more embodiments, the feature vector is a fixed size. In summary, the feature map divides the geographic region into subregions, whereby each subregion has a corresponding feature vector.

[0039] While Figure 4 shows a three-dimensional feature map, a four-dimensional feature map may be used. For example, if the autonomous system is an aircraft, the third dimension may be a geographic location in three-dimensional space, and the fourth dimension may be a feature vector.

[0040] 5 shows a diagram of the components of a virtual driver with a close-up view of the encoder model 308, in accordance with one or more embodiments of the present invention. Components in FIG. 5 that have the same reference numbers as similarly named components in FIG. 3 are the same as or similar to the similarly named components.

[0041] As shown in FIG. 5, the encoder model (308) may be a specific combination of multiple models. The encoder model (308) may include a map encoder model (502), a sensor data encoder model (504), a concatenator (506), and a combined encoder model (508). The map encoder model (502) is a machine learning model configured to convert map data into a map feature vector for each subregion of a geographic region. Specifically, as described above with reference to FIG. 4, a feature map divides a geographic region into subregions, with each subregion corresponding to a grid cell on two dimensions of the three-dimensional grid of the feature map. The map encoder model (502) generates a map encoding of the map used for the map feature vector. Thus, the output of the map encoder model (502) is a map feature map, similar to the feature map described above with reference to FIG. 4, but having only map data features. An example of a map encoder model may be or may include a convolutional neural network.

[0042] The sensor data encoder model (504) is configured to encode the sensor data (306). If the sensor data is LiDAR, the LiDAR data may be received as a list of LiDAR points. The LiDAR points in the list may be voxelized in a three-dimensional LiDAR grid, with each grid cell being for a geographic location. For each grid cell in the LiDAR grid, the grid cell's value may be set to 1 if the LiDAR point is present in the list identifying the grid cell, or 0 if the point is not present. The result of the voxelization is a binary three-dimensional grid for the geographic area specifying where the LiDAR points are located. Multiple LiDAR sweeps may be combined or voxelized in the same LiDAR grid. In such a scenario, if any of the LiDAR points in any of the LiDAR sweeps identify the geographic location of the grid cell, the grid cell of the LiDAR grid may be set to 1. Thus, if multiple LiDAR sweeps are combined current or past sweeps, the LiDAR may also reflect the immediately preceding occupancy of the three-dimensional geographic area. Although binary values ​​for grid cells of a LiDAR grid are described, grid cell values ​​may be set based on the time elapsed since the LiDAR sweep was performed. Additionally, rather than a three-dimensional LiDAR grid, a two-dimensional LiDAR grid may be used, thereby projecting the third dimension onto the bird's-eye view.

[0043] The sensor data encoder model (504) can then generate a vector embedding of the LiDAR grid. The vector embedding is a sensor data feature vector for each grid cell of the bird's-eye view of the geographic region in one or more embodiments. That is, the output of the sensor data encoder model (504) is a sensor data feature map, such as the feature map described with reference to FIG. 4, but with only sensor data features. The sensor data encoder model (504) can be a convolutional neural network.

[0044] The concatenator (506) is configured to concatenate each map feature vector with a corresponding sensor data feature vector to generate a concatenated feature vector. Two feature vectors correspond when the two feature vectors are for the same subregion of the geographic region. The concatenated feature vector may have a first portion of the map feature vector and a second portion of the sensor data feature vector. In other words, the map feature map and the sensor data feature map may have the same resolution with respect to the dimension corresponding to the geographic region. The concatenator can overlay the map feature map on the sensor data feature map to generate the concatenated feature map. Thus, the concatenated feature vector has a potential description of the geometry of the geographic region (i.e., as specified in the map data) and movement around the geographic region.

[0045] The combined encoder model (508) is an encoder model that combines the feature vectors of the map feature vector and the sensor data feature vector. Specifically, the combined encoder model can generate a set of features that represent both the map elements and the sensor data. The combined encoder model may also include convolutional layers. The combined feature map may be of the same or different resolution or size as the feature map generated by the combined encoder model (508).

[0046] Various techniques may be used to implement various encoder models. For example, a vision transformer model may be used. As another example, an encoder model may include a convolutional neural network layer connected to one or more attention layers connected to further convolutional neural network layers.

[0047] 6 shows a diagram of the components of a virtual driver with an expanded view of the implicit decoder model (310) in accordance with one or more embodiments of the present invention. Components in FIG. 6 that have the same numbers as corresponding components in FIG. 3 are the same as or similar to the corresponding components.

[0048] In one or more embodiments, the implicit decoder model (310) is configured to process query points in parallel with each other. Therefore, for purposes of illustration, a single query point is shown. However, the implicit decoder model (310) can execute the same pipeline across several query points.

[0049] The implicit decoder model (310) includes a query point feature interpolator (602) configured to interpolate a point feature vector (604) from the feature map (316). The query point feature interpolator (602) takes a geographic location as input and interpolates a new feature vector (i.e., the point feature vector (604)) from the closest feature vector in the feature map to the geographic location. Thus, while the feature map may have a predetermined resolution indicated by the size of the subregion of the geographic area corresponding to each grid cell, the determination of point attributes may be at any resolution.

[0050] The first multi-layer perceptron (606) is a set of neural network layers that takes a query point (318) and a point feature vector (604) as input and generates offsets (608) as output. The offsets specify a distance and direction from the query point (318). Each offset corresponds to an offset location within the geographic region, whereby the offset location is a physical location within the geographic region that is offset from the query point. In one or more embodiments, the number of offsets is predefined.

[0051] The offsets (608) are processed by an offset feature interpolator (610) to generate an offset feature vector (612). The offset feature interpolator (610) may perform the same function as the query point feature interpolator (602), except for the offset locations instead of geographic locations within the query points (318). For example, the offset feature interpolator (610) may use the same code blocks as the query point feature interpolator (602). The output of the offset feature interpolator (610) is the offset feature vector (612).

[0052] The cross-attention layer (614) takes the offset feature vector (612) and the point feature vector (604) and generates a combined feature vector (616). The combined feature vector has aggregate features aggregated from the offset feature vector (612) and the point feature vector (604).

[0053] The concatenator (618) is configured to concatenate the point feature vector (604) with the combined feature vector (616). For example, the concatenation can be by appending the point feature vector (604) to the end of the combined feature vector (616). The concatenator (618) generates a combined feature vector (620).

[0054] The concatenated feature vector (620) is used along with the query point (318) as input to a second multilayer perceptron (622) that produces a set of point attributes (320) as output. The second multilayer perceptron (622) is a neural network layer that can classify geographic locations within the query point as occupied or unoccupied, provide backflow, and perform other classifications.

[0055] 7 illustrates a flowchart for performing implicit occupancy, according to one or more embodiments. In one or more embodiments, prior to performing the operations of FIG. 7, a feature map is generated. The feature map is used for the current set of query points. As the autonomous system moves through the environment, new feature maps, either physical or virtual, are generated to accommodate traffic movement through the area. Thus, the generation of the feature map is performed in real time.

[0056] In one or more embodiments, the LiDAR data is acquired as a set of LiDAR sweeps of a geographic region. Each LiDAR sweep includes a set of LiDAR points. As the autonomous system moves through the environment, a LiDAR sensor of the autonomous system performs LiDAR sweeps. In the virtual environment, a sensor simulation model simulates LiDAR sweeps that would be generated based on the current state of the virtual environment. Thus, LiDAR sweep data may be provided in both the simulated environment and the virtual environment. Binary values ​​of grid cells in the three-dimensional LiDAR grid are set according to the location of the grid cell identified by a LiDAR point in a set of LiDAR points of at least one LiDAR sweep in the set of LiDAR sweeps. A sensor data encoder model is then run on the LiDAR grid to encode the LiDAR grid to generate a sensor feature map. While LiDAR sensor data is described as being used to generate the sensor feature map, camera images may also be used. In such a scenario, the camera images may be passed through a machine learning model to generate a set of bird's-eye view camera feature maps of the region over time. The bird's-eye view may be passed through a sensor data encoding model to generate a sensor data feature map.

[0057] Similarly, a road map of a geographic region may be encoded via a map encoder model to generate a map encoding, which is a map feature map. In some embodiments, the map feature map may be pre-generated.

[0058] The map encoding and the sensor encoding are combined by concatenating the map feature grid with the sensor data feature grid to generate a combined feature encoding, which is processed through a combined encoder model to generate a feature map.

[0059] The process of generating the feature maps may be performed asynchronously with running the implicit decoder model. In one or more embodiments, once the feature maps are generated, the same feature maps are used to provide point attributes responsive to the query points. Thus, for a particular query point, the same feature map is used for both the query point feature vector and the offset feature vector.

[0060] At block 702, a request for point attributes is received at query points that match a geographic location. In one or more embodiments, the implicit decoder receives the request with a set of query points. For example, the set of query points may be received from different models of a virtual driver attempting to select a trajectory for an autonomous system. The set of query points may be received as a list of query points. Each query point may include a geographic location identifier and a time of the geographic location. That is, the time may be the time at which the point attributes are requested. The implicit decoder can process each query point individually and in parallel.

[0061] At block 704, a query point feature vector is obtained from the feature map. In some embodiments, the query point feature vector may be obtained directly from the feature map. For example, the implicit decoder model may process the query points at the same resolution as the feature map. In such a scenario, the location specified in the query point is used to search for the location in the feature map that corresponds to the subregion having that location. The corresponding feature vector is returned as the query point feature vector.

[0062] In some embodiments, the query point feature vector is a combination of multiple feature vectors. A feature vector in a feature map may be associated with the centroid of the corresponding subregion to which the feature vector corresponds. Thus, rather than to the entire subregion, the feature vector is associated with a particular point within the subregion. Here, associated means that the feature vector is mapped to or otherwise linked to the centroid of the subregion (e.g., in a one-to-one mapping).

[0063] In embodiments where the query point feature vector is a combination of feature vectors, the following operations may be performed to obtain the query point feature vector: From the entire set of feature vectors in the feature map, a set of feature vectors that are adjacent to the query point in the feature map is selected. Specifically, the set of feature vectors includes feature vectors associated with adjacent centroids of subregions, whereby the adjacent centroids are adjacent to the geographic location specified at the query point. For example, four, six, or nine feature vectors associated with the four, six, or nine nearest centroids may be selected.

[0064] The selected feature vectors are interpolated to obtain a query point feature vector. A bilinear interpolation is performed using the selected feature vectors to obtain the query point feature vector. The bilinear interpolation uses a weighted sum, whereby the weights are based on the relative positions of the selected feature vectors and the query point.

[0065] At block 706, the query point feature vector is processed by a first set of multilayer perceptrons of the decoder model to obtain a set of offsets. The query points may be concatenated to the query point feature vector and processed by the first set of multilayer perceptrons. The first set of multilayer perceptrons effectively learns information about objects and parties that may cause the geographic location at the query point to be occupied at a future moment in time, without identifying the objects or parties.

[0066] At block 708, an offset feature vector is obtained from the offset and the feature map. The offset feature vector may be obtained using the same or similar techniques as obtaining the query point feature vector. In one or more embodiments, the offset is processed separately as follows: The offset is combined with the geographic location at the query point to obtain an offset point. The offset point is a geographic location that is an offset distance and direction from the geographic location at the query point. From the query point, a set of feature vectors is selected based on the proximity in the feature map of the set of feature vectors to the offset point specified by the offset. The set of feature vectors is interpolated using the relative position of the offset with respect to the set of feature vectors to obtain an offset feature vector among the multiple offset feature vectors. Selecting and interpolating the set of feature vectors is performed as described in block 704. The result is a set of offset feature vectors.

[0067] The offset feature vector and the query point feature vector are processed through a second set of multilayer perceptrons of the decoder model to generate point attributes at block 710. The second set of multilayer perceptrons determines point attributes for the query points.

[0068] In one or more embodiments, preprocessing is performed before processing the offset feature vector and the query point feature vector through a multilayer perceptron. The preprocessing includes the offset feature vector and the query point feature vector first being processed by a cross-attention layer to generate an output vector. The cross-attention layer combines features from the offset feature vector and the query point feature vector when generating an output vector that can be processed by the multilayer perceptron. Further processing may be performed before processing the output vector through the multilayer perceptron. The output vector may be concatenated with the query point feature vector to generate a concatenated vector. Thus, the concatenated vector includes both output vectors that are a combination of features for the offset point and the query point. In effect, because the query point feature vector is concatenated with the output vector, which is a combination, the query point feature vector has more focus on the concatenated vector. A second set of multilayer perceptrons is then run on the concatenated vector combined with the query point. Specifically, the neural network layers of the second set of multilayer perceptrons process the vector concatenated with the query point to generate point attributes.

[0069] In block 712, the decoder model responds to the request with point attributes. The decoder model may provide a result set of point attributes for each query point in the set of query points. One of the point attributes may be the predicted occupancy of the geographic location at the time specified by the query point. The predicted occupancy may be performed by comparing the probability of occupancy to a threshold to generate a binary value. The decoder model may output a binary value or a probability. For probability, the output of the second set of multilayer perceptrons may be a value between negative infinity and infinity. The output may be passed through a sigmoid layer, which changes the value to a probability value between 0 and 1. In some embodiments, multiple occupancy values ​​are output. Each different occupancy value may correspond to a particular type of agent or object. For example, a vector of occupancy values ​​may be output, with each position in the vector corresponding to one of a pedestrian, bicycle, car, truck, inanimate object, or other type of traffic. When the predicted occupancy is that the geographic location is occupied at that time, the set of point attributes may further include a backflow value to the query point. Specifically, a second set of multi-layer perceptrons may be further trained to predict flows to geographic locations.

[0070] In block 714, the autonomous system operates based on the point attributes. The virtual driver can use the occupancy of the query points to determine a current trajectory of the autonomous system that meets safety criteria (e.g., avoiding collisions, ensuring stopping distances, etc.) and other criteria (e.g., shortest route, reducing the number of lane changes, etc.) and facilitates travel to the destination. The virtual driver can then output control signals to one or more actuators. In a real-world environment, the control signals are used by the actuators to cause the autonomous system to perform an action, such as moving in a particular direction at a particular speed or acceleration, waiting, displaying a turn signal, or performing other action. In a simulated environment, the control signals are intercepted by a simulator that simulates the actuators and the resulting behavior of the autonomous system. The simulator simulates the autonomous system, thereby training the virtual driver. That is, the output of simulating the autonomous system in the simulated environment may be used to evaluate the behavior of the virtual driver.

[0071] Training of the system may be performed as follows: A set of training data with labeled agents and objects may be used as input. The set of training data may include past motion of the objects. In particular, a point cloud of objects may be used instead of labeled participants and objects. If a query point lands on an object, the label for the query point is the inverse vector for the query. Thus, from the set of training data, training positions, corresponding times, and backflows to the training positions are defined.

[0072] Specifically, one or more embodiments randomly sample training query points in a geographic region of interest and in future times from a set of training data. The weights of a first set of multilayer perceptrons are initialized so that the offsets have values ​​close to zero. Thus, the initial set of offset points is close to the query points. Through training, the weights of the multilayer perceptrons are updated so that the offset points become more useful and can be increased. The number of offsets is a hyperparameter for the first set of multilayer perceptrons. Training samples and training data are fed through the model. In the case of occupancy, a cross-entropy loss is used. In the case of backpropagation, an L1 loss is calculated when the sample training query points are occupied. After calculating the cross-entropy loss and the L1 loss, backpropagation may be performed to update weights throughout the system.

[0073] FIG. 8 shows an example diagram of domain maps illustrating the difference between generating occupancy values ​​for an entire grid (left map (802)) versus generating implicit occupancy for a set of query points (right map (804)). The autonomous system is an autonomous vehicle shown in the center of each map. As shown, in the left map (802), a fixed resolution grid is generated and an occupancy value is output for each cell in the fixed resolution grid. To accommodate the time-series nature of a tracked autonomous vehicle, multiple such grids are generated for each instant in time. The time corresponding to each grid point is also fixed. Therefore, a large amount of unused data with fixed resolution in both time and space may be generated.

[0074] As shown in the map on the right (804), implicit occupancy uses a set of query points along different trajectories (3 in this example). Each query point has a time when the autonomous system is predicted to be at the query point. The time interval may be the same or different along different trajectories. Furthermore, the determination of whether a query point is occupied is not limited to a fixed resolution, but rather to the query point itself. In particular, the decoder may be further trained to output whether a specified distance around the query point is occupied. Thus, the occupancy problem can be relative to the query point and a threshold distance around the query point. The result is a set of values ​​along a particular trajectory that indicates whether it is occupied or not, and if so, backflow.

[0075] 9 illustrates an example implementation (900) of an encoder model and a decoder model of a virtual driver, according to one or more embodiments. The example is for illustrative purposes only and is not intended to limit the scope of the present invention. In the example of FIG. 9, the first and second multilayer perceptrons are sets of residual blocks.

[0076]

number

[0077]

number

[0078]

number

[0079]

number

[0080]

number

[0081]

number

[0082]

number

[0083]

number

[0084]

number

[0085]

number

[0086] Thus, as shown, the system is trained to predict occupancy and flow for a specific query point. One or more embodiments can provide a unified approach to collaborative perception and prediction for autonomous driving that uses neural networks to implicitly represent occupancy and flow over time. This queryable implicit representation can more effectively and efficiently inform downstream motion planners. The implicit architecture predicts occupancy and flow more accurately than contemporary explicit approaches in both urban and highway environments.

[0087] As noted above, implicit occupancy does not identify agents or physical objects within a geographic region to predict whether a geographic location is occupied, although agents or objects may be identified for other purposes without departing from the scope of the present invention.

[0088] The embodiments may be implemented on a computing system specifically designed to achieve improved technical results. When implemented within a computing system, the features and elements of the present disclosure provide a significant technical advance over computing systems that do not implement the features and elements of the present disclosure. Any combination of mobile, desktop, server, router, switch, embedded device, or other type of hardware may be improved by including the features and elements described in the present disclosure. For example, as shown in FIG. 10A , a computing system (1000) may include one or more computer processors (1002), non-persistent storage (1004), persistent storage (1006), communication interfaces (1008) (e.g., Bluetooth interface, infrared interface, network interface, optical interface, etc.), and numerous other elements and functions that implement the features and elements of the present disclosure. The computer processor (1002) may be an integrated circuit for processing instructions. The computer processor may be one or more cores or micro-cores of a processor. The computer processor (1002) includes one or more processors. The one or more processors may include a central processing unit (CPU), a graphics processing unit (GPU), a tensor processing unit (TPU), combinations thereof, and the like.

[0089] The input device (1010) may include a touchscreen, keyboard, mouse, microphone, touchpad, electronic pen, or any other type of input device. The input device (1010) can receive input from a user in response to data and messages presented by the output device (1012). The input may include text input, audio input, video input, etc., which may be processed and transmitted by the computing system (1000) according to the present disclosure. The communication interface (1008) may include integrated circuits for connecting the computing system (1000) to a network (not shown) (e.g., a local area network (LAN), a wide area network (WAN) such as the Internet, a mobile network, or any other type of network) and / or to another device, such as another computing device.

[0090] Additionally, the output device(s) (1012) may include a display device, a printer, external storage, or any other output device. One or more of the output devices may be the same as or different from the input devices. The input and output devices may be locally or remotely connected to the computer processor (1002). Many different types of computing systems exist, and the aforementioned input and output devices may take other forms. The output device (1012) may display data and messages sent and received by the computing system (1000). The data and messages may include text, audio, video, etc., and may include the data and messages described above in other figures of this disclosure.

[0091] Software instructions in the form of computer-readable program code for carrying out the embodiments may be stored, in whole or in part, temporarily or permanently, on a non-transitory computer-readable medium such as a storage device, a diskette, a tape, a flash memory, a physical memory, or any other computer-readable storage medium. In particular, the software instructions may correspond to computer-readable program code configured to carry out one or more embodiments, which, when executed by a processor, may include sending, receiving, presenting, and displaying data and messages as described in other figures of this disclosure.

[0092] The computing system (1000) of FIG. 10A may be connected to or be part of a network. For example, as shown in FIG. 10B, the network (1020) may include multiple nodes (e.g., node X (1022), node Y (1024)). Each node may correspond to a computing system such as the computing system shown in FIG. 10A, or a group of nodes combined may correspond to the computing system shown in FIG. 10A. As an example, an embodiment may be implemented on a node of a distributed system connected to other nodes. As another example, an embodiment may be implemented on a distributed computing system having multiple nodes, each part of which may be located on a different node within the distributed computing system. Furthermore, one or more elements of the computing system (1000) described above may be located remotely and connected to other elements via a network.

[0093] Nodes in the network (1020) (e.g., node X (1022), node Y (1024)) may be configured to provide services to client devices (1026), including receiving requests and sending responses to the client devices (1026). For example, the nodes may be part of a cloud computing system. The client devices (1026) may be computing systems, such as the computing system illustrated in FIG. 10A. Additionally, the client devices (1026) may include and / or execute all or a portion of one or more embodiments.

[0094] The computing system of FIG. 10A may include functionality for presenting raw data and / or processed data, such as the results of comparisons and other processing. For example, presenting data may be achieved through various presentation methods. Specifically, data may be presented by being displayed on a user interface, transmitted to a different computing system, and stored. The user interface may include a graphical user interface (GUI) that displays information on a display device. The GUI may include various GUI widgets that organize what data is displayed and how the data is presented to the user. Furthermore, the GUI can present data directly to the user, for example, data presented as actual data values ​​via text or data rendered into a visual representation of the data by the computing device, such as via visualizing a data model.

[0095] As used herein, the term "connected to" is intended to have multiple meanings. A connection can be direct or indirect (e.g., through another component or network). A connection can be wired or wireless. A connection can be a temporary, permanent, or semi-permanent communication channel between two entities.

[0096] The various illustrations in the figures may be combined and may include or be included within features described in other figures of this application. Various elements, systems, components, and steps shown in the figures may be omitted, repeated, combined, and / or modified as shown in the figures. Thus, the scope of the present disclosure should not be considered limited to the specific configurations shown in the figures.

[0097] Ordinal numbers (e.g., first, second, third, etc.) may be used in this application as adjectives for elements (i.e., any noun in this application). The use of ordinal numbers does not imply or create any particular order of elements, nor does it limit any element to being only a single element, unless expressly disclosed, such as by use of "before," "after," "single," and other such terms. Rather, the use of ordinal numbers is to distinguish between elements. As an example, a first element may be different from a second element, and a first element may encompass two or more elements and follow (or precede) the second element in the order of elements.

[0098] Furthermore, unless otherwise stated, "or" is an inclusive or, and thus includes "and." Furthermore, items joined by "or" may include any combination of items with any number of each item, unless otherwise stated.

[0099] In the above description, numerous specific details are set forth to provide a more complete understanding of the present disclosure. However, it will be apparent to those skilled in the art that the present technology may be practiced without these specific details. In other instances, well-known features have not been described in detail to avoid unnecessarily complicating the description. Furthermore, other embodiments not expressly described above may be devised that do not depart from the scope of the claims disclosed herein. Accordingly, the scope should be limited only by the scope of the appended claims.

Claims

1. 1. A method comprising: receiving a request for point attributes at a query point that matches the geographic location; and obtaining a query point feature vector from a feature map, the feature map encoding a geographic region that includes the geographic location, the method further comprising: processing the query point feature vector with a first set of multi-layer perceptrons in a decoder model to generate a plurality of offsets; obtaining a plurality of offset feature vectors for the plurality of offsets from the feature map; processing the plurality of offset feature vectors and the query point feature vector with a second set of multilayer perceptrons in the decoder model to generate the point attributes; and responding to the request with the point attributes.

2. Obtaining the query point feature vector includes: selecting a first set of feature vectors from the feature map that are adjacent to the query point in the feature map; and performing bilinear interpolation using the first set of feature vectors to obtain the query point feature vector.

3. 3. The method of claim 2, wherein interpolating the first set of feature vectors uses a weight for each feature vector of the first set of feature vectors that depends on the relative position of the feature vector to the query point in the feature map.

4. Obtaining the plurality of offset feature vectors includes: For each of the offsets of the plurality of offsets, selecting a second set of feature vectors from the feature map based on their proximity to an offset point in the feature map specified by the offset; and performing bilinear interpolation using the second set of feature vectors to obtain an offset feature vector of the plurality of offset feature vectors.

5. The feature map is a first axis and a second axis comprising a bird's-eye view of the geographical area; and a third axis comprising a set of features generated by encoding LiDAR data and map data for the geographic region.

6. acquiring the LiDAR data as a set of LiDAR sweeps of the geographic area, each set of LiDAR sweeps including a set of LiDAR points, the method further comprising: setting binary values ​​of the grid cells within the three-dimensional grid according to positions of the grid cells identified by LiDAR points in the set of LiDAR points of at least one of the LiDAR sweeps in the set of LiDAR sweeps; The method of claim 5 , further comprising: encoding the three-dimensional grid with a sensor data encoder model.

7. The method of claim 6 , wherein the sensor data encoder model comprises a convolutional neural network.

8. The method of claim 1 , further comprising encoding the sensor data with a sensor data encoder model to obtain encoded sensor data.

9. encoding a map of the geographic region via a map encoder model to generate a map encoding; The method of claim 8 , further comprising concatenating the map encoding with the encoded sensor data to generate a joint feature encoding.

10. The method of claim 9 , further comprising processing the joint feature encoding through a joint encoder model to generate the feature map.

11. The method of claim 1 , wherein the point attribute is a predicted occupancy of the geographic location at a time specified by the query point.

12. The method of claim 1 , wherein the point attributes further include a flow-back value to the query point.

13. The method of claim 1 , wherein the query point includes an identifier of the geographic location and a time of the geographic location.

14. processing the plurality of offset feature vectors and the query point feature vector through a cross-attention layer to generate an output vector; The method of claim 1 , wherein processing the plurality of offset feature vectors and the query point feature vector with the second set of multilayer perceptrons of the decoder model is performed by processing the output vector.

15. processing the plurality of offset feature vectors and the query point feature vector through a cross-attention layer to generate an output vector; and concatenating the output vector with the query point feature vector to generate a concatenated vector; The method of claim 1 , wherein processing the plurality of offset feature vectors and the query point feature vector by the second set of multilayer perceptrons of the decoder model is performed by processing the concatenated vector.

16. A computer processor; and a non-transitory computer-readable medium for causing the computer processor to perform operations, the operations comprising: receiving a request for point attributes at a query point that matches the geographic location; and obtaining a query point feature vector from a feature map, the feature map encoding a geographic region that includes the geographic location, the operations further comprising: processing the query point feature vector with a first set of multi-layer perceptrons in a decoder model to generate a plurality of offsets; obtaining a plurality of offset feature vectors for the plurality of offsets from the feature map; processing the plurality of offset feature vectors and the query point feature vector with a second set of multilayer perceptrons in the decoder model to generate the point attributes; and responding to the request with the point attributes.

17. The feature map is a first axis and a second axis comprising a bird's-eye view of the geographical area; and a third axis comprising a set of features generated by encoding LiDAR data and map data for the geographic region.

18. The operation is processing the plurality of offset feature vectors and the query point feature vector through a cross-attention layer to generate an output vector; and concatenating the output vector with the query point feature vector to generate a concatenated vector; 17. The system of claim 16, wherein processing the plurality of offset feature vectors and the query point feature vector by the second set of multilayer perceptrons of the decoder model is performed by processing the concatenated vector.

19. computer readable program code for causing a computer system to perform operations, said operations comprising: receiving a request for point attributes at a query point that matches the geographic location; and obtaining a query point feature vector from a feature map, the feature map encoding a geographic region that includes the geographic location, the operations further comprising: processing the query point feature vector with a first set of multi-layer perceptrons in a decoder model to generate a plurality of offsets; obtaining a plurality of offset feature vectors for the plurality of offsets from the feature map; processing the plurality of offset feature vectors and the query point feature vector with a second set of multilayer perceptrons in the decoder model to generate the point attributes; and responding to the request with the point attributes.

20. The operation is processing the plurality of offset feature vectors and the query point feature vector through a cross-attention layer to generate an output vector; and concatenating the output vector with the query point feature vector to generate a concatenated vector; 20. The non-transitory computer-readable medium of claim 19, wherein processing the plurality of offset feature vectors and the query point feature vector by the second set of multilayer perceptrons of the decoder model is performed by processing the concatenated vector.

Images