Joint detection and prediction transformer

Abstract

Joint detection and prediction transformer includes functionality to perform operations that include obtaining a sensor data encoding of sensor data acquired from a sensor, initializing a spatiotemporal object structure, and obtaining a map encoding of a geographic region corresponding to the sensor data. The operations further include iteratively revising, by a transformer model, the spatiotemporal object structure using the sensor data encoding and the map encoding to generate a revised spatiotemporal object structure, decoding, by a decoder model, the spatiotemporal object structure to obtain location information and path information of at least one object in the geographic region, and outputting the location information and the path information.

Description

JOINT DETECTION AND PREDICTION TRANSFORMERCROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application is a non-provisional application of, and thereby claims benefit to U.S. Patent Application Serial Number 63 / 450,634 filed on March 7, 2023, which is incorporated herein by reference in its entirety.BACKGROUND

[0002] An autonomous system is a self-driving mode of transportation that does not require a human pilot or human driver to move in and react to the real-world environment. Rather, the autonomous system includes a virtual driver that is the decision-making portion of the autonomous system. Specifically, the virtual driver controls the actuation of the autonomous system. The virtual driver is an artificial intelligence system that learns how to interact in the real world and then performs the interaction when in the real world.

[0003] Part of interacting in the real world is detecting the location of objects and predicting the path motion of the objects in the environment. To navigate the real-world environment autonomously and more safely, predictions have to be not only accurate and generalize across many scenarios, but also made in a timely manner so that the autonomous system can react appropriately. Namely, existing techniques may compromise accuracy and generalization abilities as compared to computation needs.

[0004] Traditional systems separate detection and prediction as separate sequential tasks. The objects are first detected using the sensor data and then the path for each detected object is predicted.SUMMARY

[0005] In general, in one aspect, one or more embodiments is directed to a computer-implemented method that includes obtaining a sensor data encoding of sensor data acquired from a sensor, initializing a spatiotemporal object structure, and obtaining a map encoding of a geographic region correspondingto the sensor data. The method further includes iteratively revising, by a transformer model, the spatiotemporal object structure using the sensor data encoding and the map encoding to generate a revised spatiotemporal object structure. The method further includes decoding, by a decoder model, the spatiotemporal object structure to obtain location information and path information of at least one object in the geographic region. The method further includes outputting the location information and the path information.

[0006] In general, in one aspect, one or more embodiments is directed to a system that includes at least one computer processor, and memory for storing computer readable program code for causing the computer processor to perform operations. The operations include obtaining a sensor data encoding of sensor data acquired from a sensor, initializing a spatiotemporal object structure, and obtaining a map encoding of a geographic region corresponding to the sensor data. The operations further include iteratively revising, by a transformer model, the spatiotemporal object structure using the sensor data encoding and the map encoding to generate a revised spatiotemporal object structure. The operations further include decoding, by a decoder model, the spatiotemporal object structure to obtain location information and path information of at least one object in the geographic region. The operations further include outputting the location information and the path information.

[0007] In general, in one aspect, one or more embodiments is directed to a non- transitory computer readable medium comprising computer readable program code for causing a computer system to perform operations. The operations include obtaining a sensor data encoding of sensor data acquired from a sensor, initializing a spatiotemporal object structure, and obtaining a map encoding of a geographic region corresponding to the sensor data. The operations further include iteratively revising, by a transformer model, the spatiotemporal object structure using the sensor data encoding and the map encoding to generate a revised spatiotemporal object structure. The operations further include decoding, by a decoder model, the spatiotemporal object structure to obtainlocation information and path information of at least one object in the geographic region. The operations further include outputting the location information and the path information.

[0008] Other aspects of the invention will be apparent from the following description and the appended claims.BRIEF DESCRIPTION OF DRAWINGS

[0009] FIG. 1 shows an autonomous system with a virtual driver in accordance with one or more embodiments.

[0010] FIG. 2 shows a virtual driver in accordance with one or more embodiments.

[0011] FIG. 3 shows an example of a spatiotemporal object structure in accordance with one or more embodiments.

[0012] FIG. 4 shows an example implementation of a transformer model in accordance with one or more embodiments.

[0013] FIG. 5 shows a flowchart for joint detection and prediction in accordance with one or more embodiments.

[0014] FIG. 6 shows a flowchart for iteratively revising a spatiotemporal object structure in accordance with one or more embodiments.

[0015] FIG. 7 shows an example of joint detection and prediction using LiDAR in accordance with one or more embodiments.

[0016] FIG. 8A and FIG. 8B show a computing system in accordance with one or more embodiments of the invention.

[0017] Like elements in the various figures are denoted by like reference numerals for consistency.DETAILED DESCRIPTION

[0018] In general, embodiments are directed to joint detection and prediction of objects in a geographic region from sensor data and map data. A geographicregion includes objects and map elements. The objects and map elements each have a physical location in the geographic region that denote an exact spot or place in which the object or map element is located. The map elements are stationary in the geographic region and may be reflected in a map of the geographic region. The objects are located in the geographic region and are designed to move in the geographic region. For example, the objects may be people, bicyclists, autonomous systems, vehicles, unintelligent objects (e.g., debris moving through the geographic region), or another object that may move.

[0019] Detection is the identification of the exact physical locations of the objects in the geographic region. Here, exact is within an engineering threshold for the purposes of operating the autonomous system and may include a bounding box. Prediction is determining a future trajectory that is unknown as of the time of the capture of the sensor data. Joint detection and prediction involves creating a single unified encoding that may be concurrently decoded to perform the detection and to perform the prediction. A sensor data encoding of sensor data and a map encoding of map data is obtained. The sensor data encoding is an encoding of a current state of the geographic region as acquired by one or more sensors of an autonomous system. The map encoding is an encoding of a map of the geographic region. One or more embodiments include a transformer model that iteratively revises a spatiotemporal object structure using sensor data encoding and the map encoding to create revised spatiotemporal object structure that has the unified encoding. A decoder model than decodes the revised spatiotemporal object structure to obtain location information and path information of at least one object in the geographic region.

[0020] Turning to the Figures, FIG. 1 and FIG. 2 show example diagrams of the autonomous system and virtual driver. Turning to FIG. 1, an autonomous system (116) is a self-driving mode of transportation that does not require, but may use, a human pilot or human driver to move and react to the real-world environment. The autonomous system (116) may be completely autonomousor semi-autonomous. As a mode of transportation, the autonomous system (116) is contained in a housing configured to move through a real -world environment. Examples of autonomous systems include self-driving vehicles (e.g., self-driving trucks and cars), drones, airplanes, robots, etc.

[0021] The autonomous system (116) includes a virtual driver (102) which is the decision-making portion of the autonomous system (116). The virtual driver (102) is an artificial intelligence system that learns how to interact in the real world and interacts accordingly. The virtual driver (102) is the software executing on a processor that makes decisions and causes the autonomous system (116) to interact with the real-world including moving, signaling, and stopping or maintaining a current state. Specifically, the virtual driver (102) is decision making software that executes on hardware (not shown). The hardware may include a hardware processor, memory, or other storage device, and one or more interfaces. A hardware processor is any hardware processing unit that is configured to process computer readable program code and perform the operations set forth in the computer readable program code.

[0022] A real-world environment is the portion of the real world through which the autonomous system (116), when trained, is designed to move. Thus, the real-world environment may include concrete and land, construction, and other objects in a geographic region along with agents. The agents are the other objects in the real -world environment that are capable of moving through the real-world environment. Agents may have independent decision-making functionality. The independent decision-making functionality of the agent may dictate how the agent moves through the environment and may be based on visual or tactile cues from the real -world environment. For example, agents may include other autonomous and non-autonomous transportation systems (e.g., other vehicles, bicyclists, robots), pedestrians, animals, etc.

[0023] In the real world, the geographic region is an actual region within the real- world that surrounds the autonomous system. Namely, from the perspective ofthe virtual driver, the geographic region is the region through which the autonomous system moves.

[0024] The real-world environment changes as the autonomous system (116) moves through the real -world environment. For example, the geographic region may change, and the agents may move positions, including new agents being added and existing agents leaving.

[0025] In order to interact with the real-world environment, the autonomous system (116) includes various types of sensors (104), such as LiDAR sensors amongst other types, which are used to obtain measurements of the real-world environment, and cameras that capture images from the real-world environment. LiDAR (i.e., the acronym for “light detection and ranging”) is a sensing technique that uses light in the form of a pulsed laser to measure ranges (variable distances) to various objects in an environment. The autonomous system (116) may include other types of sensors as well. The sensors (104) provide input to the virtual driver (102).

[0026] In addition to sensors (104), the autonomous system (116) includes one or more actuators (108). An actuator is hardware and / or software that is configured to control one or more physical parts of the autonomous system based on a control signal from the virtual driver (102). In one or more embodiments, the control signal specifies an action for the autonomous system (e.g., turn on the blinker, apply breaks by a defined amount, apply accelerator by a defined amount, turn the steering wheel or tires by a defined amount, etc.). The actuator(s) (108) are configured to implement the action. In one or more embodiments, the control signal may specify a new state of the autonomous system and the actuator may be configured to implement the new state to cause the autonomous system to be in the new state. For example, the control signal may specify that the autonomous system should turn by a certain amount while accelerating at a predefined rate, while the actuator determines and causes the wheel movements and the amount of acceleration on the accelerator to achieve a certain amount of turn and acceleration rate.

[0027] Although not shown in FIG. 1, embodiments may be used in a virtual environment, such as when the virtual driver is being trained and / or tested in a simulated environment created by a simulator. In such a scenario, to the virtual driver, the simulated environment appears as the real-world environment.

[0028] FIG. 2 shows a virtual driver in accordance with one or more embodiments. The virtual driver (102) is operatively connected to sensors (104). The sensors (104) may include the various types of sensors discussed above with reference to FIG. 1. Sensors may be real or virtual sensors. Real sensors may be used, for example, on a physical autonomous system when the physical autonomous system is moving through a real world. Virtual sensors may be used for testing or training the virtual driver (102). When the sensors are virtual sensors, the interface between the virtual sensors and the virtual driver (102) match the interface between real sensors and the virtual driver in one or more embodiments.

[0029] The sensors (104) are configured to output sensor data (204) to the virtual driver (102). The sensor data (204) is data collected by the sensors of the geographic region around the autonomous system.

[0030] One type of sensor (104) is a LiDAR sensor. A LiDAR sensor is configured to emit pulsed light waves into the surrounding environment. The pulse light waves reflect off surrounding objects and return to the sensor. The sensor uses the time difference between sending pulse light wave and the pulse light wave returning to the LiDAR sensor to calculate the distance that the pulse light wave traveled. The LiDAR sensor may also record the direction and the intensity of the returned light wave. The light waves are transmitted as part of a LiDAR sweep. A LiDAR sweep is a capture of the geographic region by the LiDAR sensor along a predefined range of degrees around the LiDAR sensor.

[0031] The LiDAR sensors are configured to provide sensor data (204) in the form of a collection of LiDAR points. The LiDAR points may each have a distance, direction, and intensity. The distance and direction may be with respect to the LiDAR sensor. The LiDAR points may be translated into three-dimensional (3D) space. For example, each point in the list may be translated to being in the three-dimensional cartesian coordinate system with a value specifying an intensity. The LiDAR points when translated to 3D space is a LiDAR point cloud. Namely, a LiDAR point cloud is a single scan of a surrounding region that surrounds the LiDAR sensor. In other embodiments, the LiDAR point cloud may be a frame having multiple LiDAR sweeps of the sensing region, each sweep in the single point cloud being associated with a particular point in time.

[0032] Another type of sensor is a camera. Different types of cameras may be used. For example, the camera may be a visible-light camera, full-spectrum camera, a thermal imaging camera, or other type of camera. In such a scenario, the sensor data includes a set of one or more images from the camera.

[0033] The sensors (104) are communicatively connected to the virtual driver (102). The virtual driver (102) includes a detector predictor system (206). The detector predictor system (206) is a machine learning framework configured to perform joint detection of objects and prediction of the future paths of the objects. In joint detection and prediction, the detection of current location information of objects may be performed concurrently with the detection of the path information of the objects.

[0034] The detection is the estimated location information of the object at the time in which the sensor data is captured. In other words, the detection is the current location information. Location information of an object is information that identifies the particular location of the object. Specifically, the location information is the area in the geographic region in which the object is located. By way of an example, the location information may be a definition of a bounding box around the object, a centroid of the object, the surfaces of the object, or other position information around an object. The location information may be defined with respect to the autonomous system, the geographic region, or to another frame of reference. By way of an example, the location information may be the predicted location of the centroid of theobject as defined by a birds-eye-view (BEV) of the geographic region, and the predicted estimated height, width and length of the object. Other mechanisms for specifying the location of the object in the geographic region may be used. The location information may further include a heading of the object. Location information may be referred to as a pose of the object at a defined point in time.

[0035] The path information is the information that defines the path of the object through the geographic region in the future. The future is the time after the most recent capture of the sensor data from which the location information and path information is obtained. The path information specifies the predicted future trajectory of the object. The path information may be discretely defined as a series of location information at set of future timesteps. For example, the path information may be a sequence of poses. The path information may be continuously defined, such as a continuous trajectory defined by a line.

[0036] The sensor data encoder model (208) is a machine learning model that is configured to generate an encoding of the sensor data (204). In one or more embodiments, the encoding is an extraction of higher-level features from the sensor data. The encoding may be a vector embedding of the sensor data. In one or more embodiments, the encoding is not an explicit encoding of identified objects. For example, the encoding may encode pixel or voxel level features. The sensor data encoder model (208) may include a convolutional neural network, a graph neural network, a multilayer perceptron model, or other machine learning model. For example, for LiDAR, the sensor data encoder model (208) may be configured to voxelize one or more LiDAR point cloud and generate an encoding from the voxelized LiDAR point cloud.

[0037] The map encoder model (210) is a machine learning model that is configured to generate a map encoding of a map of the geographic region. Specifically, the encoder model (210) is configured to encode the map elements and the relationship between the map elements in the geographic region. Map elements are physical portions of the geographical region that may be reflected in a map of the geographic region. For example, the map elements may be acurb, a particular lane marker, a particular location between two lane markers or a lane marker and a curb, a light, a stop sign, or a construction zone, or other physical object / location in the geographic region. The map elements may or may not be demarcated in the real world. For example, if the map element is a particular spot in the real world that is between two lane markers, the particular spot exists in the real world and has a physical location, but the particular spot may not have any signposts or other markings in the real world that are at the particular spot. In one or more embodiments, a map of the geographic region directly or indirectly specifies the stationary locations of the map elements.

[0038] The map encoder model (210) is a software process configured to encode map elements of a geographic region as relative positions with respect to each other. Specifically, the map encoder model (210) may be configured to calculate, for each map element, the relative position of the map element with respect to other map elements in the geographic region. Thus, for each map element, a set of relative positions of the map element with respect to other map elements may be defined. In one or more embodiments, the relative positions are defined pairwise, between pairs of map elements. The map encoder model (210) may be further configured to encode the relative positions into a feature set for a pair of map elements. Additionally, a map element may have a feature set defining the map element. For example, the feature set may encode size information, the type of geographic region in which the map element is located (e.g., urban, rural, etc.), type of map element, and other features about the map element or the properties surrounding the map element.

[0039] The output of the map encoder model (210) may be map element encodings in a map graph. The map graph is a graph data structure having map element nodes connected by edges. The map element node is for an individual corresponding map element. The edges connecting two map element nodes may be associated with a relative position encoding of the corresponding pair of map element nodes. The map element node may be associated with a feature set that is generated based on general features of the map element.

[0040] In one or more embodiments, a spatiotemporal object structure (e.g., initial spatial temporal object structure (212), revised spatiotemporal object structure (214)) is a data structure that relates objects to spatiotemporal features for corresponding objects. Specifically, each object has corresponding spatiotemporal features in the spatiotemporal object structure. The spatiotemporal features may include, for each of multiple timesteps including a current time, a feature set of the object at the timestep. The feature set is a high-level feature set that has an object encoding for an object at the timestep. The feature set for the current time corresponds to detection of the object, and the feature set for the future timesteps corresponds to prediction of the object. A final feature set is for a goal location of an object. The goal location is the location in the geographic region to which the object is attempting to move. For example, the goal location may be a location five seconds into the future, whereas the predicted timesteps may be up to two seconds into the future. For example, the spatiotemporal object structure may be a three-dimensional vector structure with a first dimension corresponding to object and a second dimension corresponding to timestep. The value at the intersection of a particular object and a particular timestep in the three-dimensional vector structure is the feature set (i.e., the third dimension) for the particular object and particular timestep. An example of a spatiotemporal object structure is presented in FIG. 3.

[0041] The initial spatiotemporal object structure (212) is a spatiotemporal object structure that is prior to any revision. For example, the initial spatiotemporal object structure may have placeholders for corresponding sets of spatiotemporal features for a predefined number of objects. By way of a more specific example, the detector predictor system (206) may be configured to detect up to three hundred objects and for eleven future timesteps. In the example, the initial spatiotemporal object structure (212) may have individual placeholders for each combination of the three hundred objects and twelve timesteps. The values of the placeholders may be random, general regardless of the sensor data, or based on coarse detection. Coarse detection may optimize recall over precision. Coarse detection is detection of object location that hasmore than a threshold amount of error (e.g., more ghost objects are detected) or the probability of being accurate is less than a threshold.

[0042] The revised spatiotemporal object structure (214) is a spatiotemporal object structure that is after revision by the transformer model (216). With each iteration of the transformer model (216), the spatiotemporal features are more accurate estimations of spatial features at the various timesteps.

[0043] The transformer model (216) is a machine learning model that is configured to transform the initial spatiotemporal object structure (212) to the revised spatiotemporal object structure (214). Specifically, the transformer model (216) is configured to iteratively revise the spatiotemporal features using the sensor data encoding and the map encoding. An example of the transformer model is presented in FIG. 4.

[0044] Continuing with FIG. 1, the decoder model (218) is a machine learning model that is configured to perform the detection and the prediction from the revised spatiotemporal object structure. In one or more embodiments, the decoder model (218) includes a detection decoder (not shown) and a separate prediction decoder. The detection decoder may be a MLP that outputs centroid, yaw, length, width and confidence of the existence of the object. The prediction decoder may be a transformer model, a recurrent neural network, etc. The detection decoder (218) may be configured to output a predicted object type. In one or more embodiments, the object type is the type of physical object. For example, the type of object may be a vehicle, person, plane, ball, bicyclist, building, truck, or other type of object. The decoder model (218) may have a defined set of classes, each class being for a particular object type. In one or more embodiments, the decoder model (218) may include network layers that are configured to output a probability for each of at least a subset of classes for each object.

[0045] FIG. 3 shows an example of a spatiotemporal object structure (300) in accordance with one or more embodiments. By way of a specific example, the example spatiotemporal object structure may be a three-dimensional vectorstructure that is stored in heap memory when performing the joint detection and prediction. As shown in FIG. 3, the example spatiotemporal object structure (300) includes a first dimension for objects (302) and a second position for timesteps (304). The timesteps are the timesteps in the future that match the path of the object. Namely, the timesteps may not match iterations of the transformer model. More specifically, as the object moves in the geographic region over time, the continuous future time is discretized into a set of timesteps. The granularity of the discretization may be based on the location or type of autonomous system. For example, an autonomous system that is a robot operating or designed to operate in a sparsely populated region may have timesteps that are on the order of seconds. As another example, an autonomous system that is a vehicle designed to operate in an urban region may have timesteps that is on the order of milliseconds.

[0046] For a particular object (e.g., object 2), the spatiotemporal features include a first pair for the current time (e.g., object 2, time 0 (306)), whereby the current time is the time of the sensor data capture. The spatiotemporal features may further include multiple pairs for the future timesteps (e.g., object 2, time 1 (308) to object 2, time t (310)). The spatiotemporal features may further include a pair for the object and goal location (e.g., object 2, goal (312)).

[0047] As shown in FIG. 3, the example spatiotemporal object structure (300) has for each object and timestep, a pair that includes a feature set and anchor. The feature set is the feature set described above with reference to FIG. 3. The anchor is the predicted location of the object at the time. The anchor may also include a predicted heading of the object. For example, in BEV, the anchor may be the x, y coordinates predicting the location of the object. The x, y coordinates may be defined with respect to the boundaries or centroid of the geographic region. Other mechanisms for defining the location may be used without departing from the scope of the claims. Initially, the anchors may be the coarse prediction of the objects. As another example, the anchors may be randomized, based on a grid, or based on another initialization strategy. Thefeature set may be a learned initial set of features that is generally used across multiple geographic regions. In such a scenario, the feature set is not pretrained. The transformer model is configured to revise the anchor and feature set over one or more iterations.

[0048] FIG. 4 shows an example implementation of the transformer model (400) in accordance with one or more embodiments. The transformer model includes an object to map attention layer (402), one or more sensor type attention layers (e.g., sensor type X attention layer (404), sensor type Y attention layer (406)), an object self-attention layer (408), an object time attention layer (410), and an anchor update layer (412). Each of these layers is described below.

[0049] The term, “attention layer” corresponds to the standard definition used in the art of machine learning. An attention layer is a type of one or more neural network layers that is used to selective focus on different parts of data when processing the data. Generally, attention works by measuring the cosine similarity between two vectors or embeddings. The cosine similarity determines the importance of elements, which allows models to selectively focus on crucial inputs leading to informed predictions and decisions. For example, with the respective attention layers, the transformer model may apply greater weights to different parts of the map, sensor data, or spatiotemporal object structure, where the weights are determined by the attention mechanism.

[0050] The object to map attention layer (402) is configured to perform cross attention of the feature sets to the map element encodings. In one or more embodiments, the object to map attention layer is configured to process the spatiotemporal features for individual objects at individual timesteps in relation to the map without considering the set of objects and the set of timesteps. Stated another way, in relation to the example spatiotemporal object structure, the object to map attention layer may be configured to process each pair individually. In one or more embodiments, the object to map attention layer (402) is a k-nearest neighbor attention layer.

[0051] The one or more sensor type attention layers (e.g., sensor type X attention layer (404), sensor type Y attention layer (406)) may each be specific to a sensor type. In one or more embodiments, the sensor type attention layer is configured to process the spatiotemporal features for individual objects at individual timesteps in relation to the sensor data without considering the map, the set of objects and the set of timesteps. Stated another way, in relation to the example spatiotemporal object structure, the object to sensor data attention layer may be configured to process each pair individually. In one or more embodiments, the object to map attention layer (402) is a deformable attention layer.

[0052] The object self-attention layer (408) is configured to process the spatiotemporal features for individual objects at individual timesteps in relation to other objects at the same timestep without considering the map, the sensor data, or the set of timesteps. Stated another way, in relation to the example spatiotemporal object structure, the object self-attention layer may be configured to process a pair in relation to other pairs at the same timestep. Thus, the object self-attention layer (408) processes the predicted object with respect to other predicted objects. In one or more embodiments, the object selfattention compares pairs only at the same timestep.

[0053] The object time attention layer (410) is configured to process the spatiotemporal features for individual objects in relation to the same object at other timesteps without considering the map, the sensor data, or the other objects. Stated another way, in relation to the example spatiotemporal object structure, the object time attention layer may be configured to process a pair in relation to other pairs for the same predicted object. In one or more embodiments, the object self-attention compares pairs only for the same object.

[0054] The anchor update layer (412) is configured to revise the anchor for each of the pairs. The anchor update layer (412) may be configured to predict an offset from a current anchor position for the pair based on the feature set of thepair. In one or more embodiments, the anchor update model is a small multilayer perceptron (MLP) model have a set of neural network layers.

[0055] While FIG. 1, FIG. 2, FIG. 3, and FIG. 4 show a configuration of components, other configurations may be used without departing from the scope of the invention. For example, various components may be combined to create a single component. As another example, the functionality performed by a single component may be performed by two or more components.

[0056] FIG. 5 and FIG. 6 shows flowcharts in accordance with one or more embodiments. While the various steps in these flowcharts are presented and described sequentially, at least some of the steps may be executed in different orders, may be combined or omitted, and at least some of the steps may be executed in parallel. Furthermore, the steps may be performed actively or passively.

[0057] FIG. 5 shows a flowchart for joint detection and prediction in accordance with one or more embodiments.

[0058] In Block 502, sensor data for a geographic region from a sensor is obtained. For example, one or more cameras may take one or more pictures of the geographic region. Further, LiDAR sensors may perform one or more LiDAR sweeps. To perform a LiDAR sweep, the light pulse is transmitted by the LiDAR transmitter of the LiDAR sensor. The light pulse may reflect off an object, albeit unknown, in the environment, and the time and intensity for the reflected light pulse to return to the receiver is determined. The time is used to determine the distance to an object. The result is a LiDAR point in the LiDAR data. For the LiDAR sweep, multiple light pulses are transmitted at a variety of angles, and multiple reflected lights are received to obtain multiple points in the LiDAR point cloud. The LiDAR point cloud may be obtained as a list of LiDAR points. The list may be translated to a LiDAR point cloud.

[0059] Rather than using a real sensor, the simulator, using a sensor simulation model for the sensor, may generate simulated sensor data. Specifically, the simulator may generate a scene and render the scene. The scene is in a virtualgeographic region. Machine learning models that are part of the simulator may determine the intensity or color of the LiDAR points or pixels based on the location of the virtual sensor in the scene. The result is a simulated set of sensor data that mimics real sensor data for a particular simulated scene. The simulated sensor data may be of virtually any resolution. For example, the simulated sensor data may match the resolution of a real sensor.

[0060] In Block 504, a spatiotemporal object structure is initialized. Memory for the spatiotemporal object structure is reserved. Different techniques may be used to initialize the spatial feature sets within the spatiotemporal object structure.

[0061] For example, in a first technique, the values of the feature sets may be randomly defined. In such a scenario, random selection may be used to initialize the values. In another example, a learned set of values may be used for the spatial feature sets, whereby the learned set of values are generic across several geographic regions. The learned set of values may be learned through training the detector predictor system through the training process.

[0062] Further, the corresponding anchors related to the feature sets are initialized. In an example, the initialization may be performed by partitioning the geographic region into a two-dimensional birds-eye-view grid or a three- dimensional grid, and assuming a stationary object is in each grid cell. In another example, a coarse detector model may process the sensor data and generate a coarse detection of objects in the geographic region. The coarse detector is a lightweight detector that may return false positives. Thus, even objects having a probability of existing below a predefined threshold may still be identified by the coarse detector. For example, for LiDAR point clouds, the course detection model may be based on locations having a higher density of LiDAR points.

[0063] In Block 506, a map encoding of a geographic region corresponding to the sensor data is obtained. The map encoding is any encoding of the map of the geographic region. Generating a map encoding may be performed byapplying a neural network model to the map elements in the geographic region. The following is an example of a technique to generate a map encoding. Other techniques may be used with out departing from the scope of the invention.

[0064] In the example, physical locations of map elements are obtained from the map data. In some embodiments, one or more of the map elements are determined from the map data. For example, if the map data includes lane makers, the map elements may be a defined geographic spot between two lane markers. From the physical locations of the map elements, the relative positions of the map elements with respect to other map elements are calculated. Map element encodings are generated using the relative positions. The map element encodings encode the features of the map element (e.g., type of map element, speed limit at map element, and other features) and encode the relative position. A map encoding is generated using the map element encodings. Each map element corresponds to a map element node in the map encoding. The map element nodes are connected by edges based on adjacency between the map element nodes. The edge between two map element nodes is associated with a relative position between the two corresponding map elements to generate the map encoding. A graph neural network may be applied to the map layer to further update the map layer based on neighboring map elements. Thus, map element encodings in the map not only reflect features of itself in the map, but also information about other map elements.

[0065] In Block 508, a sensor encoding model generates a sensor data encoding of the sensor data. For camera, the sensor encoding model may include a convolutional neural network and other networks, which translate a series of camera images into a projected birds-eye-view of a region. As another example, for LiDAR sensors, a set of LiDAR points is acquired. Each LiDAR point has a location in a cartesian coordinate system and an intensity. In some embodiments, the LiDAR points may be processed through a first set of neural network layers to generate a first set of LiDAR point features. For example, for each LiDAR point, the value of the location in the three coordinates may beconcatenated together with the intensity. The result is an input feature set to the first set of neural network layers that generates an output feature set for each LiDAR point. The output feature set may be on a point level basis. For example, each LiDAR point may be associated with the first set of point features and the original location (e.g., in the cartesian coordinate system).

[0066] Each LiDAR point may be individually voxelized through the voxelization process. The geographic region is partitioned into a grid (i.e., voxel grid), whereby a one-to-one mapping may exist between the voxel in the voxel grid and non-overlapping subregions of the geographic region. During the voxelization process, the location in the geographic region of the LiDAR point is matched to the subregion of the geographic region that contains the location. The voxelization process relates the LiDAR point to the voxel mapped to the subregion. Further, an offset in the 3D space from the centroid of the voxel is identified. For the voxels in the voxel grid, each voxel may be associated with multiple sets of features from the different LiDAR points. The sets of features may be aggregated. For example, the aggregation may be to perform an averaging. The LiDAR voxels are encoded to obtain encoded voxels. In one or more embodiments, the voxel grid is processed through a convolutional neural network (CNN). The voxel grid when processed may have the three dimensions of the locations and the fourth dimension of the feature set for the voxel. The CNN updates the features of the voxels to account for the features of the surrounding voxels. Thus, the encoding updates the features of each voxel. In one or more embodiments, the size of the current voxel grid does not change when encoding the LiDAR voxels. Each voxel after the encoding is associated with a corresponding subregion of the geographic region. Additionally, the process may be repeated for a window of LiDAR point clouds, whereby the encodings in the voxel grid is updated with each subsequent point cloud (e.g., by aggregating the respective voxel encodings). Thus, the sensor data encoding may not only reflect the current sensor data, but also past sensor data.

[0067] In Block 510, a transformer model iteratively revises the spatiotemporal object structure using the sensor data encoding and the map encoding to generate a revised spatiotemporal object structure. The transformer model updates the spatiotemporal object structure with features from the map encoding, features from the sensor data encoding, features from other predicted objects in the region, and features from the same object over time. Further, the anchors may be updated. Through the iterative update, the spatiotemporal object structure becomes a more accurate representation of the geographic region including objects in the geographic region. Iteratively revising the object to timestep feature map is described in FIG. 6.

[0068] Continuing with FIG. 5, in Block 512, a detection decoder decodes the spatiotemporal object structure to obtain location information of at least one object in the geographic region. Objects that do not exist may be identified based on low confidence scores generated by the detection decoder model. In one or more embodiments, the detection decoder may operate on the initial timestep pair of feature set and anchor for each object individually. The detection decoder may be a MLP model that predicts an offset for the anchor from a current position of the anchor and the feature set. For example, the feature set may be concatenated with the anchor and then processed by the neural network of the MLP. The offset is then used to update the anchor to determine the current position of the object. Similarly, the detection decoder may generate a prediction of the dimensions of the object. For example, the detection decoder may output a length and width of the object.

[0069] In Block 514, a prediction decoder decodes the spatiotemporal object structure to obtain path information of the at least one object in the geographic region. The prediction decoder may operate in a same or similar manner as the detection decoder, but for each future timestep. In such a scenario, individually for each timestep and for each object, the feature set may be concatenated with the anchor and then processed by the MLP of the prediction decoder.

[0070] As another example of the processing by the prediction decoder, the goal and optionally the map may be used. For each timestep of an object, the goal location (in the spatiotemporal object structure) may be concatenated with the feature set and the anchor to obtain an input set. The prediction decoder may then output the object position (i.e., future way point for the timestep) into the future by processing the input set. The combination of future waypoints for an object forms the path information of the object in one or more embodiments.

[0071] In Block 516, the location information and the path information are outputted. The virtual driver of the autonomous system may use the trajectories of the agent and the virtual driver’s own path to its destination to determine a current trajectory of the autonomous system that satisfies safety criteria (e.g., avoiding collisions, having stopping distance, etc.) and other criteria (e.g., shortest path, reduced number of lane changes, etc.) and is in furtherance of the moving to the destination. The virtual driver may then output a control signal to one or more actuators. In the real -world environment, the control signal is used by an actuator that causes the autonomous system to perform an action, such as move in a particular direction at a particular speed or acceleration, wait, display a turn signal, or perform other action. In the simulated environment, the control signal is intercepted by a simulator that simulates the actuator and the resulting action of the autonomous system. The simulator simulates the autonomous system thereby training the virtual driver. Namely, the output of simulating the autonomous system in the simulated environment may be used to evaluate the actions of the virtual driver.

[0072] FIG. 6 shows a flowchart for iteratively revising a spatiotemporal object structure in accordance with one or more embodiments.

[0073] In Block 602, an object to map attention layer revises the spatiotemporal object structure. The object to map attention layer may individually process each pair in the spatiotemporal object structure. For a pair, the following operations may be performed. In one or more embodiments, the coordinates specified in the anchor are used to identify the map elements that areneighboring to the position specified by the coordinates. For example, the four adjacent map elements may be used. The heading in the anchor may be used to compute relative distances between the agents and map. Cross attention may then be performed between the map element encodings of the identified neighboring map elements to the feature set of the pair. The result of the object to map attention layer is to revise the feature set of the pair based on the map of the geographic region. The process may be repeated for each pair of spatiotemporal features. Other types of attention layers may be used to update the feature set based on the map element encodings.

[0074] In Block 604, a sensor type attention layer revises the spatiotemporal object structure. In one or more embodiments, each sensor type has a corresponding attention layer. Thus, Block 604 may be performed separately for each type of sensor. For example, a first sensor type attention layer may process LiDAR data, a second sensor type attention layer may process camera sensor data. The sensor type attention layers may process the pairs in series with other sensor type attention layers. For example, the feature set may be revised for a first sensor type and then revised for a second sensor type. To perform the sensor type attention layer, using the anchor of the pair, a set of offsets is determined. In one or more embodiments, a predefined number of offsets is used. Then, cross attention may be performed between sensor data encodings at each of the set of offset locations and the feature set of the pair. The result of the sensor type attention layer is to revise the feature set of the pair based on the sensor data of the sensor type.

[0075] In Block 606, an object self-attention layer revises the spatiotemporal object structure. The object self-attention layer considers other currently predicted objects in the region at the same timestep. In FIG. 3, the object selfattention layer performs the attention along the object dimension. The object self-attention layer has a number of different predicted objects attending to each other. For example, the number of different predicted objects may be four. Other numbers of predicted objects may be used without departing from thescope of the claims. Further, the objects that attend to each other may be based on adjacency of the objects according to respective anchors of the objects. The heading in the anchor may be used to compute relative distances between different objects. The result of the object self-attention layer is to revise the feature set of the pair based on other predicted objects in the region.

[0076] In Block 608, an object time attention layer revises the spatiotemporal object structure. The object time attention layer performs the update for an object based on past and future timesteps of the object. In FIG. 3, the object time attention layer performs the attention along the timestep dimension. The object time attention layer has the feature sets of the pairs for a single predicted object attend to each other. One of the pairs has a goal location for the object. Thus, the revision of the feature sets of the pairs also incorporates the predicted goal location for the object. The result of the object time attention layer is to revise the feature set of the pair based on other predicted objects in the region.

[0077] In Block 610, an anchor update layer revises the spatiotemporal object structure. In one or more embodiments, the anchor update layer processes the feature set concatenated with the anchor for a pair to generate an offset. The processing may be performed through a set of neural network layers of an MLP. The anchor is then revised according to the offset. In other words, the anchor is moved from the current position to the current position with the offset to create the new current position. The new current position then becomes the anchor for the next iteration or for the decoder. The process of the anchor update layer may be performed individually for each pair in the spatiotemporal feature structure.

[0078] In Block 612, a determination is made whether to perform another iteration. In one or more embodiments, the process is repeated until a set number of iterations is performed. In such embodiments, the determination is Block 612 may be whether the number of iterations is performed. If the number of iterations is not performed, the operations may repeat with Block 602 for the next set of iterations. If the number of iterations is performed, the processcontinues with FIG. 5. The processing may also stop before the set number of iterations complete, such as to satisfy latency constraints.

[0079] The processing of an iteration of the transformer model is to iteratively update the spatiotemporal object structure. The spatiotemporal object structure has placeholders for objects in one or more embodiments. Thus, even though positions exist for predicted objects, not all positions may correspond to actual objects. Prior to the first iteration, the predicted objects are inaccurate or do not exist. Further, the movement of the objects may be unknown. Through the updating process, the spatiotemporal object structure is iteratively updated to reflect a more accurate feature set and anchors for the objects.

[0080] Training of the transformer model may be performed by comparing the set of objects predicted by the decoder (i.e., detection or prediction decoder) for a particular timestep with actual locations of objects for a given map and given sensor data. The result is used to calculate a loss that is back propagated through the detector predictor system.

[0081] FIG. 7 shows an example of joint detection and prediction using LiDAR in accordance with one or more embodiments. The following example is for explanatory purposes only and not intended to limit the scope of the invention.

[0082] FIG. 7 shows the example of using raw LiDAR data (702) and high definitional maps (704) to predict path information in the form of a set of object trajectories T E RNobixFx(T+' )x3withobjects, F future hypothesis per object, and T future time steps for each trajectory. Each trajectory includes a set of waypoints| t E {0, ••• , ?}}. The waypoints denote the centroid (i.e., (t, yt)) and heading (i.e., (0t) ) of the object in birds-eye-view (BEV). Each predicted future waypoint may be related to a corresponding timestep t. The first timestep t = 0 is for object detection and is an estimation of the probability that the object exists c and the length and width in BEV (Z, w) of the bounding box of the object. In one or more embodiments, the object size is assumed to be constant over time.

[0083] The example architecture may include input encoders (e.g., LiDAR encoder (704), map encoder (708)), spatiotemporal object feature sets and anchors (710) in a spatiotemporal object structure, the transformer model (712), and the decoder model (714). The input encoders are configured to abstract heterogeneous sensory inputs, such as LiDAR and high definition (HD) maps, into feature sets and spatial anchors. The set of spatiotemporal object feature sets and anchors encoding objects’ latent attributes as well as current and future poses. The transformer model (712) includes the attention mechanisms are applied to update the object feature sets and anchors with information from the map encodings and the sensor data encodings. The decoder model (714) yields detection and prediction results (716) from the updated object feature sets (718).

[0084] In the example, at least one of the sensors is a LiDAR sensor. The LiDAR encoder (706) generates LiDAR feature sets from LiDAR sweeps (702). Multiple LiDAR sweeps (702) may be voxelized in BEV to generate voxelized point clouds. A convolutional neural network in the LiDAR encoder (706) may be applied to the voxelized point clouds to produce multi-resolution BEV feature maps. Each pixel in the BEV feature maps may be treated as a LiDAR feature set, which yields KUdarE RNudar*dwhereNUdardenotes the number of pixels in the multi-level feature maps and d is the embedding dimension.

[0085] The map tokenizer (708) generates map element encodings. In one or more embodiments, the map (708) may be preprocessed into lane graphs. A graph convolutional neural network may be used to extract map embeddings for each lane graph node. The embedding of each lane graph node as a map element encoding. Thus, a set of map element encodings may be defined as KmapE RNmap*dwhereNmis the number of lane graph nodes and d is the embedding dimension.

[0086] Spatiotemporal object feature sets may be used to jointly model the current and future states of an object. The spatiotemporal object feature sets are in the spatiotemporal object structure. The spatiotemporal object featuresets are Qobj 6 RNobj*(r+P)xd ,where t,epirstdimension Nobj represents the maximum number of objects that may be detected, and the second dimension T is the number of future timesteps to predict. The present-time feature sets Qohj model the objects’ current state in latent space and can be used for detection; the future time feature sets Qobj'"'Tcontain the latent representation of T can be used to decode multi-modal future trajectories for each object. One or more embodiments may learn the initial latent features of the object feature sets.

[0087] Each feature set may be associated with an anchor P, which defines the geometric position (%,y) of this feature set in BEV. For LiDAR feature sets, the anchors Puaar are defined as the center of the BEV pixels. For map element encodings, Pmapare the center of the lane graph nodes. For object feature sets, Pobj- simply match the centroid of the object, and are initialized from the output of an object detector header that re-uses the LiDAR feature sets.

[0088] The task of the Object Transformer (712) is to refine each object feature set and anchor by incorporating information from different sensory modalities as well as other object feature sets. One or more embodiments use a stack of L Transformer-style blocks. Each Object Transformer block contains several attention layers which update the object feature sets as follows:where LN ) denotes layer normalization anddenotes a specialized attention function with Q 2j as queries and any other feature sets as keys / values. The attention layers also encode the feature set anchors into relative positional embeddings and adds them to the relevant query, key and value features. Each attention function Atintegrates a different sensor modality or interaction, detailed in the following paragraphs.

[0089] The object to map attention (714) may be performed as follows. Object feature sets cross-attend to neighboring map feature sets to exploit the information in the lane graph. Since the map encoder is a graph convolutional network, each lane graph node embedding (or map element encoding) already encodes information about the neighborhood. Thus, the map element encodings that are far away from the object are not used. One or more embodiments may therefore limit the cross-attention to the k nearest map elemement encodings from the object anchor. This reduces the computation from 0(NobjNmap) to 0(Nobjk) , where k « Nmap. Limiting the attention to k-nearest map element encodings may greatly aid learning.

[0090] The object to LiDAR attention (716) may be performed as follows. In the object to LiDAR attention, the object feature sets attend to the LiDAR feature sets. Since the number of LiDAR feature sets Niidaris large and fully connected attention brings unaffordable memory consumption, one or more embodiments may use deformable attention to select I LiDAR feature sets near the anchor of each object feature set, which yields a query -to-LiD AR attention map of Nobj- X I where I « Nudar, significantly reducing the computational cost. The offsets used in deformable attention are predicted from the object queries QobJ.

[0091] The object self attention (718) and object time attention (720) may be performed as follows. For efficiency purposes, one or more embodiments factorized self-attention into object self-attention (718) and time self-attention (720). In object self-attention (718), each object feature set can only attend to other objects at the same timestep (the sequence dimension is size Nobj- and T + 1 is put in the batch dimension). This allows the object feature sets to gather contextual information about neighboring traffic participants, which helps predict consistent behaviors (e.g., avoid collisions between different objects’ trajectories). In time self-attention (720), each object can only attend to itself across the time dimension (the sequence dimension is of size T + 1 andNObj is put in the batch dimension), propagating information across time and enabling our decoder to predict smoother trajectories

[0092] The anchor update (722) may be performed as follows. At the end of each Object Transformer block, the anchors are refined: Pbl+j1-)=where APobj = MLP Qobj') is the correction offset to the previous anchors.

[0093] The object feature sets at current time Qofbare fed into decoders (724) that may include a detection decoder and a prediction decoder. The detection decoder produces object bounding boxes, and object feature sets at the future time Qobj' ""TThe prediction decoder generates multi-modal future trajectories.

[0094] The detection decoder may operate as follows. The final object bounding boxes (%, y, I, w, 9) are simply defined as the updated object anchors Pobj after all Object Transformer blocks. For confidence scores, the present time object feature setsare passed to a MLP that predicts categorical confidence scores RN°bjxcfor each object query and each class, where c denotes the number of classes.

[0095] The prediction decoder may operate as follows. To perform multiple future forecasting, a more complex decoder may be used for prediction. One or more embodiments may first predict the probability distribution over a set of goals for each object, sample F goals from the probability distribution, and finally complete the objects’ future trajectories based on the goal.

[0096] Specifically, the final timestep object feature setsareconcatenated to map element encodings Kmapand fed to an MLP to obtain confidence scores on which map node is the most likely goal, as well as estimate offset between the map goal and the actual last trajectory point. Based on the confidence scores, for each query, one or more embodiments sample F map nodes as goals to which the respective goal offsets are added.

[0097] To complete the future trajectories, each of these goal coordinates is concatenated to the future time object feature sets Q^j and passed to an MLPto produce F different waypoint hypothesis for a given time step. The heading at each future time step is obtained via finite differences taking into account the initial detection heading and future waypoints.

[0098] The multi-task objective is a sum of the detection loss Ldet, the prediction loss Lpred. For the detection task, one or more embodiments sum a sigmoid focal loss Ldcfor query classification with an LI loss Ldeand an loU loss Ld^ to regress the parameters of bounding boxes. To calculate the targets for these losses, one or more embodiments first match object bounding boxes B E RN x5to the ground truth bounding boxes B' E RN X5through bipartite matching. The same detection loss may be applied at the end of each Transformer encoder block to supervise the object anchor updates.

[0099] For prediction loss, one or more embodiments define the objective as a sum of a focal loss Lprce^ applied to the map nodes to classify which node on the map is the goal, a Huber loss Lprreegto regress the respective goal offsets, and another Huber loss Lptdto regress the trajectory waypoints conditioned on the ground truth goal. Note that the predicted trajectories T E RNobixFxTx2are matched with the ground truth trajectories T' Eif the predicted boxes have high loUs with the ground truth boxes. The result of the prediction loss may also be backpropagated through the network.

[0100] Embodiments may be implemented on a computing system specifically designed to achieve an improved technological result. When implemented in a computing system, the features and elements of the disclosure provide a significant technological advancement over computing systems that do not implement the features and elements of the disclosure. Any combination of mobile, desktop, server, router, switch, embedded device, or other types of hardware may be improved by including the features and elements described in the disclosure. For example, as shown in FIG. 8A, the computing system (800) may include one or more computer processors (802), non-persistent storage (804), persistent storage (806), a communication interface (808) (e.g.,Bluetooth interface, infrared interface, network interface, optical interface, etc.), and numerous other elements and functionalities that implement the features and elements of the disclosure. The computer processor(s) (802) may be an integrated circuit for processing instructions. The computer processor(s) may be one or more cores or micro-cores of a processor. The computer processor(s) (802) includes one or more processors. The one or more processors may include a central processing unit (CPU), a graphics processing unit (GPU), a tensor processing unit (TPU), combinations thereof, etc.

[0101] The input devices (810) may include a touchscreen, keyboard, mouse, microphone, touchpad, electronic pen, or any other type of input device. The input devices (810) may receive inputs from a user that are responsive to data and messages presented by the output devices (812). The inputs may include text input, audio input, video input, etc., which may be processed and transmitted by the computing system (800) in accordance with the disclosure. The communication interface (808) may include an integrated circuit for connecting the computing system (800) to a network (not shown) (e.g., a local area network (TAN), a wide area network (WAN) such as the Internet, mobile network, or any other type of network) and / or to another device, such as another computing device.

[0102] Further, the output devices (812) may include a display device, a printer, external storage, or any other output device. One or more of the output devices may be the same or different from the input device(s). The input and output device(s) may be locally or remotely connected to the computer processor(s) (802). Many different types of computing systems exist, and the aforementioned input and output device(s) may take other forms. The output devices (812) may display data and messages that are transmitted and received by the computing system (800). The data and messages may include text, audio, video, etc., and include the data and messages described above in the other figures of the disclosure.

[0103] Software instructions in the form of computer readable program code to perform embodiments may be stored, in whole or in part, temporarily or permanently, on a non-transitory computer readable medium such as a CD, DVD, storage device, a diskette, a tape, flash memory, physical memory, or any other computer readable storage medium. Specifically, the software instructions may correspond to computer readable program code that, when executed by a processor(s), is configured to perform one or more embodiments, which may include transmitting, receiving, presenting, and displaying data and messages described in the other figures of the disclosure.

[0104] The computing system (800) in FIG. 8A may be connected to or be a part of a network. For example, as shown in FIG. 8B, the network (820) may include multiple nodes (e.g., node X (822), node Y (824)). Each node may correspond to a computing system, such as the computing system shown in FIG. 8A, or a group of nodes combined may correspond to the computing system shown in FIG. 8A. By way of an example, embodiments may be implemented on a node of a distributed system that is connected to other nodes. By way of another example, embodiments may be implemented on a distributed computing system having multiple nodes, where each portion may be located on a different node within the distributed computing system. Further, one or more elements of the aforementioned computing system (800) may be located at a remote location and connected to the other elements over a network.

[0105] The nodes (e.g., node X (822), node Y (824)) in the network (820) may be configured to provide services for a client device (826), including receiving requests and transmitting responses to the client device (826). For example, the nodes may be part of a cloud computing system. The client device (826) may be a computing system, such as the computing system shown in FIG. 8A. Further, the client device (826) may include and / or perform all or a portion of one or more embodiments.

[0106] The computing system of FIG. 8A may include functionality to present raw and / or processed data, such as results of comparisons and other processing.For example, presenting data may be accomplished through various presenting methods. Specifically, data may be presented by being displayed in a user interface, transmitted to a different computing system, and stored. The user interface may include a graphical user interface (GUI) that displays information on a display device. The GUI may include various GUI widgets that organize what data is shown as well as how data is presented to a user. Furthermore, the GUI may present data directly to the user, e.g., data presented as actual data values through text, or rendered by the computing device into a visual representation of the data, such as through visualizing a data model.

[0107] As used herein, the term “connected to” contemplates multiple meanings. A connection may be direct or indirect (e.g., through another component or network). A connection may be wired or wireless. A connection may be temporary, permanent, or semi-permanent communication channel between two entities.

[0108] The various descriptions of the figures may be combined and may include or be included within the features described in the other figures of the application. The various elements, systems, components, and steps shown in the figures may be omitted, repeated, combined, and / or altered as shown from the figures. Accordingly, the scope of the present disclosure should not be considered limited to the specific arrangements shown in the figures.

[0109] In the application, ordinal numbers (e.g., first, second, third, etc.) may be used as an adjective for an element (i.e., any noun in the application). The use of ordinal numbers is not to imply or create any particular ordering of the elements nor to limit any element to being only a single element unless expressly disclosed, such as by the use of the terms "before", "after", "single", and other such terminology. Rather, the use of ordinal numbers is to distinguish between the elements. By way of an example, a first element is distinct from a second element, and the first element may encompass more than one element and succeed (or precede) the second element in an ordering of elements.

[0110] Further, unless expressly stated otherwise, or is an “inclusive or” and, as such includes “and.” Further, items joined by an or may include any combination of the items with any number of each item unless expressly stated otherwise.

[0111] In the above description, numerous specific details are set forth in order to provide a more thorough understanding of the disclosure. However, it will be apparent to one of ordinary skill in the art that the technology may be practiced without these specific details. In other instances, well-known features have not been described in detail to avoid unnecessarily complicating the description. Further, other embodiments not explicitly described above can be devised which do not depart from the scope of the claims as disclosed herein. Accordingly, the scope should be limited only by the attached claims.

Claims

CLAIMSWhat is claimed is:

1. A computer-implemented method comprising: obtaining a sensor data encoding of sensor data acquired from a sensor; initializing a spatiotemporal object structure; obtaining a map encoding of a geographic region corresponding to the sensor data; iteratively revising, by a transformer model, the spatiotemporal object structure using the sensor data encoding and the map encoding to generate a revised spatiotemporal object structure; decoding, by a decoder model, the spatiotemporal object structure to obtain location information and path information of at least one object in the geographic region; and outputting the location information and the path information.

2. The computer-implemented method of claim 1, further comprising: controlling an autonomous system comprising the sensor according to the location information and the path information.

3. The computer-implemented method of claim 1, further comprising: revising, by an object to map attention layer, the spatiotemporal object structure; revising, by a sensor type attention layer, the spatiotemporal object structure; revising, by an object self-attention layer, the spatiotemporal object structure; and revising, by an object time attention layer, the spatiotemporal object structure.

4. The computer-implemented method of claim 1, wherein: the spatiotemporal object structure comprises for an object of the at least one object, a plurality of pairs for a plurality of timesteps, each pair of the plurality of pairs comprises a feature set and an anchor for the object and a timestep of the plurality of timesteps, wherein the anchor comprises a current predicted position of the object at the timestep.

5. The computer-implemented method of claim 4, further comprising: revising, by an anchor update layer, the anchor of the pair of the plurality of pairs using the feature set of the pair.

6. The computer-implemented method of claim 4, further comprising: revising, by an object to map attention layer, the spatiotemporal object structure, wherein the object to map attention layer performs, according to the anchor of a pair of the plurality of pairs, a cross attention of the corresponding feature set to a plurality of map element encodings.

7. The computer-implemented method of claim 1, wherein the obtaining the sensor data comprises: obtaining a plurality of LiDAR sweeps of the geographic region; voxelizing the plurality of LiDAR sweeps into a plurality of voxelized point clouds; and executing a convolutional neural network on the plurality of voxelized point clouds to obtain a plurality of feature maps.

8. The computer-implemented method of claim 1, wherein the decoder model comprises a detection decoder and a prediction decoder, and wherein the method further comprises: decoding, by the detection decoder, the spatiotemporal object structure to obtain the location information of the at least one object in the geographic region; and decoding, by the prediction decoder, the spatiotemporal object structure to obtain the path information of the at least one object in the geographic region.

9. The computer-implemented method of claim 1, further comprising: decoding the spatiotemporal object structure to predict a type of the at least one object.

10. The computer-implemented method of claim 1, further comprising: partitioning the geographic region into a grid; andinitializing the spatiotemporal object structure to have an object in each of a plurality of grid cells of the grid.

11. The computer-implemented method of claim 1, further comprising: processing, by a coarse detector model, the geographic region to identify a plurality of predicted object locations in the geographic region; and initializing the spatiotemporal object structure with the plurality of predicted object locations.

12. The computer-implemented method of claim 1, further comprising: obtaining the sensor data for the geographic region from a sensor; and generating, by a sensor encoding model, a sensor data encoding of the sensor data.

13. A system comprising: at least one computer processor; and memory for storing computer readable program code for causing the at least one computer processor to perform operations comprising: obtaining a sensor data encoding of sensor data acquired from a sensor; initializing a spatiotemporal object structure; obtaining a map encoding of a geographic region corresponding to the sensor data; iteratively revising, by a transformer model, the spatiotemporal object structure using the sensor data encoding and the map encoding to generate a revised spatiotemporal object structure; decoding, by a decoder model, the spatiotemporal object structure to obtain location information and path information of at least one object in the geographic region; and outputting the location information and the path information.

14. The system of claim 13, wherein the operations further comprise: controlling an autonomous system comprising the sensor according to the location information and the path information.

15. The system of claim 13, wherein the operations further comprise: revising, by an object to map attention layer, the spatiotemporal object structure; revising, by a sensor type attention layer, the spatiotemporal object structure; revising, by an object self-attention layer, the spatiotemporal object structure; and revising, by an object time attention layer, the spatiotemporal object structure.

16. The system of claim 13, wherein: the spatiotemporal object structure comprises for an object of the at least one object, a plurality of pairs for a plurality of timesteps, each pair of the plurality of pairs comprises a feature set and an anchor for the object and a timestep of the plurality of timesteps, wherein the anchor comprises a current predicted position of the object at the timestep.

17. The system of claim 13, wherein the obtaining the sensor data comprises: obtaining a plurality of LiDAR sweeps of the geographic region; voxelizing the plurality of LiDAR sweeps into a plurality of voxelized point clouds; and executing a convolutional neural network on the plurality of voxelized point clouds to obtain a plurality of feature maps.

18. The system of claim 13, wherein the decoder model comprises a detection decoder and a prediction decoder, and wherein the operations further comprise: decoding, by a detection decoder, the spatiotemporal object structure to obtain location information of the at least one object in the geographic region; and decoding, by a prediction decoder, the spatiotemporal object structure to obtain path information of the at least one object in the geographic region.

19. A non-transitory computer readable medium comprising computer readable program code for causing a computer system to perform operations comprising: obtaining a sensor data encoding of sensor data acquired from a sensor; initializing a spatiotemporal object structure;obtaining a map encoding of a geographic region corresponding to the sensor data; iteratively revising, by a transformer model, the spatiotemporal object structure using the sensor data encoding and the map encoding to generate a revised spatiotemporal object structure; decoding, by a decoder model, the spatiotemporal object structure to obtain location information and path information of at least one object in the geographic region; and outputting the location information and the path information.

20. The non-transitory computer readable medium of claim 19, wherein the operations further comprise: revising, by an object to map attention layer, the spatiotemporal object structure; revising, by a sensor type attention layer, the spatiotemporal object structure; revising, by an object self-attention layer, the spatiotemporal object structure; and revising, by an object time attention layer, the spatiotemporal object structure.

Images