Generating Lane Segments Using Embeddings for Automated Vehicle Navigation

Abstract

Presented herein are systems and methods for generating a path for autonomously navigating through an environment. A computing system can identify a tensor including an encoding derived from sensor data from an ego and map data defining a topology of an environment surrounding the ego. The computing system can determine a first index value defining a point in a first grid by applying at least a first portion of the encoding to a machine learning (ML) model. The computing system can determine a second index value defining a point in a second grid within the first grid by applying at least a second portion of the encoding and the first index value to the ML model. The computing system can generate tokens for the path through the environment based on the first index value and the second index value at the point. The computing system can store a graph to include the tokens.

Description

[Technical Field]

[0001] [Cross-reference to related patent applications] This application claims priority to U.S. Provisional Application No. 63 / 377,954, filed September 30, 2022, which is incorporated by reference herein in its entirety for all purposes.

[0002] The present disclosure generally relates to artificial intelligence-based modeling techniques that analyze image data and predict occupancy attributes of an ego's surroundings. [Background technology]

[0003] Due to rapid advances in computer technology, autonomous navigation techniques used for autonomous vehicles and robots (collectively, egos) have become widespread. These advances enable safer and more reliable autonomous navigation for egos. Egos often need to navigate through complex and dynamic environments and terrains that may include vehicles, traffic, pedestrians, cyclists, and various other static or dynamic obstacles. To navigate such complex and dynamic environments, a computing system in the ego can perform path planning through the environment using sensor data about the surrounding environment. The path planning can identify a trajectory or lane along which the ego is to traverse through the environment. Understanding the ego's surroundings is necessary for making informed and appropriate decisions to avoid collisions and successfully execute the path plan. However, techniques for analyzing and processing sensor data can be too extensive and time-consuming to effectively navigate the ego through the environment. Summary of the Invention

[0004] To facilitate autonomous navigation of an ego (e.g., a vehicle or robot) through an environment, the ego's computing system may be configured with a trained artificial intelligence (AI) or machine learning (ML) model. The ML model may take input from a set of embeddings that encode a low-dimensional representation of sensor data (e.g., from a video of the surrounding environment) and map data (e.g., a navigation map of the environment). Using the input, the ML model may be used to generate a graph with a set of tokens that specify linguistic representations of potential lane segments in the environment through which the ego may navigate. The graph may correspond to a sparse set of lane segments and their connectivity specified using coefficients (e.g., spline coefficients).

[0005] Aspects of the present disclosure relate to systems, methods, devices, apparatuses, and non-transitory computer-readable media for generating paths for autonomously navigating through an environment. One or more processors can identify a tensor including a plurality of encodings derived from sensor data from an ego and map data defining a topology of an environment surrounding the ego. The one or more processors can determine a first index value defining a point within a first plurality of points in a first grid defined over the environment by applying at least a first portion of the plurality of encodings to a machine learning (ML) model. The one or more processors can determine a second index value defining a point within a second plurality of points in a second grid within a subset of the first plurality of points in the first grid by applying at least a second portion of the plurality of encodings and the first index value to the ML model. The one or more processors can generate tokens for at least one of the plurality of paths through the environment based on the first index value and the second index value at the point. The one or more processors can store a graph to include tokens to be used to autonomously navigate the ego through the environment via one or more of a plurality of paths.

[0006] In one embodiment, the one or more processors can classify the second point as a continuation topology type dependent on the first point by applying at least a third portion of the plurality of encodings and a third index value at the second point to an ML model. In response to classifying the second point as a continuation topology type, the one or more processors can determine a plurality of spline coefficients that define one path among a plurality of paths between the first point and the second point through the environment. The one or more processors can generate a second token based on the third index value at the second point and the continuation topology type. The one or more processors can update the graph to include the second token and the plurality of spline coefficients to be used to autonomously navigate the ego through the environment.

[0007] In another embodiment, the one or more processors can classify the second point as a branching topology type from the point to the third point by applying at least a third portion of the plurality of encodings and a third index value at the second point to an ML model. In response to classifying the second point as a terminal topology type, the one or more processors can determine a path defined by the first point and the second point through the environment. The one or more processors can generate a second token based on the third index value at the second point and the terminal topology type. The one or more processors can update the graph to include the second token to be used to autonomously navigate the ego through the environment.

[0008] In yet another embodiment, the one or more processors can identify a second graph including a plurality of tokens to be used to autonomously navigate the second ego through the environment via one or more of the second plurality of paths. The one or more processors can use the graph and the second graph to determine that at least one first path of the plurality of paths in the ego intersects with at least one second path of the plurality of paths in the second ego. The one or more processors can perform an action with respect to at least one of the ego or the second ego in response to determining that the at least one first path intersects with the second path.

[0009] In yet another embodiment, the one or more processors can use sensor data from the ego to identify the presence of a stationary second ego in the environment. The one or more processors can use the graph to determine that at least one first path of the plurality of paths in the ego intersects with the stationary second ego. In yet another embodiment, the one or more processors can classify the point as a topology type indicative of the beginning of at least one of the plurality of paths by applying at least a third portion of the plurality of encodings and the second index value to an ML model. The one or more processors can generate a token for at least one of the plurality of paths through the environment based on the topology type.

[0010] In yet another embodiment, the one or more processors can use the plurality of tokens of the graph to determine a trajectory defining the ego's navigation via one of a plurality of paths through the environment. In yet another embodiment, the one or more processors can present, via a graphical user interface (GUI), a graph defining a plurality of paths for a topology of the environment surrounding the ego. In yet another embodiment, the one or more processors can generate tokens using (i) a first embedding generated from a first index value, (ii) a second embedding generated from a second index value, and (iii) one or more embeddings associated with the points. [Brief explanation of the drawings]

[0011] Non-limiting embodiments of the present disclosure are described by way of example with reference to the accompanying drawings, which are schematic and not intended to be drawn to scale, and unless indicated as representing background art, the figures represent aspects of the present disclosure.

[0012] [Figure 1A] 1 illustrates components of an AI-enabled visual data analysis system, according to an exemplary embodiment.

[0013] [Figure 1B] 10 illustrates various sensors associated with an ego according to an exemplary embodiment.

[0014] [Figure 1C] 1 illustrates components of a vehicle according to an exemplary embodiment.

[0015] [Figure 2] FIG. 1 illustrates a block diagram of a system for generating a tensor representation of a path for autonomously navigating through an environment in accordance with an illustrative embodiment.

[0016] [Figure 3A]FIG. 1 is an illustration of a block diagram of a process for generating a path for autonomously navigating through an environment in accordance with an illustrative embodiment; [Figure 3B] FIG. 1 is an illustration of a block diagram of a process for generating a path for autonomously navigating through an environment in accordance with an illustrative embodiment; [Figure 3C] FIG. 1 is an illustration of a block diagram of a process for generating a path for autonomously navigating through an environment in accordance with an illustrative embodiment; [Figure 3D] FIG. 1 is an illustration of a block diagram of a process for generating a path for autonomously navigating through an environment in accordance with an illustrative embodiment; [Figure 3E] FIG. 1 is an illustration of a block diagram of a process for generating a path for autonomously navigating through an environment in accordance with an illustrative embodiment; [Figure 3F] FIG. 1 is an illustration of a block diagram of a process for generating a path for autonomously navigating through an environment in accordance with an illustrative embodiment; [Figure 3G] FIG. 1 is an illustration of a block diagram of a process for generating a path for autonomously navigating through an environment in accordance with an illustrative embodiment; [Figure 3H] FIG. 1 is an illustration of a block diagram of a process for generating a path for autonomously navigating through an environment in accordance with an illustrative embodiment;

[0017] [Figure 4] 1 illustrates a diagram of a scenario in which a first ego uses a graph representing a path to autonomously navigate through an environment, according to an exemplary embodiment.

[0018] [Figure 5] Illustrates a scenario in which a first ego uses a graph to detect when a path it is traversing is about to intersect with a path a second ego is about to traverse.

[0019] [Figure 6] 1 illustrates a diagram of a scenario in which a first ego uses a graph to detect the presence of a stationary second ego in an environment, according to an exemplary embodiment.

[0020] [Figure 7] FIG. 1 illustrates a flow diagram of a method for generating a path for autonomously navigating through an environment in accordance with an illustrative embodiment; DETAILED DESCRIPTION OF THE INVENTION

[0021] Reference will now be made to the exemplary embodiments illustrated in the drawings, and specific language will be used herein to describe the embodiments. It will nevertheless be understood that no limitation of the scope of the claims or this disclosure is intended thereby. Alterations and further modifications of the features shown herein, and further applications of the principles of the subject matter shown herein that will occur to those skilled in the art in possession of this disclosure, are to be considered within the scope of the subject matter disclosed herein. Other embodiments may be used and / or other changes may be made without departing from the spirit or scope of the present disclosure. The exemplary embodiments described in the detailed description are not meant to limit the presented subject matter.

[0022] The ego may be an autonomous vehicle (e.g., a car, truck, bus, motorcycle, all-terrain vehicle, cart), robot, or other automated device. The ego may use one or more artificial intelligence (AI) algorithms or machine learning (ML) models to autonomously navigate the ego through an environment. To facilitate autonomous navigation, the ego may use a lane detection algorithm to recognize lane segments on roads in the environment as the ego traverses. For example, the ego may acquire sensor data (e.g., light detection and ranging (LiDAR) and optical imagery) and apply an image segmentation model to detect lines along the road surface and recognize lane segments on the road. However, this approach may be limited to detecting lane segments from several different types of geometric shapes, such as a single lane and its adjacent lanes along a road, with minimal ability to detect branching and merging. As a result, the image segmentation model may constrain the ego's autonomous navigation to highly structured environments, such as highways with fewer lanes. Furthermore, it may be difficult, if not impossible, for Ego to rely on this type of model to navigate autonomously through more complex environments, such as intersections on local roads.

[0023] To address these and other technical constraints, a computing system at the ego may be configured with a set of AI algorithms and ML models to generate a graph that defines a set of lane segments (sometimes referred to herein as paths) and connectivity. To that end, the computing system at the ego may acquire sensor data (e.g., optical camera images and LiDAR) and map data (e.g., a navigation map) of the ego's surroundings. Using a first set of encoders, the computing system may generate a set of tensors (or embeddings) as reduced-dimensional representations of the sensor data. The computing system at the ego may enrich the set of tensors by applying a second set of encoders on the map data to add tensors to embed values ​​related to topology and road layout. The resulting set of tensors may be a rich and dense representation of the ego's surroundings.

[0024] Using the output set of tensors, the computing system at ego can apply a third set of encoders (e.g., autoregressive decoders) to generate a set of tokens in the graph. Each token can specify various characteristics of points that form one or more of the lane segments through the environment. In applying the encoders to the set of tensors, the computing system can determine a first index value to specify the location of the specified point within the coarse grid. Using the first index value, the computing system can calculate a second index value to specify the location of the point within a finer grid. The specification of the points within the grid allows the computing system to classify the points by topology type, such as start point, continuation, branch, or terminal point, among others.

[0025] Subsequently, if a point is linked to another point (e.g., classified as a continuation or branch), the computing system can also identify an index value for the referenced point. The computing system can also calculate spline coefficients that define the connectivity between the two points to define corresponding lane segments through the environment. A third set of encoders allows the computing system to generate embeddings to represent the index values ​​and topology types, and then combine the embeddings to form tokens for insertion into the graph. By repeating this process, the ego's computing system can generate a set of tokens in the graph to define a linguistic representation of potential lane segments in the environment through which the ego can navigate. Using the lane segments defined by the graph, the ego can navigate autonomously through the environment.

[0026] FIG. 1A is a non-limiting example of system components capable of implementing the methods and systems discussed herein. For example, an analytics server may train an AI model and use the trained AI model to generate occupancy datasets and / or maps for one or more egos. FIG. 1A illustrates components of an AI-enabled visual data analytics system 100. System 100 may include analytics server 110a, system database 110b, administrator computing device 120, egos 140a-b (collectively ego 140), ego computing devices 141a-c (collectively ego computing devices 141), and server 160. System 100 is not limited to the components described herein and may include additional or other components not shown for the sake of brevity, which components should be considered within the scope of the embodiments described herein.

[0027] The components referred to herein may be connected via a network 130. Examples of network 130 may include, but are not limited to, a private or public LAN, a WLAN, a MAN, a WAN, and the Internet. Network 130 may include wired and / or wireless communications according to one or more standards and / or over one or more transport media.

[0028] Communications over network 130 may occur according to various communication protocols, such as Transmission Control Protocol / Internet Protocol (TCP / IP), User Datagram Protocol (UDP), IEEE communications protocols, etc. In one example, network 130 may include wireless communications according to the Bluetooth® set of specifications or another standard or proprietary wireless communications protocol. In another example, network 130 may also include communications over cellular networks, including, for example, Global System for Mobile Communications (GSM), Code Division Multiple Access (CDMA), or Enhanced Data for Global Evolution (EDGE) networks.

[0029] System 100 illustrates an example of a system architecture and components that may be used to train and execute one or more AI models, such as AI model 110c. Specifically, as depicted in FIG. 1A and described herein, analytics server 110a may train AI model 110c using data retrieved from ego 140 (e.g., by using data streams 172 and 174) using methods described herein. Once AI model 110c is trained, each ego 140 may access the trained AI model 110c and execute it. For example, vehicle 141a with ego computing device 140a may send its camera feed to trained AI model 110c and generate a graph (e.g., data stream 174) that defines lane segments in the environment. Furthermore, data captured and / or predicted by AI model 110c for ego 140 (during inference) may be used to improve AI model 110c. Thus, system 100 implements a continuous loop that can periodically improve the accuracy of AI model 110c. Furthermore, system 100 implements a loop that can use data received by ego 140 in a learning phase in addition to an inference phase.

[0030] Analytics server 110a may be configured to collect, process, and analyze navigation data (e.g., images captured while navigating) and various sensor data collected from ego 140. The collected data may then be processed and prepared into a training dataset. The training dataset may then be used to train one or more AI models, such as AI model 110c. Analytics server 110a may also be configured to collect visual data from ego 140. Using AI model 110c (trained using the methods and systems discussed herein), analytics server 110a may generate a dataset and / or an occupancy map for ego 140. Analytics server 110a may display the occupancy map on ego 140 and / or transmit the occupancy map / dataset to ego computing device 141, administrator computing device 120, and / or server 160.

[0031] Although FIG. 1A shows AI model 110c as a component of system database 110b, AI model 110c may be stored in a different or separate component, such as cloud storage or any other data repository accessible to analytics server 110a.

[0032] The analytics server 110a may also be configured to display an electronic platform showing various training attributes for training the AI ​​model 110c. The electronic platform may be displayed on the administrator computing device 120 to allow an analyst to monitor the training of the AI ​​model 110c. An example of an electronic platform generated and hosted by the analytics server 110a may be a web-based application or website configured to display the training dataset collected from the ego 140 and / or the training status / metrics of the AI ​​model 110c.

[0033] Analysis server 110a may be any computing device equipped with a processor and non-transitory machine-readable storage capable of performing the various tasks and processes described herein. Non-limiting examples of such computing devices may include workstation computers, laptop computers, server computers, etc. Although system 100 includes a single analysis server 110a, system 100 may include any number of computing devices operating in a distributed computing environment, such as a cloud environment.

[0034] Ego 140 may represent various electronic data sources that transmit data associated with a previous or current navigation session to analytics server 110a. Ego 140 may be any device configured for navigation, such as vehicle 140a and / or truck 140c. Ego 140 is not limited to being a vehicle and may include robotic devices as well. For example, ego 140 may include robot 140b, which may represent a general-purpose, bipedal, autonomous humanoid robot capable of navigating various terrains. Robot 140b may be provided with software that enables balance, navigation, perception, or interaction with the physical world. Robot 140b may also include various cameras configured to transmit visual data to analytics server 110a.

[0035] Although referred to herein as “ego,” ego 140 may or may not be an autonomous device configured for automatic navigation. For example, in some embodiments, ego 140 may be controlled by a human operator or by a remote processor. ego 140 may include various sensors, such as those depicted in FIG. 1B . The sensors may be configured to collect data as ego 140 navigates various terrains (e.g., roads). Analytics server 110a may collect data provided by ego 140. For example, analytics server 110a may obtain navigation session and / or road / terrain data (e.g., images of ego 140 navigating roads) from various sensors, such that the collected data is ultimately used by AI model 110c for training purposes.

[0036] As used herein, a navigation session corresponds to a trip in which ego 140 travels a route, regardless of whether the trip was autonomous or controlled by a human. In some embodiments, the navigation session may be for data collection and model training purposes. However, in some other embodiments, ego 140 may refer to a vehicle purchased by a consumer, and the purpose of the trip may be classified as daily use. A navigation session may begin when ego 140 travels more than a threshold distance (e.g., 0.1 miles, 100 feet) from a non-moving location or exceeds a threshold speed (e.g., greater than 0 mph, greater than 1 mph, greater than 5 mph). A navigation session may end when ego 140 is returned to a non-moving location and / or turned off (e.g., when the driver exits the vehicle).

[0037] Ego 140 may correspond to a group of egos monitored by analytics server 110a to train AI model 110c. For example, drivers of vehicles 140a may authorize analytics server 110a to monitor data associated with their respective vehicles. As a result, analytics server 110a may collect sensor / camera data using various methods discussed herein and generate a training dataset for training AI model 110c accordingly. Analytics server 110a may then apply the trained AI model 110c to analyze data associated with ego 140 and predict an occupancy map for ego 140. Furthermore, additional / ongoing data associated with ego 140 may be processed and added to the training dataset, allowing analytics server 110a to recalibrate AI model 110c accordingly. Thus, system 100 performs a loop in which navigation data received from ego 140 can be used to train AI model 110c. Ego 140 may include a processor that executes the trained AI model 110c for navigation purposes. During navigation, ego 140 can collect additional data about their navigation session and can use the additional data to calibrate AI model 110c. That is, ego 140 corresponds to an ego that can be used to train, run / use, and recalibrate AI model 110c. In a non-limiting example, ego 140 corresponds to a vehicle purchased by a customer that can navigate autonomously using AI model 110c and simultaneously improve AI model 110c.

[0038] Egos 140 may be equipped with various technologies that enable them to gather data from their surroundings and (potentially) navigate autonomously. For example, ego 140 may be equipped with an inference chip for running self-driving software.

[0039] Various sensors for each ego 140 may monitor and transmit collected data associated with different navigation sessions to analytics server 110a. FIGS. 1B-1C show block diagrams of sensors incorporated within ego 140, according to one embodiment. The number and location of each sensor discussed with respect to FIGS. 1B-1C may depend on the type of ego discussed in FIG. 1A. For example, robot 140b may include different sensors than vehicle 140a or truck 140c. For example, robot 140b may not include airbag activation sensor 170q. Additionally, the sensors for vehicle 140a and truck 140c may be in different locations than those shown in FIG. 1C.

[0040] As discussed herein, various sensors incorporated within each ego 140 may be configured to measure various data associated with each navigation session. Analytics server 110a may periodically collect the data monitored and collected by these sensors, which is processed according to methods described herein and used to train and / or execute AI model 110c to generate an occupancy map.

[0041] Ego 140 may include user interface 170a. User interface 170a may refer to the user interface of an ego computing device (e.g., ego computing device 141 of FIG. 1A). User interface 170a may be implemented as a display screen integrated with or coupled to a vehicle's interior, a head-up display, a touchscreen, etc. User interface 170a may include input devices such as a touchscreen, knobs, buttons, a keyboard, a mouse, a gesture sensor, a steering wheel, etc. In various embodiments, user interface 170a may be adapted to provide user input (e.g., as types of signals and / or sensor information) to other devices or sensors of ego 140 (e.g., the sensors shown in FIG. 1B), such as controller 170c.

[0042] User interface 170a may also be implemented with one or more logic devices that may be adapted to execute instructions, such as software instructions, that implement any of the various processes and / or methods described herein. For example, user interface 170a may be adapted to form a communication link, send and / or receive communications (e.g., sensor signals, control signals, sensor information, user input, and / or other information), or perform various other processes and / or methods. In another example, a driver may use user interface 170a to control the temperature of ego 140 or activate its features (e.g., autonomous driving or steering system 170o). Accordingly, user interface 170a may monitor and collect driving session data in conjunction with other sensors described herein. User interface 170a may also be configured to display various data generated / predicted by analytics server 110a and / or AI model 110c.

[0043] Orientation sensor 170b may be implemented as one or more of a compass, float, accelerometer, and / or other digital or analog device capable of measuring the orientation of ego 140 (e.g., the magnitude and direction of roll, pitch, and / or yaw relative to one or more reference orientations, such as gravity and / or magnetic north). Orientation sensor 170b may be adapted to provide orientation measurements at ego 140. In other embodiments, orientation sensor 170b may be adapted to provide roll, pitch, and / or yaw rate of ego 140 using a time series of orientation measurements. Orientation sensor 170b may be positioned and / or adapted to provide orientation measurements relative to a particular coordinate frame of ego 140.

[0044] Controller 170c may be implemented as any suitable logic device (e.g., a processing device, microcontroller, processor, application specific integrated circuit (ASIC), field programmable gate array (FPGA), memory storage device, memory reader, or other device or combination of devices) that can be adapted to execute, store, and / or receive appropriate instructions, such as software instructions that implement control loops for controlling various operations of ego 140. Such software instructions may also process sensor signals, determine sensor information, provide user feedback (e.g., via user interface 170a), interrogate devices regarding operating parameters, select operating parameters for devices, or implement methods for performing any of the various operations described herein.

[0045] Communications module 170e may be implemented as any wired and / or wireless interface configured to communicate sensor data, configuration data, parameters, and / or other data and / or signals to any feature shown in FIG. 1A (e.g., analytics server 110a). As described herein, in some embodiments, communications module 170e may be implemented in a distributed manner, such that portions of communications module 170e are implemented within one or more elements and sensors shown in FIG. 1B. In some embodiments, communications module 170e may delay communication of sensor data. For example, when ego 140 does not have network connectivity, communications module 170e may store sensor data in temporary data storage and transmit the sensor data when ego 140 is identified as having adequate network connectivity.

[0046] Speed ​​sensor 170d may be implemented as an electronic pitot tube, a metering gear or wheel, a water speed sensor, a wind speed sensor, a wind speed sensor (e.g., direction and magnitude), and / or other device capable of measuring or determining the linear velocity of ego 140 (e.g., within the surrounding medium and / or aligned with the longitudinal axis of ego 140) and providing such measurement as a sensor signal that can be communicated to various devices.

[0047] Gyroscope / accelerometer 170f may be implemented as an electronic sextant, a semiconductor device, an integrated chip, an accelerometer sensor, or other system or device capable of measuring angular velocity / acceleration and / or linear acceleration (e.g., direction and magnitude) of ego 140 and providing such measurements as sensor signals that can be communicated to other devices, such as analytics server 110a. Gyroscope / accelerometer 170f may be positioned and / or adapted to make such measurements relative to a particular coordinate frame of ego 140. In various embodiments, gyroscope / accelerometer 170f may be mounted in a common housing and / or module with other elements shown in FIG. 1B to ensure a common frame of reference or known transformations between frames of reference.

[0048] Global Navigation Satellite System (GNSS) 170h may be implemented as a global positioning satellite receiver and / or another device capable of determining the absolute and / or relative position of ego 140 based on, for example, radio signals received from space-born and / or terrestrial sources and providing such measurements as sensor signals that can be communicated to various devices. In some embodiments, GNSS 170h may be adapted to determine the velocity, speed, and / or yaw rate of ego 140 (e.g., using a time series of position measurements), such as the absolute velocity and / or yaw component of the angular velocity of ego 140.

[0049] Temperature sensor 170i may be implemented as a thermistor, an electrical sensor, an electrical thermometer, and / or other device capable of measuring a temperature associated with ego 140 and providing such measurement as a sensor signal. Temperature sensor 170i may be configured to measure an environmental temperature associated with ego 140, such as a cockpit or dash temperature, which may be used to estimate the temperature of one or more elements of ego 140.

[0050] Humidity sensor 170j may be implemented as a relative humidity sensor, an electrical sensor, an electrical relative humidity sensor, and / or another device capable of measuring the relative humidity associated with ego 140 and providing such measurement as a sensor signal.

[0051] Steering sensor 170g may be adapted to physically adjust the orientation of ego 140 according to one or more control signals provided by a logic device, such as controller 170c, and / or user input. Steering sensor 170g may include one or more actuators and control surfaces of ego 140 (e.g., rudder or other type of steering or trim mechanism) and may be adapted to physically adjust the control surfaces to various positive and / or negative steering angles / positions. Steering sensor 170g may also be adapted to sense the current steering angles / positions of such steering mechanisms and provide such measurements.

[0052] Propulsion system 170k may be implemented as a propeller, turbine, or other thrust-based propulsion system, a mechanical wheeled and / or tracked propulsion system, a wind / sail-based propulsion system, and / or other type of propulsion system that may be used to provide motive power to ego 140. Propulsion system 170k may also monitor the direction of motive power and / or thrust of ego 140 relative to a coordinate frame of reference of ego 140. In some embodiments, propulsion system 170k may be coupled to and / or integrated with steering sensor 170g.

[0053] Occupant restraint sensor 170l may monitor the seat belt detection and lock / unlock assembly, as well as other occupant restraint subsystems. Occupant restraint sensor 170l may include various environmental and / or status sensors, actuators, and / or other devices that facilitate operation of safety mechanisms associated with operation of ego 140. For example, occupant restraint sensor 170l may be configured to receive movement and / or status data from other sensors shown in FIG. 1B. Occupant restraint sensor 170l may determine whether a safety mechanism (e.g., a seat belt) is engaged.

[0054] Camera 170m may refer to one or more cameras integrated into ego 140, as depicted in FIG. 1C, or may include multiple cameras integrated into (or retrofitted to) ego 140. Camera 170m may be an inward-facing or outward-facing camera of ego 140. For example, as depicted in FIG. 1C, ego 140 may include one or more inward-facing cameras that can monitor and collect video of occupants of ego 140. Ego 140 may include eight outward-facing cameras. For example, ego 140 may include front camera 170m-1, forward-looking side camera 170m-2, forward-looking side camera 170m-3, rearward-looking side camera 170m-4 on each front fender, camera 170m-5 on each side (e.g., integrated into the B-pillar), and rear camera 170m-6.

[0055] 1B, radar 170n and ultrasonic sensor 170p may be configured to monitor the distance of ego 140 to other objects, such as other vehicles or immovable objects (e.g., trees or garage doors). Ego 140 may also include an autonomous driving or steering system 170o configured to autonomously navigate ego 140 using data collected via various sensors (e.g., radar 170n, speed sensor 170d, and / or ultrasonic sensor 170p).

[0056] Thus, automated driving or steering system 170o may analyze various data collected by one or more sensors described herein to identify driving data. For example, automated driving or steering system 170o may calculate the risk of a forward collision based on ego 140's speed and its distance to another vehicle on the road. Autonomous driving or steering system 170o may also determine whether the driver is touching the steering wheel. Autonomous driving or steering system 170o may transmit the analyzed data to various features discussed herein, such as an analytics server.

[0057] Airbag deployment sensor 170q may predict or detect a crash and cause one or more airbags to deploy or inflate. Airbag deployment sensor 170q may transmit data regarding the deployment of the airbags, including data associated with the event that caused the deployment.

[0058] 1A , administrator computing device 120 may represent a computing device operated by a system administrator. Administrator computing device 120 may be configured to display data retrieved or generated by analytics server 110a (e.g., various analytics metrics and risk scores), allowing the system administrator to monitor various models utilized by analytics server 110a, review feedback, and / or facilitate training of AI models 110c maintained by analytics server 110a.

[0059] Ego 140 may be any device configured to navigate various routes, such as vehicle 140a or robot 140b. As discussed with respect to FIGS. 1B-1C, ego 140 may include various telemetry sensors. Ego 140 may also include ego computing device 141. Specifically, each ego may have its own ego computing device 141. For example, truck 140c may have ego computing device 141c. For simplicity, ego computing devices are collectively referred to as ego computing device 141. Ego computing device 141 may control the presentation of content on ego 140's infotainment system, process commands related to the infotainment system, aggregate sensor data, manage communication of data to electronic data sources, receive updates, and / or send messages. In one configuration, ego computing device 141 communicates with an electronic control unit. In another configuration, ego computing device 141 is an electronic control unit. Ego computing device 141 may include a processor and non-transitory machine-readable storage media capable of performing the various tasks and processes described herein. For example, AI model 110c described herein may be stored and executed (or directly accessed) by ego computing device 141. Non-limiting examples of ego computing device 141 may include a vehicle multimedia and / or display system.

[0060] In one example of training AI model 110c, analytics server 110a can collect data from ego 140 to train AI model 110c. Before executing AI model 110c to generate or predict graphs defining lane segments, analytics server 110a may train AI model 110c using various methods. Training enables AI model 110c to incorporate data from one or more cameras of one or more ego 140 (without having to receive radar data) and predict occupancy data in the ego's surroundings. The operations described in this example may be performed by any number of computing devices (e.g., processors of ego 140) operating in the distributed computing system described in FIGS. 1A and 1B.

[0061] To train the AI ​​model 110c, the analytics server 110a may first use one or more egos 140 to drive a particular route. While driving, the egos 140 may generate navigation session data using one or more of their sensors (including one or more cameras). For example, one or more egos 140 equipped with various sensors may navigate a specified route. As the one or more egos 140 traverse the terrain, their sensors may capture continuous (or periodic) data about their surroundings. The sensors may indicate occupancy around the one or more egos 140. For example, the sensor data may indicate various objects having mass around the one or more egos 140 as they navigate their route.

[0062] In operation, as one or more egos 140 navigate, their sensors collect data and transmit the data to analytics server 110a, as depicted in data stream 172. In some embodiments, one or more egos 140 may include one or more high-resolution cameras that capture a continuous stream of visual data from around one or more egos 140 as they navigate through a route. Analytics server 110a may then use the camera feeds to generate a second dataset, with visual elements / representations of different voxels around one or more egos 140 included in the second dataset. In operation, as one or more egos 140 navigate, their cameras collect data and transmit the data to analytics server 110a, as depicted in data stream 172. For example, ego computing device 141 may send image data to analytics server 110a using data stream 172.

[0063] Analytics server 110a may generate a training dataset using data collected from ego 140 (e.g., camera feeds received from ego 140). The training dataset may identify or include a set of examples. Each example may identify or include input data and expected output data from the input data. In each example, the input may include collected data such as sensor data (e.g., video or images from one or more cameras) and map data (e.g., a navigation map) from ego 140. The output may include, among other things, environmental features (e.g., attributes collected from sensor data), map features (e.g., attributes in a navigation map such as topology features and road layout), classifications (e.g., types of topology), and output tokens (e.g., combinations of environmental features, map features, and classifications) to be included in a graph defining lane segments. In some embodiments, the output may be created by a human reviewer examining the input data.

[0064] Using the training dataset, the analytics server 110a may feed the set of training datasets to the AI ​​model 110c and obtain a set of predicted outputs (e.g., environment features, map features, classification, and output tokens). The analytics server 110a may then train the AI ​​model 110c by comparing the predicted data with ground truth data to determine a difference and adjusting the internal weights and parameters of the AI ​​model 110c proportional to the determined difference according to a loss function. The analytics server 110a may train the AI ​​model 110c in a similar manner until the predictions of the trained AI model 110c closely meet a particular threshold (e.g., recall or precision).

[0065] In some embodiments, the analytics server 110a may use supervised training methods. For example, using ground truth and received visual data, the AI ​​model 110c may train itself to predict outputs. As a result, once trained, the AI ​​model 110c may receive sensor data and map data, analyze the received data, and generate tokens. In some embodiments, the analytics server 110a may use unsupervised methods in which the training dataset is unlabeled. Because labeling the data in a training dataset can be time-consuming and require excessive computing power, the analytics server 110a may utilize unsupervised training techniques to train the AI ​​model 110c.

[0066] Upon establishment of AI model 110c, analytics server 110a can transmit, send, or distribute weights for AI model 110c to each ego computing device 141a-c. Upon receipt, ego computing devices 141a-c can store and maintain AI model 110c in local storage. Once stored and loaded, ego computing devices 141a-141c can use the weights in processing newly acquired data (e.g., sensor and map data) to create graphs for defining lane segments to autonomously navigate each ego 140a-140c through the environment. From time to time, analytics server 110a can transmit, send, or distribute updated weights for AI model 110c to update the instances of AI model 110c on ego computing devices 141a-c.

[0067] Referring now to FIG. 2, a block diagram of the architecture of system 200 for generating a tensor representation of a path for autonomously navigating through an environment is depicted. System 200 may be implemented using any of the components described herein, such as ego computing devices 141a-c and AI model 110c. System 200 may include the components and steps described herein. However, other embodiments may include additional or alternative components and steps, or may omit one or more components and steps. System 200 and its architecture may be implemented by an analysis server (e.g., a computer similar to analysis server 110a) or an ego computing device (e.g., ego computing devices 141a-c) or across multiple computing systems. However, one or more steps of system 200 may be performed by any number of computing devices operating in the distributed computing system described in FIGS. 1A-1C (e.g., processors of ego 140 and / or ego computing device 141). For example, one or more computing devices of an ego may locally implement some or all of the components and steps described in FIG. 2.

[0068] System 200 may include at least one vision component 205 for processing sensor data. Vision component 205 may include a set of cameras, such as at least one main camera 202A, at least one left or right pillar camera 202B, and at least one backup camera 202C, among others. The set of cameras 202A-C (generally referred to herein as cameras 202) may be instances of camera 170m and, as discussed herein, may be multiple cameras integrated into (or retrofitted to) ego 140. Each camera 202 may acquire and collect images (e.g., optical visual images) of the environment surrounding ego 140. The images may take the form of a set of frames for video acquired by camera 202. In some embodiments, vision component 205 may include other sensors, such as radar 170n and ultrasonic sensor 170p, as discussed herein. In some embodiments, vision component 205 may rely solely on optical images captured via optical camera 202.

[0069] The vision component 205 may include artificial intelligence (AI) algorithms or machine learning (ML) models for processing sensor data from a set of cameras and other sensors. The ML models may be trained and established as discussed herein. In general, the ML models of the vision component 205 may generate a set of embeddings corresponding to a reduced-dimensional representation of the sensor data. The ML models in the vision component 205 may include, for example, among others, a set of self-regulating networks (RegNets) 210A-C (hereinafter generally referred to as residual networks 210), a set of feature pyramid networks (FPNs) 215A-C (hereinafter generally referred to as feature pyramid networks 215), at least one transformer 220, and at least one video module 225.

[0070] Each self-regulating network 210 may include, among other things, a set of weights arranged according to a set of convolutional recurrent neural networks (RNNs) (e.g., including a set of long short-term memory networks (LSTMs) or gated recurrent units (GRNs)) with operators (e.g., connecters and activation functions). Each self-regulating network 210 may search, identify, or receive sensor data from a corresponding camera 202 (or other sensor). Upon receipt, the self-regulating network 210 may process the sensor data according to the set of weights to generate a set of embeddings (e.g., feature maps). The set of embeddings may be a reduced-dimensional representation of the sensor data, particularly with spatiotemporal features. The self-regulating network 210 may provide the output set of embedding transfers to a respective feature pyramid network 215.

[0071] Each feature pyramid network 215 can include a set of weights arranged according to a set of convolutional neural networks (CNNs) at multiple scales to detect features in the input data. Each feature pyramid network 215 can retrieve, identify, or receive an output set of embeddings from the corresponding self-regulating network 210. Upon receipt, the feature pyramid network 215 can process the set of embeddings from the self-regulating network 210 using the set of weights to generate another set of embeddings. The set of embeddings can be a further reduced-dimensional representation of the sensor data. Furthermore, the transformer 220 can include a set of weights arranged according to a transformer architecture with a multi-head attention mechanism. The feature pyramid network 215 can provide an output set of embedding transfers to the transformer 220.

[0072] The transformer 220 can receive, collect, or aggregate the set of embeddings generated by the feature pyramid network 215 and further generated by the self-regulating network 210 from the sensor data. The transformer 220 can process the set of embeddings according to the set of weights to generate another set of embeddings (or output tokens). The video module 225 can receive, collect, or aggregate the set of embeddings output by the transformer 220. Upon receipt, the video module 225 can combine the embedding sets corresponding to sensor data acquired over a defined period of time. This combination allows the video module 225 to generate an aggregated set of embeddings to feed forward. The resulting set of embeddings can represent or define a reduced-dimensional representation of the sensor data acquired via the camera 202 (or other sensor).

[0073] System 200 may include at least one map component 230 for processing map data, such as a navigation map 235. Navigation map 235 may include or identify data that defines a map or topology of the environment surrounding the ego, such as terrain type, elevation, or semantic labels of roads (e.g., navigable routes, unpaved paths, streets, roads, highways, bus lanes, lane counts, and ramps) or other features (e.g., geometric shapes, buildings, signage, and flora) by coordinates within the environment (e.g., geographic positioning system (GPS) coordinates). Map component 230 may search, obtain, or retrieve a navigation map 235 for the ego's location within the environment. For example, map component 230 may retrieve a navigation map 235 of a defined size (e.g., 2 km x 2 km) along the ego's direction of travel.

[0074] The map component 230 may include at least one lane guidance module 240 for augmenting a set of embeddings from the vision component 205 with a set of embeddings derived from a navigation map 235. The lane guidance module 240 may include, among other things, a set of weights arranged according to an encoder (e.g., a set of convolutional neural networks (CNNs)). The lane guidance module 240 may search, obtain, or identify a navigation map 235 that defines a topology surrounding the ego. Upon identification, the lane guidance module 240 may use the set of weights to process data from the navigation map 235 to generate a set of embeddings. The set of embeddings may be a low-dimensional representation of relevant features in the navigation map 235. In some embodiments, the lane guidance module 240 may process data from the navigation map 235 using the set of embeddings from the lane component 205. The lane guidance module 240 may combine the set of embeddings derived from the sensor data with the set of embeddings derived from the navigation map 235 to produce, output, or generate at least one tensor 250. Tensor 250 may include an aggregated set of embeddings derived from both sensor data and map data.

[0075] The system 200 may include at least one lane language component 255 for processing the tensor 250 from the map component 230. The lane language component 255 may include at least one lane encoder 260. The lane encoder 260 may include, among other things, a set of weights arranged according to an encoder or decoder, such as an autoregressive decoder. The autoregressive decoder of the lane encoder 260 may include a set of models for processing portions of an input embedding from the tensor 250 to generate an output embedding that depends on another output embedding derived from a previous portion of the input embedding. From processing the tensor 250, the lane encoder 260 may produce, output, or generate a set of lane instances 265 and at least one adjacency matrix 270. The lane instances 265 may define one or more lane segments through which an ego can potentially navigate an environment. The adjacency matrix 270 may define the connectivity or relationships between the lane segments defined in the lane segments 265. The lane instances 265 and the adjacency matrix 270 can be collectively referred to as a lane graph or language. Further details regarding the functionality of the lane language component 255 and the lane encoder 260 are provided herein in conjunction with Figures 3A-3H.

[0076] 3A-3H, block diagrams of a process 300 for generating a path for autonomously navigating through an environment are depicted. Process 300 may be implemented using any of the components described herein, such as ego computing devices 141a-c and AI model 110c. Process 300 may include the steps described herein. However, other embodiments may include additional or alternative steps or omit one or more steps. Process 300 may be performed by an analysis server (e.g., a computer similar to analysis server 110a) or by an ego computing device (e.g., ego computing devices 141a-c). However, one or more steps of process 300 may be performed by any number of computing devices operating in the distributed computing system described in FIGS. 1A-1C (e.g., processors of ego 140 and / or ego computing device 141, or a centralized service such as analysis server 110a). For example, one or more computing devices of an ego may locally perform some or all of the steps described in FIGS. 3A-H.

[0077] Starting with FIG. 3A , under process 300, a computing system can retrieve, receive, or identify at least one vector space encoding 302 (e.g., tensor 250) to be used to form or generate at least one graph 304. The computing system executing process 300 may correspond to ego 140 or ego computing device 141 on a centralized service, such as analytics server 110a. Vector space encoding 302 (sometimes referred to herein as a tensor) can identify or include a set of encodings (sometimes referred to herein as a set of embeddings or feature maps). The set of encodings for vector space encoding 302 can be generated, determined, or derived from sensor data (e.g., data from cameras and other sensors) acquired by the ego and map data (e.g., navigation map 235) that defines the topology of the environment surrounding the ego. The encodings can be low-dimensional representations of latent features derived from the sensor data and map data. The graph 304 can be used to define a set of paths (sometimes referred to herein as lane segments) along which the ego can autonomously navigate the environment.

[0078] The identification enables the computing system to locate, determine, or identify at least one point 306A within a set of grid points within a first grid 308 defined over the environment represented by map 310. Point 306A may correspond to a starting point from which the ego navigates through the environment defined by map 310. Grid 308 may specify or define a set of grid points (or coordinates) at a coarser or lower resolution than the original resolution of the grid points defined by map 310. Map 310 may correspond to a layout of the topology surrounding the ego and may be obtained as part of map data (e.g., navigation map 235).

[0079] The computing system may input, provide, or apply at least a portion of the vector space encoding 302 to a first point prediction unit 310. The portion of the vector space encoding 302 input to the first point prediction unit 310 may correspond to a point 306A defined within a set of grid points in a first grid 308 in the environment. In some embodiments, the computing system may identify or select the portion of the vector space encoding 302 to input based on the point 306A. The first point prediction unit 310 may correspond to a portion of a machine learning (ML) model (e.g., the lane encoder 260 discussed herein) and may include, among other things, a set of weights arranged according to cross-attention, self-attention, and a transformer. In providing, the computing system may process the portion of the vector space encoding 302 according to the weights of the first point prediction unit 310 to calculate, determine, or generate at least one coordinate 330A corresponding to the point 306A. Using this point, the computing system may use the first point prediction unit 310 to calculate, generate, or determine a first index value 330'A at the point 306A.

[0080] 3B, a computing system may locate, determine, or identify at least one point 306A within a set of grid points within a second grid 308′ defined over the environment represented by map 312. The computing system may correspond to ego computing device 141 or a centralized service, such as analytics server 110a, in ego 140. Point 306A may correspond to a starting point from which the ego navigates through the environment defined by map 312. Second grid 308′ may specify or define a set of grid points (or coordinates) at a finer or higher resolution than the resolution of first grid 308. Second grid 308′ may correspond to a subset of first grid 308 that surrounds or is around the ego. Map 312 may correspond to a layout of a topology surrounding the ego and may be obtained as part of map data (e.g., navigation map 235).

[0081] The computing system may input, provide, or apply at least a portion of the vector space encoding 302 and the output of the first point prediction unit 310 (e.g., an embedding derived from the first index value 330′A) to the second point prediction unit 312. The portion of the vector space encoding 302 may correspond to a point 306A defined within a set of grid points within a first grid 308′ in the environment. The portion of the vector space encoding 302 input to the second point prediction unit 312 may be the same as or different from the portion of the vector space encoding 302 input to the first point prediction unit 310. In some embodiments, the computing system may identify or select the portion of the vector space encoding 302 to input based on the point 306A.

[0082] The second point prediction unit 312 may correspond to a portion of a machine learning (ML) model (e.g., the lane encoder 260 discussed herein) and may include, among other things, a set of weights arranged according to cross-attention, self-attention, and a transformer. In providing, the computing system may process the portion of the vector space encoding 302 according to the weights of the second point prediction unit 312 to calculate, determine, or generate at least one coordinate 332A corresponding to the point 306A. Using the point, the computing system may use the second point prediction unit 312 to calculate, generate, or determine a first index value 332′A at the point 306A.

[0083] 3C , a computing system may input, provide, or apply at least a portion of the vector space encoding 302 to a topology prediction unit 314. The computing system may correspond to a centralized service, such as an ego computing device 141 in an ego 140 or an analysis server 110a. Further, the computing system may apply the output of the first point prediction unit 310 (e.g., an embedding derived from the first index value 330′A) or the output of the second point prediction unit 312 (e.g., an embedding derived from the second index value 332′A) to the topology prediction unit 314. The portion of the vector space encoding 302 input to the topology prediction unit 314 may be the same as or different from the portion of the vector space encoding 302 input to the first point prediction unit 310 or the second point prediction unit 312. The topology prediction unit 314 may correspond to a portion of a machine learning (ML) model (e.g., the lane encoder 260 discussed herein) and may include, among other things, a set of weights arranged according to cross-attention, self-attention, and a transformer.

[0084] By applying the topology prediction unit 314, the computing system can determine, categorize, or classify the point 306A as at least one topology type 334A. The topology type 334A can semantically specify, identify, or define the function of the point 306A with respect to the path along which the ego navigates through the environment as represented by the map 310. The topology type can include, for example, a start type, a continuation type, a branch type, or a terminal type, among others. In the illustrated example, the topology type 334A can be a start type to indicate the start of at least one of the paths.

[0085] 3D , a computing system may input, provide, or apply at least a portion of the vector space encoding 302 to a point attribute predictor 316. The computing system may correspond to a centralized service, such as the ego computing device 141 in the ego 140 or the analytics server 110a. Further, the computing system may apply the output of the first point prediction unit 310 (e.g., an embedding derived from the first index value 330′A), the output of the second point prediction unit 312 (e.g., an embedding derived from the second index value 332′A), or the output (e.g., an embedding derived from the topology type 334′A) to the point attribute predictor 316. The portion of the vector space encoding 302 input to the point attribute predictor 316 may be the same as or different from the portion of the vector space encoding 302 input to the first point prediction unit 310, the second point prediction unit 312, or the topology prediction unit 314.

[0086] The point attribute predictor 316 may correspond to a portion of a machine learning (ML) model (e.g., the lane encoder 260 discussed herein) and may include, among other things, a set of weights arranged according to intersection attention, self-attention, and a transducer. By applying the point attribute predictor 316, the computing system may generate or determine a set of point attributes 336A for the point 306A. The set of point attributes 336A may identify or include, for example, at least one branch identifying an index value of a previous point from which the current point 306A forms a branch, at least one merge point identifying an index value of a previous point from which the current point 306A forms a merge, and a set of spline coefficients 338A. In the illustrated example, because the current point 306A is the starting point, there are no other points, and therefore the set of point attributes 336A may be zero.

[0087] The generated output allows the computing system to generate individual embeddings using predictors in the ML model to form or output a set of embeddings 340A at the points 306A. Using the first point predictor 310, the computing system can determine or generate at least one corresponding embedding 330″A from a first index value 330′A. Using the second point predictor 312, the computing system can determine or generate at least one corresponding embedding 332″A from a second index value 332′A. Using the second point predictor 312, the computing system can determine or generate at least one corresponding embedding 332″A from a second index value 332′A. Using the topology type predictor 314, the computing system can determine or generate at least one corresponding embedding 334′A from a topology type 334A. Similarly, using the point attribute predictor 316, the computing system can determine or generate a set of embeddings 336′A. The set of embeddings 336′A may lack spline coefficients.

[0088] Upon generating the individual embeddings (e.g., embeddings 330″A, 332″A, 334′A, and 336′A), the computing system can write, create, or generate a set of embeddings 340A. Based on the set of embeddings 340A, the computing system can create, output, or generate at least one token 342A for at least one of the paths through the environment. In some embodiments, the computing system can generate the token 342A based on the first index value 330A, the second index value 332A, the topology type 334A, or the set of point attributes 336A, or any combination thereof. The generation can cause the computing system to add, insert, or include the token 342A in the graph 304. The computing system can store and maintain the graph 304 including the token 342A to be used to autonomously navigate the ego through the environment via paths. The graph 304 may be maintained as one or more data structures (eg, an array, a linked list, a matrix, a tree, a hash, a heap, a table, or a graph) on a data storage.

[0089] 3E, a computing system may repeat some of the functionality described herein with respect to a second point 306B. The computing system may correspond to an ego computing device 141 or a centralized service, such as an analytics server 110a, in the ego 140. Identifying the second point 306B enables the computing system to locate, determine, or identify at least one point 306B within a set of grid points within a first grid 308 defined over the environment represented by the map 310. The point 306B may correspond to an intermediate point that the ego will pass through while navigating through the environment defined by the map 310. The grid 308 may specify or define a set of grid points (or coordinates) at a coarser or lower resolution than the original resolution of the grid points defined by the map 310. The map 310 may correspond to a layout of the topology surrounding the ego and may be obtained as part of map data (e.g., the navigation map 235).

[0090] The computing system may input, provide, or apply at least a portion of the vector space encoding 302 to a first point prediction unit 310. The portion of the vector space encoding 302 input to the first point prediction unit 310 may correspond to a point 306B defined within a set of grid points in a first grid 308 in the environment. In some embodiments, the computing system may identify or select the portion of the vector space encoding 302 to input based on the point 306B. In providing, the computing system may process the portion of the vector space encoding 302 according to the weights of the first point prediction unit 310 to calculate, determine, or generate at least one coordinate 330B corresponding to the point 306B. Using this point, the computing system may use the first point prediction unit 310 to calculate, generate, or determine a first index value 330′B at the point 306B.

[0091] The computing system can locate, determine, or identify at least one point 306B within a set of grid points within a second grid 308′ defined over the environment represented by the map 312. The point 306B can correspond to a starting point from which the ego navigates through the environment defined by the map 312. The second grid 308′ can specify or define a set of grid points (or coordinates) at a finer or higher resolution than the resolution of the first grid 308. The second grid 308′ can correspond to a subset of the first grid 308 that surrounds or is around the ego. The map 312 can correspond to a layout of a topology surrounding the ego and can be obtained as part of map data (e.g., navigation map 235).

[0092] The computing system may input, provide, or apply at least a portion of the vector space encoding 302 and the output of the first point prediction unit 310 (e.g., an embedding derived from the first index value 330′B) to the second point prediction unit 312. The portion of the vector space encoding 302 may correspond to a point 306B defined within a set of grid points in a first grid 308′ within the environment. The portion of the vector space encoding 302 input to the second point prediction unit 312 may be the same as or different from the portion of the vector space encoding 302 input to the first point prediction unit 310. In some embodiments, the computing system may identify or select the portion of the vector space encoding 302 to input based on the point 306B. In providing, the computing system may process the portion of the vector space encoding 302 according to the weights of the second point prediction unit 312 to calculate, determine, or generate at least one coordinate 332B corresponding to the point 306B. Using the point, the computing system may use the second point prediction unit 312 to calculate, generate, or determine a first index value 332'B at the point 306B.

[0093] The computing system may input, provide, or apply at least a portion of the vector space encoding 302 to the topology prediction unit 314. Additionally, the computing system may apply the output of the first point prediction unit 310 (e.g., an embedding derived from the first index value 330′B) or the output of the second point prediction unit 312 (e.g., an embedding derived from the second index value 332′B) to the topology prediction unit 314. The portion of the vector space encoding 302 input to the topology prediction unit 314 may be the same as or different from the portion of the vector space encoding 302 input to the first point prediction unit 310 or the second point prediction unit 312.

[0094] By applying the topology prediction unit 314, the computing system can determine, categorize, or classify the point 306B as at least one topology type 334B. The topology type 334B can semantically specify, identify, or define the function of the point 306B with respect to the path along which the ego navigates through the environment as represented by the map 310. The topology type can include, for example, a start type, a continuation type, a branch type, or a termination type, among others. In the illustrated example, the topology type 334B can be a continuation type to indicate a continuation of at least one of the paths between the point 306A and the point 306B.

[0095] The computing system may input, provide, or apply at least a portion of the vector space encoding 302 to the point attribute predictor 316. Additionally, the computing system may apply the output of the first point prediction unit 310 (e.g., an embedding derived from the first index value 330′B), the output of the second point prediction unit 312 (e.g., an embedding derived from the second index value 332′B), or the output (e.g., an embedding derived from the topology type 334′B) to the point attribute predictor 316. The portion of the vector space encoding 302 input to the point attribute predictor 316 may be the same as or different from the portion of the vector space encoding 302 input to the first point prediction unit 310, the second point prediction unit 312, or the topology prediction unit 314.

[0096] By applying the point attribute predictor 316, the computing system can generate or determine a set of point attributes 336B for point 306B. The set of point attributes 336B can identify or include, for example, at least one branch point identifying an index value of a previous point from which the current point 306B forms a branch, at least one junction point identifying an index value of a previous point from which the current point 306B forms a junction, and a set of spline coefficients 338B. In the illustrated example, the current point 306B is a continuation point dependent on point 306A and is not a branch or junction, so the index may be zero. The set of spline coefficients 338B can include a set of values ​​that define a spline curve 338′B between points 306A and 306B. The set of spline coefficients 338B can define a path between points 306A and 306B through the environment.

[0097] The generated output allows the computing system to generate individual embeddings using predictors in the ML model to form or output a set of embeddings 340A at points 306B. Using the first point predictor 310, the computing system can determine or generate at least one corresponding embedding 330″B from a first index value 330′B. Using the second point predictor 312, the computing system can determine or generate at least one corresponding embedding 332″B from a second index value 332′B. Using the second point predictor 312, the computing system can determine or generate at least one corresponding embedding 332″B from a second index value 332′B. Using the topology type predictor 314, the computing system can determine or generate at least one corresponding embedding 334′B from a topology type 334B. Similarly, using the point attribute predictor 316, the computing system can determine or generate a set of embeddings 336′B.

[0098] Upon generating the individual embeddings (e.g., embeddings 330″B, 332″B, 334′B, and 336′B), the computing system can write, create, or generate a set of embeddings 340A. Based on the set of embeddings 340A, the computing system can create, output, or generate at least one token 342B in at least one of the paths through the environment. The computing system can also apply the self-attention unit 318 in determining the tokens 342B. In some embodiments, the computing system can generate the tokens 342B based on the first index value 330B, the second index value 332B, the topology type 334B, or the set of point attributes 336B, or any combination thereof. Upon generation, the computing system can add, insert, or include the tokens 342B in the graph 304. The computing system can update the graph 304 to include the tokens 342B to be used to autonomously navigate the ego through the environment via the paths.

[0099] Continuing with FIG. 3F, the computing system may repeat some of the functionality described herein with respect to the third point 306C. The computing system may correspond to the ego computing device 141 or a centralized service, such as the analytics server 110a, in the ego 140. Identifying the third point 306C enables the computing system to locate, determine, or identify at least one point 306C within a set of grid points within a first grid 308 defined over the environment represented by the map 310. The point 306C may correspond to a termination point near the boundary of the map 310 and may be a point along which the ego is intended to navigate through the environment defined by the map 310. The grid 308 may specify or define a set of grid points (or coordinates) at a coarser or lower resolution than the original resolution of the grid points defined by the map 310. The map 310 may correspond to a layout of the topology surrounding the ego and may be obtained as part of map data (e.g., the navigation map 235).

[0100] The computing system may input, provide, or apply at least a portion of the vector space encoding 302 to a first point prediction unit 310. The portion of the vector space encoding 302 input to the first point prediction unit 310 may correspond to a point 306C defined within a set of grid points in a first grid 308 in the environment. In some embodiments, the computing system may identify or select the portion of the vector space encoding 302 to input based on the point 306C. In providing, the computing system may process the portion of the vector space encoding 302 according to the weights of the first point prediction unit 310 to calculate, determine, or generate at least one coordinate 330C corresponding to the point 306C. Using this point, the computing system may use the first point prediction unit 310 to calculate, generate, or determine a first index value 330′C at the point 306C.

[0101] The computing system can locate, determine, or identify at least one point 306C within a set of grid points within a second grid 308′ defined over the environment represented by the map 312. The point 306C can correspond to a starting point from which the ego navigates through the environment defined by the map 312. The second grid 308′ can specify or define a set of grid points (or coordinates) at a finer or higher resolution than the resolution of the first grid 308. The second grid 308′ can correspond to a subset of the first grid 308 that surrounds or is around the ego. The map 312 can correspond to a layout of a topology surrounding the ego and can be obtained as part of map data (e.g., navigation map 235).

[0102] The computing system may input, provide, or apply at least a portion of the vector space encoding 302 and the output of the first point prediction unit 310 (e.g., an embedding derived from the first index value 330′C) to the second point prediction unit 312. The portion of the vector space encoding 302 may correspond to a point 306C defined within a set of grid points in a first grid 308′ within the environment. The portion of the vector space encoding 302 input to the second point prediction unit 312 may be the same as or different from the portion of the vector space encoding 302 input to the first point prediction unit 310. In some embodiments, the computing system may identify or select the portion of the vector space encoding 302 to input based on the point 306C. In providing, the computing system may process the portion of the vector space encoding 302 according to the weights of the second point prediction unit 312 to calculate, determine, or generate at least one coordinate 332C corresponding to the point 306C. Using this point, the computing system may use the second point prediction unit 312 to calculate, generate, or determine a first index value 332'C at point 306C.

[0103] The computing system may input, provide, or apply at least a portion of the vector space encoding 302 to the topology prediction unit 314. Additionally, the computing system may apply the output of the first point prediction unit 310 (e.g., an embedding derived from the first index value 330′C) or the output of the second point prediction unit 312 (e.g., an embedding derived from the second index value 332′C) to the topology prediction unit 314. The portion of the vector space encoding 302 input to the topology prediction unit 314 may be the same as or different from the portion of the vector space encoding 302 input to the first point prediction unit 310 or the second point prediction unit 312.

[0104] By applying the topology prediction unit 314, the computing system can determine, categorize, or classify the point 306C as at least one topology type 334C. The topology type 334C can semantically specify, identify, or define the function of the point 306C with respect to the path along which the ego navigates through the environment as represented by the map 310. The topology type can include, for example, a start type, a continuation type, a branch type, or a termination type, among others. In the illustrated example, the topology type 334C can be a continuation type to indicate a continuation of at least one of the paths between the point 306A and the point 306C.

[0105] The computing system may input, provide, or apply at least a portion of the vector space encoding 302 to the point attribute predictor 316. Additionally, the computing system may apply the output of the first point prediction unit 310 (e.g., an embedding derived from the first index value 330′C), the output of the second point prediction unit 312 (e.g., an embedding derived from the second index value 332′C), or the output (e.g., an embedding derived from the topology type 334′C) to the point attribute predictor 316. The portion of the vector space encoding 302 input to the point attribute predictor 316 may be the same as or different from the portion of the vector space encoding 302 input to the first point prediction unit 310, the second point prediction unit 312, or the topology prediction unit 314.

[0106] By applying the point attribute predictor 316, the computing system can generate or determine a set of point attributes 336C for point 306C. The set of point attributes 336C can identify or include, for example, at least one branch point identifying an index value of a previous point from which the current point 306C forms a branch, at least one junction point identifying an index value of a previous point from which the current point 306C forms a junction, and a set of spline coefficients 338C. In the illustrated example, the current point 306C is a continuation point dependent on point 306A, and is not a branch or junction, so the index may be zero. The set of spline coefficients 338C can include a set of values ​​that define a spline curve 338'C between points 306A and 306C. The set of spline coefficients 338C can define a path between points 306A and 306C through the environment.

[0107] In some embodiments, the computing system may also determine whether point 306C is a termination point toward the boundary of the map 310 based on coordinates from the first grid 308 or the second grid 308′. A point may be determined to be a termination point if the coordinates of the point are within the confluence of the boundaries of the map 310 defined by the first grid 308 or the second grid 308′. The termination point may correspond to the end of a path (or line segment) opposite the initial starting point (e.g., point 306A). To determine this, the computing system may identify the coordinates of point 306C defined by the first grid 308 or the second grid 308′. Upon identification, the computing system may compare the coordinates of point 306C with coordinates corresponding to the boundary of the acquired map 310. When the coordinates of point 306C are within the confluence of the boundaries of the acquired map 310 (e.g., corresponding to 1 m-10 m), the computing system may determine that point 306C is a termination point (e.g., as shown). Further, the computing system may determine that a path is defined between point 306A and point 306C (e.g., via point 306B). Otherwise, if the coordinates of point 306C are outside the confluence of the boundaries of the acquired map 310 (e.g., corresponding to 1 m-10 m), the computing system may determine that point 306C is not a terminal point (e.g., as shown).

[0108] The generated output allows the computing system to generate individual embeddings using predictors in the ML model to form or output a set of embeddings 340A at points 306C. Using the first point predictor 310, the computing system can determine or generate at least one corresponding embedding 330″C from a first index value 330′C. Using the second point predictor 312, the computing system can determine or generate at least one corresponding embedding 332″C from a second index value 332′C. Using the second point predictor 312, the computing system can determine or generate at least one corresponding embedding 332″C from a second index value 332′C. Using the topology type predictor 314, the computing system can determine or generate at least one corresponding embedding 334′C from a topology type 334C. Similarly, using the point attribute predictor 316, the computing system can determine or generate a set of embeddings 336′C.

[0109] Upon generating the individual embeddings (e.g., embeddings 330″C, 332″C, 334′C, and 336′C), the computing system can write, create, or generate a set of embeddings 340A. Based on the set of embeddings 340A, the computing system can create, output, or generate at least one token 342C in at least one of the paths through the environment. The computing system may also apply the self-attention unit 318 in determining the tokens 342C. In some embodiments, the computing system may generate the tokens 342C based on the first index value 330C, the second index value 332C, the topology type 334C, or the set of point attributes 336C, or any combination thereof. Upon generation, the computing system can add, insert, or include the tokens 342C in the graph 304. The computing system can update the graph 304 to include the tokens 342C to be used to autonomously navigate the ego through the environment via the paths.

[0110] 3G, a computing system may repeat some of the functionality described herein with respect to the fourth point 306D. The computing system may correspond to an ego computing device 141 or a centralized service, such as an analytics server 110a, in the ego 140. Identifying the fourth point 306D enables the computing system to locate, determine, or identify at least one point 306D within a set of grid points within a first grid 308 defined over the environment represented by the map 310. The point 306D may correspond to an intermediate point that the ego will traverse while navigating through the environment defined by the map 310. The grid 308 may specify or define a set of grid points (or coordinates) at a coarser or lower resolution than the original resolution of the grid points defined by the map 310. The map 310 may correspond to a layout of the topology surrounding the ego and may be obtained as part of map data (e.g., the navigation map 235).

[0111] The computing system may input, provide, or apply at least a portion of the vector space encoding 302 to a first point prediction unit 310. The portion of the vector space encoding 302 input to the first point prediction unit 310 may correspond to a point 306D defined within a set of grid points in a first grid 308 in the environment. In some embodiments, the computing system may identify or select the portion of the vector space encoding 302 to input based on the point 306D. In providing, the computing system may process the portion of the vector space encoding 302 according to the weights of the first point prediction unit 310 to calculate, determine, or generate at least one coordinate 330D corresponding to the point 306D. Using this point, the computing system may use the first point prediction unit 310 to calculate, generate, or determine a first index value 330′D at the point 306D.

[0112] The computing system can locate, determine, or identify at least one point 306D within a set of grid points within a second grid 308′ defined over the environment represented by the map 312. The point 306D can correspond to a starting point from which the ego navigates through the environment defined by the map 312. The second grid 308′ can specify or define a set of grid points (or coordinates) at a finer or higher resolution than the resolution of the first grid 308. The second grid 308′ can correspond to a subset of the first grid 308 that surrounds or is around the ego. The map 312 can correspond to a layout of a topology surrounding the ego and can be obtained as part of map data (e.g., navigation map 235).

[0113] The computing system may input, provide, or apply at least a portion of the vector space encoding 302 and the output of the first point prediction unit 310 (e.g., an embedding derived from the first index value 330′D) to the second point prediction unit 312. The portion of the vector space encoding 302 may correspond to a point 306D defined within a set of grid points in a first grid 308′ within the environment. The portion of the vector space encoding 302 input to the second point prediction unit 312 may be the same as or different from the portion of the vector space encoding 302 input to the first point prediction unit 310. In some embodiments, the computing system may identify or select the portion of the vector space encoding 302 to input based on the point 306D. In providing, the computing system may process the portion of the vector space encoding 302 according to the weights of the second point prediction unit 312 to calculate, determine, or generate at least one coordinate 332D corresponding to the point 306D. Using the point, the computing system may use the second point prediction unit 312 to calculate, generate, or determine a first index value 332'D at the point 306D.

[0114] The computing system may input, provide, or apply at least a portion of the vector space encoding 302 to the topology prediction unit 314. Additionally, the computing system may apply the output of the first point prediction unit 310 (e.g., an embedding derived from the first index value 330′D) or the output of the second point prediction unit 312 (e.g., an embedding derived from the second index value 332′D) to the topology prediction unit 314. The portion of the vector space encoding 302 input to the topology prediction unit 314 may be the same as or different from the portion of the vector space encoding 302 input to the first point prediction unit 310 or the second point prediction unit 312.

[0115] By applying the topology prediction unit 314, the computing system can determine, categorize, or classify the point 306D as at least one topology type 334D. The topology type 334D can semantically specify, identify, or define the function of the point 306D with respect to the path along which the ego navigates through the environment as represented by the map 310. The topology type can include, for example, a start type, a continuation type, a branch type, or a termination type, among others. In the illustrated example, the topology type 334D can be a branch type indicating a continuation of at least one of the path between the point 306A and the point 306D and a branch by the extension point 306C to the path defined between the point 306A and the point 306B.

[0116] The computing system may input, provide, or apply at least a portion of the vector space encoding 302 to the point attribute predictor 316. Additionally, the computing system may apply the output of the first point prediction unit 310 (e.g., an embedding derived from the first index value 330′D), the output of the second point prediction unit 312 (e.g., an embedding derived from the second index value 332′D), or the output (e.g., an embedding derived from the topology type 334′D) to the point attribute predictor 316. The portion of the vector space encoding 302 input to the point attribute predictor 316 may be the same as or different from the portion of the vector space encoding 302 input to the first point prediction unit 310, the second point prediction unit 312, or the topology prediction unit 314.

[0117] By applying the point attribute predictor 316, the computing system can generate or determine a set of point attributes 336D for point 306D. The set of point attributes 336D can identify or include, for example, at least one branch identifying an index value of a previous point from which the current point 306D forms a branch, at least one junction point identifying an index value of a previous point from which the current point 306D forms a junction, and a set of spline coefficients 338D. In the illustrated example, the current point 306D is a branch dependent on point 306A, so the index in the branch may refer to the index value of point 306A. The set of spline coefficients 338D can include a set of values ​​that define a spline curve 338'D between points 306A and 306D. The set of spline coefficients 338D can define a path between points 306A and 306D through the environment.

[0118] Using the generated output, the computing system can generate individual embeddings using predictors in the ML model to form or output a set of embeddings 340A at point 306D. Using the first point predictor 310, the computing system can determine or generate at least one corresponding embedding 330″D from a first index value 330′D. Using the second point predictor 312, the computing system can determine or generate at least one corresponding embedding 332″D from a second index value 332′D. Using the second point predictor 312, the computing system can determine or generate at least one corresponding embedding 332″D from a second index value 332′D. Using the topology type predictor 314, the computing system can determine or generate at least one corresponding embedding 334′D from a topology type 334D. Similarly, using the point attribute predictor 316, the computing system can determine or generate a set of embeddings 336′D.

[0119] Upon generating the individual embeddings (e.g., embeddings 330″D, 332″D, 334′D, and 336′D), the computing system can write, create, or generate a set of embeddings 340A. Based on the set of embeddings 340A, the computing system can create, output, or generate at least one token 342D for at least one of the paths through the environment. The computing system can also apply the self-attention unit 318 in determining the tokens 342D. In some embodiments, the computing system can generate the tokens 342D based on the first index value 330D, the second index value 332D, the topology type 334D, or the set of point attributes 336D, or any combination thereof. Upon generation, the computing system can add, insert, or include the tokens 342D in the graph 304. The computing system can update the graph 304 to include the tokens 342D to be used to autonomously navigate the ego through the environment via the paths.

[0120] Continuing with FIG. 3H, the computing system may repeat the above functions any number of times to reach terminal point 306N. The computing system may correspond to ego computing device 141 in ego 140 or a centralized service such as analysis server 110a. The computing system may repeat some of the functions described herein with respect to point 306N. Upon identifying point 306N, the computing system may have determined an index value based on grids 308 and 308′ described herein and may have determined a corresponding embedding in the index value. The computing system may classify point 306N as at least one topology type that identifies an end-of-sentence (or end-of-graph) topology point. The end-of-sentence may correspond to the completion of a graph derived from map 310 obtained for the ego's surroundings. In the illustrated example, the computing system may determine point 306N as an end-of-sentence topology type. The computing system may complete the generation of graph 304 by generating and inserting a token at point 306N into graph 304 in a manner similar to that discussed herein. The graph 304 may be continuously generated and updated as the ego navigates through the environment.

[0121] This generation allows the ego's computing system to use graph 304 to autonomously navigate the ego through the environment along one of the paths defined using the set of tokens in graph 304. The computing system can generate, calculate, or determine at least one trajectory using the set of tokens in graph 304. The trajectory can identify, specify, or define the ego's navigation through the environment via a corresponding path of the set of paths defined by graph 304. In determining, the computing system can monitor the ego's position and movement within the environment and select one of the set of paths based on the position and movement (e.g., by proximity). Upon selection, the computing system can project or determine a trajectory using a path from the ego's position within the environment. The use of these trajectories is described in more detail below with respect to Figures 4-6.

[0122] In some embodiments, the computing system may display, render, or present on a graphical user interface (GUI) a graph 304 that defines a set of paths through the environment. The set of paths may represent potential lanes, paths, or trajectories along which the ego can navigate through the environment. The GUI may be rendered on a display communicatively coupled to the computing system, such as on a touch display screen within an autonomous vehicle (e.g., the ego). The set of paths may be presented on the GUI as defined by graph 304 (e.g., overlaid as shown generally along the right) against the topology of the environment surrounding the ego, as defined by map 310. Each path may be presented as one or more nodes and edges between the nodes. Each node may correspond to one of the points (e.g., points 306A-N), and each edge may correspond to a line or curve between corresponding pairs of nodes using a set of spline coefficients.

[0123] Referring now to FIG. 4, a diagram of a scenario of an environment 400 is depicted in which a first ego autonomously navigates through the environment using a graph representing a path. In the environment 400, the ego 405 can navigate through road intersections. As the ego 405 traverses the environment 400, cameras (and other sensors) on the ego 405 can acquire a set of videos, such as a first video 415A from a left front side camera, a second video 415B from a center front camera, and a third video 415C from a right front camera. A computing device on the ego 405 can also acquire map data that defines various information about the topology of the environment 400. Using the acquired data, the computing device can generate a graph, as detailed herein, to define a set of paths 420A-420C along roads. Using sensor data, the computing device can detect the presence of other egos 425 as well as human bystanders 435 within the environment 400. Based on this and other data, the computing device on ego 405 may decide to autonomously navigate along path 420B.

[0124] Referring now to FIG. 5, a diagram of an environment scenario 500 is depicted in which a first ego uses a graph to detect when a path traversed by the first ego is about to intersect with a path a second ego is about to traverse. In environment 505, a first ego 505 and a second ego 510 (or non-autonomous vehicles) may be navigating through a road intersection. As ego 505 traverses environment 500, a camera (and other sensors) at ego 505 can acquire at least one video 515. From video 515, a computing device at ego 505 can detect the presence of other egos (or vehicles on roads). Furthermore, the computing device at ego 505 can generate a graph, as detailed herein, to define a set of paths 520A and 520B along roads through the environment as discussed herein. Similarly, a computing device at ego 510 can generate a graph, as detailed herein, to define path 525. In some embodiments, a computing device at ego 505 can generate multiple graphs for detected egos in the environment, including a graph defining a set of potential paths at ego 510.

[0125] Through generation, the computing device at ego 505 can identify a graph generated for its own navigation through the environment. Additionally, the computing device at ego 505 can identify graphs at other egos 510 with its own set of tokens to define the ego 510's autonomous navigation through the environment via one or more paths. Once identified, the computing device can use the two graphs to determine whether the path 520A of ego 505 intersects with the path 525 of the other ego 510. In determining, the computing device can determine a predicted trajectory for each ego 510. When the paths do not intersect, the computing device at ego 505 can continue to navigate autonomously along the path (e.g., path 520A). On the other hand, if the paths do intersect, the computing device can detect or determine that ego 505 and ego 510 have a potential collision. In response to this determination, the computing device at ego 505 (or ego 510) can take action with respect to ego 505 (or ego 515) to avoid the potential collision. For example, a computing device at ego 505 can stop ego 505 from propelling or change the trajectory of ego 515 away from ego 505 .

[0126] Referring now to FIG. 6, a diagram of a scenario 600 is depicted in an environment in which a first ego uses a graph to detect the presence of a stationary second ego within the environment. In the environment 600, the ego 605 can navigate along roads. As the ego 605 traverses the environment 600, cameras (and other sensors) on the ego 605 can acquire a set of videos, such as a first video 615A from a left front side camera, a second video 615B from a center front camera, and a third video 615C from a right front camera. A computing device on the ego 605 can also acquire map data that defines various information about the topology of the environment 600. Using the acquired data, the computing device can generate a graph, as detailed herein, to define at least one path 620.

[0127] Together, based on the sensor data, the computing device at ego 605 can determine or detect the presence of other egos 620 and 625 within environment 600. From the sensor data, the computing device can determine or identify the presence of a stationary ego 620 within environment 600. This detection can allow the computing device on ego 605 to determine that at least one path defined by the graph at ego 605 intersects with the ego 620 identified as stationary. In response to detecting the intersection, the computing device at ego 605 can perform an action with respect to ego 605 to avoid the stationary ego 620. For example, the computing device at ego 605 can use the sensor data and map data to calculate a new trajectory for moving ego 620 around on a road. The calculation can allow the computing device to control ego 605 to take a detour trajectory.

[0128] Referring now to FIG. 7, a flow diagram of a method 700 for generating a path for autonomously navigating through an environment is depicted. Method 700 can be implemented using any of the components described herein, such as ego computing devices 141a-c and AI model 110c. Method 700 may include the steps described herein. However, other embodiments may include additional or alternative steps, or omit one or more steps. Method 700 may be performed by an analysis server (e.g., a computer similar to analysis server 110a) or an ego computing device (e.g., ego computing devices 141a-c). However, one or more steps of process 300 may be performed by any number of computing devices operating in the distributed computing system described in FIGS. 1A-1C (e.g., processors of ego 140 and / or ego computing device 141, or a centralized service such as analysis server 110a). For example, one or more computing devices of an ego may locally perform some or all of the steps described in FIG. 7.

[0129] Under method 700, in step 705, a computing system (e.g., a processor of ego 140 and / or ego computing device 141, or a centralized service such as analytics server 110a) may obtain, search, or identify a tensor (e.g., vector space encoding 302). The sensors may include a set of encodings derived from sensor data acquired via one or more cameras and map data that defines the topology of the environment surrounding the ego. The encodings may be low-dimensional representations of features in the sensor data and map data to be used to define lane segments in the environment.

[0130] In step 710, the computing system may determine, select, or identify points within the environment (e.g., points 306A-N). The points may correspond to positions, locations, or spots along which the ego navigates through the environment defined by the map data. The points may be start points, intermediate (or continuation) points, or end points that define one or more potential lane segments within the environment. The map data may correspond to a layout of the topology surrounding the ego and may be used to identify points within the environment.

[0131] In step 715, the computing system may calculate, determine, or generate a first index value for a point within a first grid (e.g., first grid 308) defined across the environment. The first grid may correspond to a set of grid points at a coarser or lower resolution than the original resolution of the map. To generate, the computing system may apply a portion of a tensor corresponding to the point to a first prediction unit. The first prediction unit may include, among other things, cross-attention, self-attention, and a set of weights arranged according to a transformer. By applying, the computing system may use the set of coordinates to determine a set of coordinates corresponding to the point within the first grid and a first index value at the point.

[0132] In step 720, the computing system may calculate, determine, or generate a second index value for a point within a second grid (e.g., second grid 308′) defined across the environment. The second grid may correspond to a set of grid points at a finer or higher resolution than the first grid. To generate, the computing system may apply the portion of the tensor corresponding to the point and the first index value (or an embedding derived therefrom) to a second prediction unit. The second prediction unit may include, among other things, cross-attention, self-attention, and a set of weights arranged according to a transformer. By applying, the computing system may use the set of coordinates to determine a set of coordinates corresponding to the point within the second grid and a second index value for the point.

[0133] In step 725, the computing system can categorize, assign, or classify a topology type of the point. The topology type can semantically define the function of the point relative to the lane segment along which the ego is to navigate through the environment. The topology type can include, for example, a start type, a continuation type, a branch type, or a termination type, among others. To classify, the computing system can apply the portion of the tensor corresponding to the point along with the first or second index value (or derived embedding) to a topology prediction unit. The second prediction unit can include, among others, cross-attention, self-attention, and a set of weights arranged according to a transformer. Upon application, the computing system can classify the point into one of the topology types.

[0134] In step 730, the computing system can determine, generate, or identify point attributes. The point attributes may be, among other things, at least one of a bifurcation point identifying the index value of a previous point from which the current point forms a divergence, at least one junction point identifying the index value of a previous point from which the current point forms a junction, and a set of spline coefficients from another point. To identify them, the computing system can apply a portion of the tensor along with other outputs (e.g., an embedding derived from the index values ​​and topology type) to a point attribute predictor. The point attribute predictor can include, among other things, cross-attention, self-attention, and a set of weights arranged according to the transformer. Upon application, the computing system can identify a set of point attributes.

[0135] In step 735, the computing system may generate, determine, or create an embedding set using the first index value, the second index value, the topology type, and the set of point attributes. Each embedding may correspond to a respective output from the machine learning model or may be a reduced-dimensional representation of the output. The embeddings may be generated by applying model weights. In step 740, the computing system may write, generate, or create tokens using the set of embeddings. To create the tokens, the computing system may combine the sets of embeddings. Each token may correspond to a point that defines a portion of one or more lane segments through the environment.

[0136] In step 745, the computing system may add, include, or insert tokens into the graph. The graph may semantically define a set of potential lane segments along which the ego can navigate through the environment. In step 750, the computing system may determine whether there are additional points in the environment. The determination may be based on the topology type. If the topology type indicates an end of a sentence, the computing system may determine that there are no points to be evaluated in the environment. On the other hand, if the topology type indicates another type other than an end of a sentence, the computing system may repeat method 700 from step 710. In step 755, the computing system may store and maintain the graph. The graph may be used by the ego to perform autonomous navigation through the environment.

[0137] Additionally or alternatively, the analytics server may send the generated map to a downstream software application or another server. The predicted results may be further analyzed and used in various models and / or algorithms to perform various actions. For example, a software model or processor associated with ego's autonomous navigation system may receive occupancy data predicted by a trained AI model and make navigation decisions accordingly.

[0138] The various illustrative logic blocks, modules, circuits, and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, computer software, or a combination of both. To clearly illustrate this interchangeability between hardware and software, the various illustrative components, blocks, modules, circuits, and steps have been described generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the particular application and design constraints imposed on the overall system. Those skilled in the art may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present disclosure or the claims.

[0139] Computer software-implemented embodiments may be implemented in software, firmware, middleware, microcode, hardware description languages, or any combination thereof. A code segment or machine-executable instruction may represent a procedure, a function, a subprogram, a program, a routine, a subroutine, a module, a software package, a class, or any combination of instructions, data structures, or program statements. A code segment may be coupled to another code segment or a hardware circuit by passing and / or receiving information, data, arguments, parameters, or memory contents. Information, arguments, parameters, data, etc. may be passed, forwarded, or transmitted via any suitable means including memory sharing, message passing, token passing, network transmission, etc.

[0140] The actual software code or specialized control hardware used to implement these systems and methods is not a limitation of the claimed features or this disclosure. Accordingly, the operation and behavior of the systems and methods are described without reference to specific software code, with the understanding that software and control hardware can be designed to implement the systems and methods based on the description herein.

[0141] If implemented in software, the functions may be stored as one or more instructions or code on a non-transitory, computer-readable, or processor-readable storage medium. The steps of a method or algorithm disclosed herein may be embodied in a processor-executable software module, which may reside on a computer-readable or processor-readable storage medium. Non-transitory computer-readable or processor-readable media includes both computer storage media and tangible storage media that facilitate transfer of a computer program from one place to another. Non-transitory processor-readable storage media may be any available medium that can be accessed by a computer. By way of example, and not limitation, such non-transitory processor-readable media may include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other tangible storage medium that can be used to store desired program code in the form of instructions or data structures and that can be accessed by a computer or processor. As used herein, disk and disc include compact discs (CDs), laser discs, optical discs, digital versatile discs (DVDs), Blu-ray discs, and floppy disks, with a "disc" typically reproducing data magnetically and a "disc" reproducing data optically using a laser. Combinations of the above should also be included within the scope of computer-readable media. Furthermore, the operations of a method or algorithm may reside as one or any combination or set of code and / or instructions on a non-transitory processor-readable medium and / or computer-readable medium, which may be incorporated into a computer program product.

[0142] The previous description of the disclosed embodiments is provided to enable any person skilled in the art to make or use the embodiments described herein and variations thereof. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the principles defined herein may be applied to other embodiments without departing from the spirit or scope of the subject matter disclosed herein. Thus, the present disclosure is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the following claims and the principles and novel features disclosed herein.

[0143] While various aspects and embodiments have been disclosed, other aspects and embodiments are contemplated. The various disclosed aspects and embodiments are for purposes of illustration and are not intended to be limiting, with the true scope and spirit being indicated by the following claims.

Claims

1. 1. A method of generating a path for autonomously navigating through an environment, comprising: Identifying, by one or more processors, a tensor comprising a plurality of encodings derived from sensor data from an ego and map data defining a topology of an environment surrounding said ego; determining, by the one or more processors, a first index value defining a point within a first plurality of points of a first grid defined over the environment by applying at least a first portion of the plurality of encodings to a machine learning (ML) model; determining, by the one or more processors, second index values ​​that define points in the second plurality of points of a second grid within a subset of the first plurality of points of the first grid by applying at least a second portion of the plurality of encodings and the first index values ​​to the ML model; generating, by the one or more processors, a token for at least one of a plurality of paths through the environment based on the first index value and the second index value at the point; storing, by the one or more processors, a graph to include the tokens to be used to autonomously navigate the ego through the environment via one or more of the plurality of paths; A method comprising:

2. classifying, by the one or more processors, the second point as a continuous topology type that is dependent on the first point by applying at least a third portion of the plurality of encodings and a third index value at the second point to the ML model; determining, by the one or more processors, in response to classifying the second point as the continuous topology type, a plurality of spline coefficients that define one of the plurality of paths between the first point and the second point through the environment; generating, by the one or more processors, a second token based on a third index point at the second point and the continuation topology type; updating, by the one or more processors, the graph to include the second token and the plurality of spline coefficients to be used to autonomously navigate the ego through the environment; The method of claim 1 further comprising:

3. classifying, by the one or more processors, the second point as a branching topology type from the point relative to a third point by applying at least a third portion of the plurality of encodings and a third index value at the second point to the ML model; determining, by the one or more processors, in response to classifying the second point as the branch topology type, a fourth index value referencing the token at the point; generating, by the one or more processors, a second token in a first path that is different from a second path associated with the third point based on the third index value, the fourth index value, and the branch topology type; updating, by the one or more processors, the graph to include the second token to be used to autonomously navigate the ego through the environment; The method of claim 1 further comprising:

4. classifying, by the one or more processors, the second point as a termination topology type that is dependent on the first point by applying at least a third portion of the plurality of encodings and a third index value at the second point to the ML model; determining, by the one or more processors, a path defined by the first point and the second point through the environment in response to the second point being the termination topology type; generating, by the one or more processors, a second token based on the third index point at the second point and the termination topology type; updating, by the one or more processors, the graph to include the second token to be used to autonomously navigate the ego through the environment; The method of claim 1 further comprising:

5. identifying, by the one or more processors, a second graph including a plurality of tokens to be used to autonomously navigate a second ego through the environment via one or more of a second plurality of paths; determining, by the one or more processors, using the graph and the second graph, that at least one first path of the plurality of paths in the ego intersects with at least one second path of the second plurality of paths in the second ego; performing, by the one or more processors, an action with respect to at least one of the ego or the second ego in response to determining that at least one first path intersects with the second path; The method of claim 1 further comprising:

6. identifying, by the one or more processors, the presence of a second ego stationary within the environment using the sensor data from the ego; determining, by the one or more processors, using the graph, that at least one first path of the plurality of paths in the ego intersects with the stationary second ego; The method of claim 1 further comprising:

7. classifying, by the one or more processors, the point as a topology type that indicates a start of at least one of the plurality of paths by applying at least a third portion of the plurality of encodings and the second index value to the ML model; The method of claim 1 , wherein generating the token further comprises generating the token for at least one of the plurality of paths through the environment based on the topology type.

8. 2. The method of claim 1, further comprising determining, by the one or more processors, a trajectory that defines navigation of the ego via one of the multiple paths through the environment using multiple tokens of the graph.

9. 10. The method of claim 1, further comprising presenting, by the one or more processors, via a graphical user interface (GUI), the graph defining the plurality of paths relative to the topology of the environment surrounding the ego.

10. 2. The method of claim 1, wherein generating the token further comprises generating the token using (i) a first embedding generated from the first index value, (ii) a second embedding generated from the second index value, and (iii) one or more embeddings associated with the point.

11. 1. A system for generating a path for autonomously navigating through an environment, comprising: one or more processors coupled to a memory, the processors comprising: identifying a tensor comprising a plurality of encodings derived from sensor data from an ego and map data defining a topology of an environment surrounding the ego; determining a first index value defining a point within a first plurality of points of a first grid defined over the environment by applying at least a first portion of the plurality of encodings to a machine learning (ML) model; determining second index values ​​defining points in the second plurality of points of a second grid within a subset of the first plurality of points of the first grid by applying at least a second portion of the plurality of encodings and the first index values ​​to the ML model; generating a token for at least one of a plurality of paths through the environment based on the first index value and the second index value at the point; storing a graph to include the tokens to be used to autonomously navigate the ego through the environment via one or more of the plurality of paths; The system is configured as follows:

12. The one or more processors: classifying the second point as a continuous topology type dependent on the first point by applying at least a third portion of the plurality of encodings and a third index value at the second point to the ML model; determining a plurality of spline coefficients that define one of the plurality of paths between the first point and the second point through the environment in response to classifying the second point as the continuous topology type; generating a second token based on the third index point at the second point and the continuation topology type; updating the graph to include the second token and the plurality of spline coefficients to be used to autonomously navigate the ego through the environment. The system of claim 11 further configured to:

13. The one or more processors: classifying the second point as a branching topology type from the point relative to a third point by applying at least a third portion of the plurality of encodings and a third index value at the second point to the ML model; determining a fourth index value referencing the token at the point in response to classifying the second point as the branch topology type; generating a second token in a first path that is different from a second path associated with the third point based on the third index value, the fourth index value, and the branch topology type; updating the graph to include the second token to be used to autonomously navigate the ego through the environment; The system of claim 11 further configured to:

14. The one or more processors: classifying the second point as a termination topology type dependent on the first point by applying at least a third portion of the plurality of encodings and a third index value at the second point to the ML model; determining a path defined by the first point and the second point through the environment in response to classifying the second point as the termination topology type; generating a second token based on the third index point at the second point and the termination topology type; updating the graph to include the second token to be used to autonomously navigate the ego through the environment; The system of claim 11 further configured to:

15. The one or more processors: identifying a second graph including a plurality of tokens to be used to autonomously navigate the second ego through the environment via one or more of a second plurality of paths; using the graph and the second graph to determine that at least one first path of the plurality of paths in the ego intersects with at least one second path of the second plurality of paths in the second ego; performing an action with respect to at least one of the ego or the second ego in response to determining that at least one first path intersects with the second path; The system of claim 11 further configured to:

16. The one or more processors: using the sensor data from the ego to identify the presence of a second ego stationary within the environment; using the graph to determine that at least one first path of the plurality of paths in the ego intersects with the stationary second ego; The system of claim 11 further configured to:

17. The one or more processors: classifying the point as a topology type that indicates the start of at least one of the plurality of paths by applying at least a third portion of the plurality of encodings and the second index value to the ML model; generating the token for at least one of the plurality of paths through the environment based on the topology type; The system of claim 11 further configured to:

18. 12. The system of claim 11, wherein the one or more processors are further configured to determine a trajectory that defines navigation of the ego via one of the multiple paths through the environment using multiple tokens of the graph.

19. 12. The system of claim 11, wherein the one or more processors are further configured to present the graph defining the plurality of paths relative to the topology of the environment surrounding the ego via a graphical user interface (GUI).

20. 12. The system of claim 11, wherein the one or more processors are further configured to generate the token using (i) a first embedding generated from the first index value, (ii) a second embedding generated from the second index value, and (iii) one or more embeddings associated with the point.

Images