Artificial intelligence modeling techniques for joint behavioral planning and prediction

Abstract

A method includes: detecting one or more proxy objects in a space around an autologous object using image data captured by a camera of the autologous object; storing a hierarchical node graph, the hierarchical node graph comprising a destination layer and a plurality of interaction layers of interaction nodes following the destination layer, the destination layer comprising one or more destination nodes; adding an interaction node to an interaction layer of interaction nodes in the plurality of interaction layers; determining a trajectory score for each of the plurality of trajectories based on one or more node scores of one or more nodes corresponding to the trajectory within the hierarchical node graph; and selecting a trajectory of the plurality of trajectories for the autologous object based on the trajectory score for the trajectory.

Description

[0001] Cross - Reference to Related Applications

[0002] This application claims priority to U.S. Provisional Application No. 63 / 377,954, filed Sep. 30, 2022, and U.S. Provisional Application No. 63 / 378,028, filed Sep. 30, 2022, each of which is hereby incorporated by reference in its entirety for all purposes. Technical Field

[0003] The present disclosure generally relates to artificial intelligence-based modeling techniques for selecting suitable trajectories for an ego body. Background Art

[0004] Due to the rapid development of computer technology, autonomous navigation techniques for autonomous vehicles and robots (collectively referred to as ego bodies) have become ubiquitous. These advancements allow for safer and more reliable autonomous navigation of ego bodies. Ego bodies typically need to navigate in complex and dynamic environments and terrains, which can include vehicles, traffic, pedestrians, cyclists, and various other static or dynamic obstacles. Understanding the surrounding environment of an ego body is necessary for intelligent and capable decision-making to avoid collisions. Summary of the Invention

[0005] For the reasons above, there is a desire for methods and systems that can analyze the surrounding environment of an ego body and select trajectories that avoid collisions with objects in the surrounding environment of the ego body. A system (e.g., a computing system of an ego body) implementing the systems and methods herein can do so using a hierarchical node graph with an interactive node layer, where these interactive nodes represent potential interactions or non-interactions with one or more agent objects detected by the system in the surrounding environment of the ego body. The system can generate scores for corresponding interactive nodes of the hierarchical node graph based on different variables, such as physics-based constraints, comfort, likelihood of intervention, and / or humanoid discriminators. The system can combine the scores for individual nodes to determine a score for that node. The system can identify trajectories represented by the hierarchical node graph. Trajectories can respectively include different variations of interactive nodes of different layers and are linked to each other within the hierarchical node graph. The system can generate a trajectory score for a trajectory based on the node scores of the interactive nodes of the corresponding trajectory, such as by performing a function on the corresponding node scores of the corresponding trajectory. The system can compare the trajectory scores with each other to select the trajectory with the highest trajectory score. The system can use the selected trajectory to operate the ego body. This process can greatly simplify trajectory selection techniques such that an ego body can make faster decisions using less processing power than traditional techniques, which may attempt to determine possible trajectories for objects in the surrounding environment, which may require a large amount of processing resources on a busy street.

[0006] In one embodiment, a method includes: using, by a processor, image data captured by a camera of a self-object to detect one or more proxy objects in a space around the self-object; storing, by the processor, a hierarchical node graph that includes: a purpose layer that includes one or more purpose nodes corresponding to purposes for the self-object to achieve; a plurality of interaction layers of interaction nodes after the purpose layer, each interaction node corresponding to at least one of a plurality of trajectories for the self-object in view of one or more proxy objects and corresponding to a node score, the plurality of interaction layers including an initial interaction layer of interaction nodes and a plurality of subsequent interaction layers of interaction nodes after the initial interaction layer of interaction nodes, wherein each interaction node in the plurality of subsequent interaction layers depends on at least one interaction node in a previous interaction layer in the plurality of interaction layers; in response to determining that a node score of a first interaction node in the initial interaction layer of interaction nodes exceeds a threshold, adding, by the processor, a second interaction node to a subsequent interaction layer of interaction nodes in the plurality of subsequent interaction layers, the second interaction node being linked to the first interaction node; determining, by the processor, a trajectory score for each of the plurality of trajectories based on one or more node scores of one or more nodes corresponding to the trajectories; and selecting, by the processor, a trajectory from the plurality of trajectories for the self-object based on the trajectory score for the trajectory.

[0007] The method may further include: controlling, by the processor, the self-object according to the selected trajectory.

[0008] Determining the trajectory score for a trajectory may include aggregating, by the processor, one or more node scores of one or more nodes of the trajectory.

[0009] Selecting a trajectory may include selecting, by the processor, a trajectory in response to determining that the trajectory score for the trajectory is higher than the trajectory scores of other trajectories in the plurality of trajectories.

[0010] The method may include: executing, by the processor, a neural network to determine a node score for each interaction node of the hierarchical node graph.

[0011] The node score for each interaction node may correspond to a comfort level associated with the interaction node.

[0012] The node score for each interaction node may correspond to a comfort level associated with the interaction node and a likelihood of intervention associated with the interaction node.

[0013] The method may further include: generating, by the processor, the hierarchical node graph in response to detecting one or more proxy objects in a space around the self-object using image data captured by a camera of the self-object.

[0014] The method may further include: executing, by a processor, an analysis protocol to determine a node score for each interaction node of a hierarchical node graph; comparing, by the processor, the node score with a threshold; and removing, by the processor, from the hierarchical node graph each interaction node of the hierarchical node graph that corresponds to a node score less than the threshold.

[0015] The method may further include: in response to determining that adding a second interaction node to a subsequent layer of interaction nodes in a plurality of subsequent layers causes the number of nodes in the hierarchical node graph to exceed a threshold, removing, by the processor, a third node from the hierarchical node graph based on the node score for the third node.

[0016] In another embodiment, an autonomous object may include: a camera; a processor; and a non-transitory computer-readable medium configured to be executed by the processor. The processor may be configured to use image data captured by the camera to detect one or more proxy objects in a space around the autonomous object; store a hierarchical node graph that includes: a destination layer that includes one or more destination nodes corresponding to a destination for autonomous object implementation; a plurality of interaction layers of interaction nodes after the destination layer, each interaction node corresponding to at least one of a plurality of trajectories for the autonomous object in view of one or more proxy objects and corresponding to a node score, the plurality of interaction layers including an initial interaction layer of interaction nodes and a plurality of subsequent interaction layers of interaction nodes after the initial interaction layer of interaction nodes, wherein each interaction node in the plurality of subsequent interaction layers depends on at least one interaction node in a previous interaction layer in the plurality of interaction layers; in response to determining that a node score of a first interaction node in the initial interaction layer of interaction nodes exceeds a threshold, adding a second interaction node to a subsequent interaction layer of interaction nodes in the plurality of subsequent interaction layers, the second interaction node being linked to the first interaction node; determining a trajectory score for each of the plurality of trajectories based on one or more node scores of one or more nodes corresponding to the trajectories; and selecting, for the autonomous object, a trajectory from the plurality of trajectories based on the trajectory score for the trajectory.

[0017] The processor may further be configured to control the autonomous object according to the selected trajectory.

[0018] The processor may be configured to determine a trajectory score for a trajectory by aggregating one or more node scores of one or more nodes of the trajectory.

[0019] The processor may be configured to select a trajectory by selecting a trajectory in response to determining that the trajectory score for the trajectory is higher than the trajectory scores of other trajectories in the plurality of trajectories.

[0020] The processor may further be configured to execute a neural network to determine a node score for each interaction node of the hierarchical node graph.

[0021] The node score for each interaction node can correspond to the comfort level associated with the interaction node.

[0022] The node score for each interaction node can correspond to the comfort level associated with the interaction node and the likelihood of intervention associated with the interaction node.

[0023] The processor can also be configured to generate a hierarchical node map in response to detecting one or more proxy objects in the space around the self-object using image data captured by a camera of the self-object.

[0024] The processor can also be configured to execute an analysis protocol to determine a node score for each interaction node of the hierarchical node map; compare the node score with a threshold; and remove from the hierarchical node map each interaction node of the hierarchical node map corresponding to a node score less than the threshold.

[0025] The processor can also be configured to, in response to determining that adding a second interaction node to a subsequent layer of interaction nodes in a plurality of subsequent layers causes the number of nodes in the hierarchical node map to exceed a threshold, remove a third node from the hierarchical node map based on the node score for the third node. Description of the Drawings

[0026] Non-limiting embodiments of the present disclosure are described by way of example in connection with the accompanying drawings, which are schematic and not intended to be drawn to scale. Unless indicated as representing the background art, the drawings represent various aspects of the present disclosure.

[0027] Figure 1A Illustrates components of an AI-enabled visual data analysis system according to an embodiment.

[0028] Figure 1B Illustrates various sensors associated with the self according to an embodiment.

[0029] Figure 1C Illustrates components of a vehicle according to an embodiment.

[0030] Figure 2 Illustrates a flowchart of a process performed in an AI-enabled visual data analysis system according to an embodiment.

[0031] Figure 3 Illustrates a roadway scene according to an embodiment.

[0032] Figures 4A to 4E Illustrates a hierarchical node map for a roadway scene according to an embodiment. Detailed Description

[0033] Reference will now be made to the illustrative embodiments depicted in the accompanying drawings, and specific language will be used here to describe them. However, it is to be understood that no limitation of the scope of the claims or of the present disclosure is thereby intended. Changes and further modifications of the features of the invention described herein, as well as additional applications of the principles of the subject matter described herein, which would be apparent to those of ordinary skill in the relevant art and having the present disclosure in hand, will be considered to be within the scope of the subject matter disclosed herein. Other embodiments may be used and / or other changes may be made without departing from the spirit or scope of the present disclosure. The illustrative embodiments described in the detailed description are not meant to be limiting of the subject matter presented.

[0034] An entity driving on a road (e.g., an autonomous vehicle such as a car, truck, bus, motorcycle, all-terrain vehicle, cart, robot, or other automated device) must continuously monitor the entity's surroundings for other objects on the road. The entity (e.g., the entity's processor) can detect different objects such as pedestrians, other vehicles, or animals, and encounter scenarios where the entity must drive around an object to reach a desired destination or desired end, such as when a pedestrian is crossing the road and the entity turns left into the rightmost lane. In such a scenario, the entity can navigate around the pedestrian by determining different potential trajectories for the pedestrian and any other objects in the area, as well as a potential trajectory for the entity. The entity can analyze each of the potential trajectories in the potential trajectories using an optimization function to determine a trajectory for the entity to achieve the goal of turning left into the rightmost lane while avoiding the pedestrian. Given the large number of variables involved, this may require a large amount of processing power and time to make a decision. The entity can perform such a decision multiple times per second (e.g., every 50 milliseconds) to autonomously navigate the road. Thus, an autonomous entity can use a large amount of processing power and, consequently, energy and time, to make decisions about how to navigate the roadway. Over time, using such processing power can cause the entity to be delayed in reaching its destination and use an increasing amount of energy.

[0035] An autonomous vehicle or system implementing the systems and methods described herein can overcome the above-described technical deficiencies. For example, a processor of the system or autonomous vehicle can implement a hierarchical node graph that includes different node layers of a scene encountered by the autonomous vehicle. The hierarchical node graph can include goal nodes corresponding to goals for the autonomous vehicle to achieve (such as avoiding pedestrians, reaching a target lane or parking space, etc.) and interaction nodes corresponding to interactions and / or non-interactions between the autonomous vehicle and agent objects (such as objects that are moving or can move in the environment around the autonomous vehicle). Examples of interactions can include the autonomous vehicle passing before or after or contacting an agent object, but an interaction does not require contact with the agent object. Examples of non-interactions can include the autonomous vehicle performing an action to completely avoid an agent object. The processor can identify such nodes in the hierarchical node graph for the autonomous vehicle and agent objects detected by the autonomous vehicle using its camera. The processor can determine a score for each of the interaction nodes and / or goal nodes based on intervention likelihood (such as human intervention likelihood), a humanoid discriminator, and / or physics-based constraints of the corresponding interaction node or goal node. The processor can identify different trajectories in the hierarchical node graph, each trajectory including goal nodes and one or more interaction nodes in sequential order within the hierarchical node graph. The processor can determine a trajectory score for each trajectory. The processor can select the trajectory corresponding to the highest trajectory score. The processor can use the selected trajectory to control the autonomous vehicle. In this way, the autonomous vehicle can determine an optimal trajectory for itself without determining the trajectories of objects in the surrounding environment or using complex cost functions, thereby reducing the processing cost and time for selecting a trajectory for autonomous driving.

[0036] The hierarchical node graph can have hierarchical node layers. The processor can use the hierarchical configuration of the layers to determine how to traverse the hierarchical node graph to determine potential trajectories for controlling the autonomous vehicle. For example, the hierarchical node graph can include a goal layer that includes one or more goal nodes. Each goal node can be a start for one or more trajectories for the autonomous vehicle. Next, the hierarchical node graph can include multiple interaction layers, an initial interaction layer, and one or more subsequent interaction layers. Each interaction layer can include one or more interaction nodes. The interaction nodes of the initial interaction layer can each be linked to at least one goal node. The interaction nodes of the first subsequent interaction layer can each be linked to at least one interaction node in the initial interaction layer. The interaction nodes in subsequent interaction layers up to the first subsequent interaction layer can each be linked to at least one interaction node in a previous interaction layer. The processor can use the links between the nodes to traverse different trajectories and combine the scores of the interaction nodes and / or goal nodes of the trajectories to select a trajectory for the autonomous vehicle.

[0037] To avoid generating a hierarchical node graph that is too large and may require substantial computational resources to maintain and evaluate different trajectories, the processor may prune the hierarchical node graph over time. For example, the processor may compare the node scores of the nodes in the hierarchical node graph with a threshold. In response to determining that the node score of a node is less than the threshold, the processor may remove that node from the hierarchical node graph. In one example, the processor may remove nodes with undesirable characteristics, such as the self that contacts an object, to avoid expending processing resources on trajectories that the processor would not select. The processor may remove such nodes and over time expand to other nodes with higher scores to maintain a hierarchical node graph that the processor can use for efficient control of the self.

[0038] Figure 1A is a non-limiting example of a component of a system in which the methods and systems discussed herein may be implemented. Figure 1A Illustrates components of an artificial intelligence (AI)-enabled visual data analysis system 100. The system 100 may include an analysis server 110a, a system database 110b, an administrator computing device 120, selves 140a to 140b (collectively referred to as (multiple) selves 140), self computing devices 141a to 141c (collectively referred to as self computing devices 141), and a server 160. The system 100 is not limited to the components described herein and may include additional or other components not shown for the sake of brevity, which will be considered within the scope of the embodiments described herein.

[0039] The above components may be connected via a network 130. Examples of the network 130 may include, but are not limited to, a private or public LAN, WLAN, MAN, WAN, and the Internet. The network 130 may include wired and / or wireless communication according to one or more standards and / or via one or more transmission media.

[0040] Communication over the network 130 may be performed according to various communication protocols, such as the Transmission Control Protocol and Internet Protocol (TCP / IP), User Datagram Protocol (UDP), and IEEE communication protocols. In one example, the network 130 may include wireless communication according to the Bluetooth specification set or another standard or proprietary wireless communication protocol. In another example, the network 130 may also include communication over a cellular network, including, for example, a GSM (Global System for Mobile Communications), CDMA (Code Division Multiple Access), or EDGE (Enhanced Data for Global Evolution) network.

[0041] The system 100 illustrates an example of a system architecture and components that may be used to implement one or more AI models, such as (multiple) AI models 110c and a hierarchical node graph 110d. Specifically, as Figure 1AAs depicted and described herein, the analytics server 110a can use the methods discussed herein to generate and use a hierarchical node map 110d for autonomous navigation using data retrieved from the vehicle 140 (e.g., by using data streams 172 and 174). In one example, the AI model(s) 110c can detect the occupancy of different voxels representing regions surrounding the vehicle 140 based on image data captured by the cameras of the vehicle 140. Based on the occupancy data, the analytics server 110a or the vehicle 140 can detect different proxy objects (e.g., moving objects) in the region surrounding the vehicle 140. The vehicle 140 or the analytics server 110a can use the detected proxy objects to generate the hierarchical node map 110d to include destination nodes and / or interaction nodes. The destination nodes can indicate the respective destinations to be achieved by the vehicle 140, and the interaction nodes can indicate potential interactions or non-interactions between the vehicle 140 and the proxy objects or between the proxy objects themselves. The vehicle 140 or the analytics server 110a can generate node scores for the nodes of the hierarchical node map 110d. The vehicle 140 can use the node scores to generate trajectory scores for different trajectories, which include different variations of linked nodes within the hierarchical node map 110d. The vehicle 140 can select a trajectory based on the trajectory scores of the trajectories (e.g., based on the trajectory having the highest trajectory score). The vehicle 140 can operate autonomously according to the selected trajectory. Thus, the system 100 depicts a navigation method using a hierarchical node map, which is faster and requires fewer processing resources than conventional trajectory generation and / or selection methods.

[0042] In Figure 1A , the AI model 110c and the hierarchical node map 110d are illustrated as components of the system database 110b, but the AI model 110c and the hierarchical node map 110d can be stored in different or separate components, such as a cloud storage device or any other data repository accessible to the analytics server 110a or the vehicle 140.

[0043] The analytics server 110a can also be configured to display an electronic platform that illustrates various training attributes for training the AI model 110c. The electronic platform can be displayed on the administrator computing device 120 such that an analyst can monitor the training of the AI model 110c. An example of an electronic platform generated and hosted by the analytics server 110a can be a web-based application or website configured to display the training data set collected from the vehicle 140 and / or the training status / metrics of the AI model 110c.

[0044] The analysis server 110a can be any computing device including a processor and a non-transitory machine-readable storage device capable of performing the various tasks and processes described herein. Non-limiting examples of such computing devices can include workstation computers, laptop computers, server computers, etc. Although the system 100 includes a single analysis server 110a, the system 100 can include any number of computing devices operating in a distributed computing environment such as a cloud environment.

[0045] The ego 140 can represent various electronic data sources that send data associated with their previous or current navigation sessions to the analysis server 110a. The ego 140 can be any device configured for navigation, such as the vehicle 140a and / or the truck 140c. The ego 140 is not limited to being a vehicle and can also include robotic devices. For example, the ego 140 can include a robot 140b, which can represent a general-purpose, bipedal, autonomous humanoid robot capable of navigating various terrains. The robot 140b can be equipped with software for achieving balance, navigation, perception, or interacting with the physical world. The robot 140b can also include various cameras configured to send visual data to the analysis server 110a.

[0046] Although referred to herein as the "ego", the ego 140 may or may not be an autonomous device configured for autonomous navigation. For example, in some embodiments, the ego 140 can be controlled by a human operator or a remote processor. The ego 140 can include various sensors, such as Figure 1B the sensors depicted in. The sensors can be configured to collect data as the ego 140 navigates various terrains (e.g., roads). The analysis server 110a can collect the data provided by the ego 140. For example, the analysis server 110a can obtain navigation session and / or road / terrain data (e.g., images of the ego 140 navigating on a road) from various sensors, such that the collected data is ultimately used by the AI model 110c for training purposes.

[0047] As used herein, a navigation session corresponds to the journey of the ego 140's travel route, regardless of whether the journey is autonomous or human-controlled. In some embodiments, the navigation session can be used for data collection and model training purposes. However, in some other embodiments, the ego 140 can refer to a vehicle purchased by a consumer, and the purpose of the journey can be classified as daily use. A navigation session can start when the ego 140 moves from a non-moving position by more than a threshold distance (e.g., 0.1 mile, 100 feet) or at a rate exceeding a threshold (e.g., exceeding 0 mph, exceeding 1 mph, exceeding 5 mph). A navigation session can end when the ego 140 returns to a non-moving position and / or shuts down (e.g., when the driver leaves the vehicle).

[0048] The self 140 can represent a set of selves monitored by the analysis server 110a to generate the hierarchical node graph 110d. For example, a driver of the vehicle 140a can authorize the analysis server 110a to monitor data associated with their corresponding vehicle. As a result, the analysis server 110a can use the various methods discussed herein to collect sensor / camera data and detect proxy objects in the environment of the self 140 from which the sensor / camera data is collected. The analysis server 110a can gradually construct the hierarchical node graph 110d from the data by adding destination nodes and interaction nodes that are linked to each other in a sequence of layers from different scenarios, where the self 140 provides data to these scenarios. The analysis server 110a can generate node scores for different nodes and prune out nodes with low node scores or otherwise node scores below a threshold. The analysis server 110a can deploy or send the hierarchical node graph to different selves 140 for autonomous driving.

[0049] Over time, the self 140 can send additional data about different scenarios to the analysis server 110a. The analysis server 110a can update the hierarchical node graph 110d based on the data over time and send an updated version of the hierarchical node graph 110d to the self 140 for autonomous driving. Thus, the system 100 depicts a loop in which the navigation data received from the self 140 can be used to update the hierarchical node graph 110d. The self 140 can include a processor that processes the hierarchical node graph 110d for navigation purposes (e.g., selecting a trajectory for navigation).

[0050] The self 140 can be equipped with various technologies that allow the self to collect data from its surrounding environment and (possibly) navigate autonomously. For example, the self 140 can be equipped with an inference chip to run autonomous driving software.

[0051] The various sensors for each self 140 can monitor the data collected associated with different navigation sessions and send the data to the analysis server 110a. Figures 1B to 1C A block diagram of sensors integrated within the self 140 according to an embodiment is illustrated. The quantity and location of each sensor discussed with respect to Figures 1B to 1C can depend on the type of self discussed in Figure 1A . For example, the robot 140b can include different sensors than the vehicle 140a or the truck 140c. For example, the robot 140b may not include an airbag activation sensor 170q. Also, the sensors of the vehicle 140a and the truck 140c can be positioned differently than Figure 1C illustrated.

[0052] As discussed herein, the various sensors integrated within each ego 140 can be configured to measure various data associated with each navigation session. The analytics server 110a can periodically collect the data monitored and collected by these sensors, where the data is processed according to the methods described herein and is used to generate the hierarchical node map 110d and / or execute the AI model 110c to generate an occupancy map to detect proxy objects in the space around the ego 140. Moreover, the hierarchical node map 110d and / or the executed AI model 110c can generate trajectory recommendations for the ego 140.

[0053] The ego 140 can include a user interface 170a. The user interface 170a can refer to the user interface of an ego computing device (such as Figure 1A the ego computing device 141 in). The user interface 170a can be implemented as a display screen, a head-up display, a touch screen, etc. integrated with or coupled to the interior of the vehicle. The user interface 170a can include input devices such as a touch screen, a knob, a button, a keyboard, a mouse, a gesture sensor, a steering wheel, etc. In various embodiments, the user interface 170a can be adapted to provide user input (such as as a signal and / or sensor information) to other devices or sensors of the ego 140 (such as Figure 1B the sensors illustrated) (such as the controller 170c).

[0054] The user interface 170a can also be implemented with one or more logic devices that can be adapted to execute instructions, such as software instructions, to implement any of the various processes and / or methods described herein. For example, the user interface 170a can be adapted to form a communication link, send and / or receive communications (such as sensor signals, control signals, sensor information, user input, and / or other information) or execute various other processes and / or methods. In another example, the driver can use the user interface 170a to control the temperature of the ego 140 or activate its features (such as the autonomous driving or steering system 170o). Thus, the user interface 170a can monitor and collect driving session data in combination with other sensors described herein. The user interface 170a can also be configured to display various data generated / predicted by the analytics server 110a and / or the AI model 110c.

[0055] The orientation sensor 170b can be implemented as a compass, a float, an accelerometer, and / or one or more of other digital or analog devices capable of measuring the orientation of the self-body 140 (e.g., the magnitude and direction of roll, pitch, and / or yaw of the self-body 140 relative to one or more reference orientations such as gravity and / or magnetic north). The orientation sensor 170b can be adapted to provide heading measurements to the self-body 140. In other embodiments, the orientation sensor 170b can be adapted to provide roll, pitch, and / or yaw rates to the self-body 140 using a time series of orientation measurements. The orientation sensor 170b can be positioned and / or adapted to make orientation measurements relative to a particular coordinate system of the self-body 140.

[0056] The controller 170c can be implemented as any suitable logic device (e.g., a processing device, a microcontroller, a processor, an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a memory storage device, a memory reader, or other device or combination of devices) that can be adapted to execute, store, and / or receive appropriate instructions, such as software instructions implementing control loops for controlling various operations of the self-body 140. Such software instructions can also implement methods for processing sensor signals, determining sensor information, providing user feedback (e.g., via the user interface 170a), querying the device for operating parameters, selecting operating parameters for the device, or performing any of the various operations described herein.

[0057] The communication module 170e can be implemented as any wired and / or wireless interface configured to transmit sensor data, configuration data, parameters, and / or other data and / or signals to Figure 1A any of the features shown (e.g., the analytics server 110a). As described herein, in some embodiments, the communication module 170e can be implemented in a distributed manner such that portions of the communication module 170e are implemented within Figure 1B one or more of the elements and sensors shown. In some embodiments, the communication module 170e can delay the transmission of sensor data. For example, when the self-body 140 does not have network connectivity, the communication module 170e can store the sensor data in a temporary data storage device and send the sensor data when the self-body 140 is identified as having appropriate network connectivity.

[0058] The speed sensor 170d can be implemented as an electronic pitot tube, a metering gear or wheel, a water speed sensor, a wind speed sensor, a wind rate sensor (e.g., direction and magnitude), and / or other devices capable of measuring or determining the linear speed of the self-body 140 (e.g., in the surrounding medium and / or aligned with the longitudinal axis of the self-body 140) and providing such measurement as a sensor signal that can be transmitted to various devices.

[0059] The gyroscope / accelerometer 170f can be implemented as one or more electronic sextants, semiconductor devices, integrated chips, accelerometer sensors, or other systems or devices capable of measuring the angular velocity / acceleration and / or linear acceleration (e.g., direction and magnitude) of the vehicle body 140 and providing such measurements as sensor signals, which can be transmitted to various devices such as the analysis server 110a. The gyroscope / accelerometer 170f can be positioned and / or adapted to make such measurements with respect to a specific coordinate system of the vehicle body 140. In various embodiments, the gyroscope / accelerometer 170f can be implemented in a common housing and / or module with Figure 1B the other elements depicted to ensure a common reference frame or a known transformation between reference frames.

[0060] The Global Navigation Satellite System (GNSS) 170h can be implemented as a global positioning satellite receiver and / or another device capable of determining the absolute and / or relative position of the vehicle body 140 based on wireless signals received, for example, from space and / or ground sources and capable of providing measurements such as sensor signals, which can be transmitted to various devices. In some embodiments, the GNSS 170h can be adapted to determine the rate, speed, and / or yaw rate of the vehicle body 140 (e.g., using a time series of position measurements), such as the yaw component of the absolute rate and / or angular rate of the vehicle body 140.

[0061] The temperature sensor 170i can be implemented as a thermistor, an electrical sensor, an electrical thermometer, and / or other devices capable of measuring the temperature associated with the vehicle body 140 and providing such measurements as sensor signals. The temperature sensor 170i can be configured to measure the ambient temperature associated with the vehicle body 140, such as the cockpit or dashboard temperature, and for example, this ambient temperature can be used to estimate the temperature of one or more elements of the vehicle body 140.

[0062] The humidity sensor 170j can be implemented as a relative humidity sensor, an electrical sensor, an electrical relative humidity sensor, and / or another device capable of measuring the relative humidity associated with the vehicle body 140 and providing such measurements as sensor signals.

[0063] The steering sensor 170g can be adapted to physically adjust the heading of the vehicle body 140 according to one or more control signals provided by a logic device (such as the controller 170c) and / or user input. The steering sensor 170g can include one or more actuators and control surfaces (e.g., a rudder or other types of steering or trim mechanisms) of the vehicle body 140 and can be adapted to physically adjust the control surfaces to various positive and / or negative steering angles / positions. The steering sensor 170g can also be adapted to sense the current steering angle / position of such a steering mechanism and provide such measurements.

[0064] Propulsion system 170k can be implemented as a propeller, turbine, or other thrust-based propulsion system, mechanical wheeled and / or tracked propulsion system, wind / sail-based propulsion system, and / or other types of propulsion systems that can be used to power the vehicle body 140. Propulsion system 170k can also monitor the direction of the power and / or thrust of the vehicle body 140 relative to the reference coordinate system of the vehicle body 140. In some embodiments, propulsion system 170k can be coupled to sensor 170g and / or integrated with steering sensor 170g.

[0065] The passenger restraint sensor 170l can monitor seat belt detection and locking / unlocking assemblies and other passenger restraint subsystems. The passenger restraint sensor 170l can include various environmental and / or status sensors, actuators, and / or other devices that facilitate the operation of safety mechanisms associated with the operation of the vehicle body 140. For example, the passenger restraint sensor 170l can be configured to receive motion and / or status data from Figure 1B the other sensors depicted. The passenger restraint sensor 170l can determine whether a safety measure (e.g., seat belt) is being used.

[0066] As Figure 1C depicted, the camera 170m can refer to one or more cameras integrated within the vehicle body 140, and can include multiple cameras integrated (or retrofitted) into the vehicle body 140. The camera 170m can be an internal or external camera of the vehicle body 140. For example, as Figure 1C depicted, the vehicle body 140 can include one or more internal cameras that can monitor and collect footage of the passengers of the vehicle body 140. The vehicle body 140 can include eight external cameras. For example, the vehicle body 140 can include a front camera 170m-1, front side cameras 170m-2, 170m-3, rear side cameras 170m-4 on each front fender, cameras 170m-5 on each side (e.g., integrated within the B-pillar), and a rear camera 170m-6.

[0067] Referring to Figure 1B , the radar 170n and ultrasonic sensor 170p can be configured to monitor the distance of the vehicle body 140 to other objects (such as other vehicles or immovable objects (e.g., trees or garage doors)). The vehicle body 140 can also include an autonomous driving or steering system 170o configured to use data collected via various sensors (such as radar 170n, speed sensor 170d, and / or ultrasonic sensor 170p) to navigate the vehicle body 140 autonomously.

[0068] Accordingly, the autonomous driving or steering system 170o can analyze various data collected by one or more of the sensors described herein to identify driving data. For example, the autonomous driving or steering system 170o can calculate the risk of a forward collision based on the speed of the vehicle body 140 and its distance to another vehicle on the road. The autonomous driving or steering system 170o can also determine whether the driver is touching the steering wheel. The autonomous driving or steering system 170o can send the analyzed data to various features discussed herein, such as the analytics server.

[0069] The airbag activation sensor 170q can predict or detect a collision and cause the activation or deployment of one or more airbags. The airbag activation sensor 170q can send data regarding the airbag deployment, including data associated with the event that caused the deployment.

[0070] Referring again to Figure 1A , the administrator computing device 120 can represent a computing device operated by a system administrator. The administrator computing device 120 can be configured to display data retrieved or generated by the analytics server 110a (such as various analytics metrics and risk scores), where the system administrator can monitor the various models utilized by the analytics server 110a, review feedback, and / or facilitate the training of the (multiple) AI models 110c maintained by the analytics server 110a and / or the individual vehicle bodies 140 and / or the generation of the hierarchical node graph 110d.

[0071] (Multiple) vehicle bodies 140 can be any device configured to navigate various routes, such as the vehicle 140a or the robot 140b. As discussed with respect to Figures 1B to 1C , the vehicle body 140 can include various telemetry sensors. The vehicle body 140 can also include a vehicle body computing device 141. Specifically, each vehicle body can have its own vehicle body computing device 141. For example, the truck 140c can have a vehicle body computing device 141c. For the sake of brevity, the vehicle body computing devices are collectively referred to as (multiple) vehicle body computing devices 141. The vehicle body computing device 141 can control the content presentation on the infotainment system of the vehicle body 140, process commands associated with the infotainment system, aggregate sensor data, manage the communication of data to electronic data sources, receive updates, and / or send messages. In one configuration, the vehicle body computing device 141 communicates with the electronic control unit. In another configuration, the vehicle body computing device 141 is the electronic control unit. The vehicle body computing device 141 can include a processor and a non-transitory machine-readable storage medium capable of performing the various tasks and processes described herein. For example, the (multiple) AI models 110c described herein can be stored and executed (or directly accessed) by the vehicle body computing device 141. Non-limiting examples of the vehicle body computing device 141 can include vehicle multimedia and / or display systems.

[0072] In one example of how the self's computing device 141 of the self 140 can generate and / or use the hierarchical node map 110d for navigation, when the self's computing device 141 controls the self 140 for autonomous driving, the cameras of the self 140 can generate image data of the space around the self 140. The self's computing device 141 can execute the AI model(s) 110c to automatically detect proxy objects in the space, such as by generating and analyzing different voxels representing the space for occupancy characteristics from the image data. The self's computing device 141 can determine a task for the self 140 to perform, such as making a left turn. In response to determining the task, the self's computing device 141 can generate or use the hierarchical node map 110d to determine a trajectory to be used to control the self 140 to perform the task.

[0073] For example, to generate the hierarchical node map 110d, the self's computing device 141 can generate one or more destination nodes corresponding to the task. The destination nodes can correspond to different purposes or goals for the task, such as making a left turn, safely making a left turn, avoiding exceeding a specific speed, etc. The self's computing device 141 can generate one or more interaction nodes. The interaction nodes can correspond to different interactions or non-interactions between the self 140 and the proxy objects detected in the space around the self 140. For example, the self's computing device 141 can determine how the self 140 can interact with the proxy object by determining the identity or classification of the proxy object (e.g., pedestrian or vehicle), the current position of the proxy object, the current position of the proxy object relative to the target position of the task, the current position of the proxy object relative to the self 140, and / or the current state of the proxy object (e.g., moving or not moving, moving speed, moving direction, size, etc.). The self's computing device 141 can use machine learning techniques, use one or more functions, or query a memory with sensor data generated about the proxy object to determine such identity or classification and characteristics of the proxy object. The self's computing device 141 can use the determined characteristics to determine different interactions that can occur by using a machine learning model, by using one or more functions, or by querying a memory. The self's computing device 141 can generate interaction nodes for the interactions or non-interactions that can occur when attempting to achieve the corresponding purposes of the destination nodes. The self's computing device 141 can link the interaction nodes to the destination nodes on which the interaction nodes depend accordingly.

[0074] The self-computing device 141 can determine node scores for the nodes of the hierarchical node graph 110d. The self-computing device 141 can do so for the data of the respective nodes using a function or machine learning techniques. For example, the self-computing device 141 can input the data of the respective interaction nodes into a machine learning model (such as a neural network, a support vector machine, a random forest, etc.) and execute the machine learning model for each interaction node. The machine learning model can output a node score for the interaction node based on the execution. The self-computing device 141 can store the score in the respective node for which the score was generated.

[0075] The machine learning model can be trained to output a node score based on factors such as comfort, physics-based constraints (such as the likelihood of an impact), humanoid discriminators (such as the likelihood that a human would perform the same action), and / or intervention likelihood. The machine learning model can be trained to do so, for example, using a tagging technique that indicates scores for the respective factors. The machine learning model can be trained to aggregate the scores for the respective nodes to generate a node score for the node. In some cases, multiple machine learning models can be used to generate scores for individual factors for each node of a hierarchical data structure.

[0076] The self-computing device 141 can expand the hierarchical node graph 110d over time. The self-computing device 141 can do so based on the scores of the nodes of the hierarchical node graph 110d. For example, the self-computing device 141 can identify any interaction nodes having a node score less than a threshold (such as at least one node score). The self-computing device 141 can determine not to expand on the identified interaction nodes and accordingly insert a label into such interaction nodes or memory, or remove the low-scoring nodes from the hierarchical node graph.

[0077] For an interaction node with a node score exceeding a threshold, the self - computing device 141 can determine another interaction that depends on (e.g., after or relying on) a transaction. In one example, if a pedestrian is crossing a road, the interaction of the initial interaction node can be to turn left after the pedestrian crosses the road. However, the self - computing device 141 can detect an oncoming vehicle that may be traveling in the same direction as the space that the self 140 is about to enter. Thus, the self - computing device 141 can generate an interaction node that is linked to the initial interaction node corresponding to letting the pedestrian cross the road but corresponding to letting the oncoming vehicle pass. The self - computing device 141 can generate another interaction node that is linked to the initial interaction node but corresponding to passing in front of the oncoming vehicle. The self - computing device 141 can also generate an interaction node for not interacting with the pedestrian and the oncoming vehicle, such as waiting for the pedestrian and the oncoming vehicle to leave the space or turning in a direction (e.g., turning right) where the self 140 will not interact with the pedestrian or the oncoming vehicle. The self - computing device 141 can repeat the node scoring and expansion process over time for any number of proxy objects in the space around the self 140.

[0078] The self - computing device 141 can generate trajectory scores for different trajectories outlined by the hierarchical node graph 110d. For example, the hierarchical node graph 110d can include one or more trajectories, each starting with a destination node and including interaction nodes linked to each other and to the destination node. In some cases, the destination node may not be included in the trajectory. The self - computing device 141 can determine the trajectory score for each trajectory based on the node scores of the nodes that make up the corresponding trajectory. The self - computing device 141 can retrieve the node scores from the corresponding nodes and perform a function (e.g., using aggregation or summation techniques, determining an average or weighted average, determining a median, etc.), or use machine - learning techniques on the retrieved node scores to generate the trajectory score for the corresponding trajectory. In some cases, the self - computing device 141 can determine multiple trajectory scores for each trajectory based on the node scores for different factors.

[0079] The self - computing device 141 can select a trajectory from the trajectories based on the trajectory scores. The self - computing device 141 can compare the trajectory scores together to determine the trajectory with a trajectory score that meets the condition. For example, the self - computing device 141 can identify the trajectory corresponding to the highest trajectory score. In another example, the self - computing device 141 can determine a combination of trajectory scores for the trajectories that meet the condition (e.g., the combination of trajectory scores closest to the solution space for the combined factors) or the combination of trajectory scores with the highest average, weighted average, sum, weighted sum, or median. The self - computing device 141 can control the self 140 according to the trajectory identified or selected from the hierarchical node graph 110d.

[0080] In the case where the analysis server 110a generates the hierarchical node graph 110d, the analysis server 110a can use techniques similar to those of the self-computing device 141 to generate the hierarchical node graph 110d. The analysis server 110a can do so using data from multiple selves 140 for different scenarios. In some cases, the analysis server 110a can remove (e.g., indicate or mark not to perform further calculations on the nodes of the trajectory itself or remove the nodes) low-scoring trajectories from the hierarchical node graph 110d so that the self 140 does not waste processing resources determining whether to implement low-scoring trajectories. The analysis server 110a can send the hierarchical node graph 110d to the self 140, and the self 140 can use the hierarchical node graph 110d by identifying interaction nodes and / or proxy objects corresponding to or matching the scenarios faced by the self 140. The self 140 can update the hierarchical node graph 110d with new interaction nodes and / or destination nodes based on the data collected by the self 140 for each scenario, and thus update the new trajectories.

[0081] Figure 2 FIG. illustrates a flowchart of a method 200 performed in an AI-enabled visual data analysis system according to an embodiment. Method 200 may include steps 202 to 210. However, other embodiments may include additional or alternative steps, or one or more steps may be omitted. Method 200 may be performed by a self-computing device (e.g., a computer similar to the self-computing device 141 or a processor of the self 140). However, one or more steps of method 200 may be performed by any number of computing devices (e.g., the analysis server 110a) operating in a Figures 1A to 1C distributed computing system as described. For example, one or more computing devices of the self may perform Figure 2 some or all of the steps described.

[0082] Usage method 200, the autonomous computing device can implement a node hierarchical data structure for trajectory selection for the self. To this end, the autonomous computing device can detect one or more proxy objects (e.g., objects that are moving or can move) in the space or environment around the self (e.g., the self-object). The autonomous computing device can store a node data structure that includes a destination node indicating the purpose for self-implementation and / or interaction nodes corresponding to different potential interactions and / or non-interactions between the self and the proxy objects. The autonomous computing device can determine the trajectory scores of different trajectories for following different permutations of the destination node and the interaction nodes based on the node scores of the trajectory nodes. The trajectory scores can correspond to the comfort level, physics-based constraints, intervention likelihood, and / or discriminator (e.g., humanoid discriminator) of the corresponding trajectories. The autonomous computing device can compare the trajectory scores to identify the highest trajectory score. The autonomous computing device can select the trajectory with the highest trajectory score. The autonomous computing device can use the selected trajectory to control the self.

[0083] At step 202, the autonomous computing device detects one or more proxy objects in the space around the self-object. The autonomous computing device can use the image data captured by the camera of the self-object to detect one or more proxy objects. For example, the camera of the self-object can generate image data (e.g., images or videos) of the environment around the self-object over time and / or when the self-object is driving. The autonomous computing device can process the image data when the camera uses object recognition technology to generate the images, such as by executing a machine learning model or an artificial intelligence model, to detect different objects in the image data. The autonomous computing device can identify the positions of the detected objects relative to the self-object. In one example, the autonomous computing device can detect objects in the image data by detecting the occupancy of different voxels representing the space around the self-object.

[0084] An autonomous computing device can determine the type of an object detected by the autonomous computing device. For example, the type of the object can be a stationary object and an agent object. The autonomous computing device can use a lookup technique in a memory to determine the type of the object. For example, the autonomous computing device can detect an object from image data. In response to detecting the object, the autonomous computing device can use a lookup in the memory to match the detected object with an object stored in the memory. The object stored in the memory can have stored an association with the object type. The autonomous computing device can determine the type of the detected object based on the match with the object stored in the memory. In some cases, a machine learning model or an artificial intelligence model that the autonomous computing device executes to detect the object can additionally determine the type of the object. In some cases, the autonomous computing device can determine the identity or classification of the object (e.g., determine whether the object is a sign or a pedestrian), and based on this determination, determine the type of the object (e.g., use a lookup in the memory for the identity or classification). The autonomous computing device can detect the object and determine the type of the object in any way.

[0085] At step 204, the autonomous computing device stores a hierarchical node graph. The hierarchical node graph can include a purpose layer that includes one or more purpose nodes corresponding to purposes for autonomous object implementation. The hierarchical node graph can also include one or more interaction layers of interaction nodes after the purpose layer. Each interaction node can correspond to an interaction or non - interaction between the autonomous object and at least one agent object (e.g., a pedestrian, a passing car, etc.) detected by the autonomous computing device in the space around the autonomous object. The purpose nodes can correspond to purposes for the autonomous object to accomplish (e.g., steering to the left - most lane, avoiding hitting a pedestrian, etc.). Each node can be a data structure (e.g., a table or a Strapi model) that stores data specific to that node. The multiple interaction layers can include an initial interaction layer of interaction nodes and / or one or more subsequent interaction layers of interaction nodes after the initial layer of interaction nodes.

[0086] Nodes in adjacent layers within the hierarchy of the hierarchical node graph can be linked to one or more nodes in a previous and / or subsequent layer within the hierarchy (e.g., store identifiers of the linked nodes). For example, a destination node in a destination layer can be separately linked to at least one interaction node in an initial interaction layer; each interaction node in the initial interaction layer can be linked to at least one interaction node in a first subsequent interaction layer of the hierarchical nodes, etc. The link can indicate a dependency or causality between the nodes. For example, an initial interaction node (e.g., an interaction node in the initial interaction layer) can be linked to a subsequent interaction node (e.g., an interaction node in a subsequent interaction layer). The initial interaction node can correspond to waiting for a pedestrian to pass before performing a left turn. The subsequent interaction node can be waiting for oncoming vehicles to pass before performing a left turn. The subsequent interaction node may depend on the initial interaction node because the interaction of the subsequent interaction node is only possible if the interaction of the initial interaction node occurs first. In some cases, the dependency can indicate an order property of the interaction. For example, the interaction of the subsequent interaction as described above can occur after the interaction of the interaction node. The hierarchical node graph can include any number of interaction nodes in different interaction layers, which are linked to interaction nodes in other interaction layers of the hierarchical node graph in this way.

[0087] The link between a destination node in the destination layer and an interaction node in the initial interaction layer can have a dependency similar to the dependency between interaction nodes. For example, the purpose can be to perform a left turn. Compared to the purpose of turning in another direction, performing a left turn can enable different interactions with different agent objects (e.g., vehicles or pedestrians). Therefore, the interaction nodes linked to the destination node for making a left turn can take into account the agent objects that may be affected by whether and / or how the self-object makes a left turn or that affect whether and / or how the self-object makes a left turn.

[0088] An interaction node can also correspond to the lack of interaction with one or more other agent objects. For example, a destination node can correspond to the purpose of making a left turn. The self-computing device can detect a pedestrian on the lane for turning and another vehicle approaching from the right. There can be interaction nodes in the hierarchical node graph for the inactivity of the self-object, such as turning right to avoid the pedestrian and the car, or remaining stationary until the path is clear. Each such interaction node can be an interaction node in the hierarchical node graph.

[0089] In some cases, the self-computing device can generate a hierarchical node graph. The self-computing device can generate a hierarchical node in response to determining at least one task to be completed for the self-object (e.g., determining a left turn to follow a predetermined or configured path). The task can be a purpose. The self-computing device can further generate a hierarchical node graph in response to detecting one or more agent objects in the space around the self-object.

[0090] For example, in response to determining that a goal has been completed and / or detecting one or more proxy objects in the space around the ego object, the ego computing device may generate one or more goal nodes for the goal layer of the hierarchical node graph. The ego computing device may generate one or more goal nodes by querying memory for different goals to be completed by the ego computing device based on the task. Examples of such goals for a left turn task may be to avoid hitting any pedestrians, avoid hitting any other vehicles, ensure that the turn is not too sharp, ensure that the speed of the ego object remains below a threshold, etc. The ego computing device may generate one or more goal nodes by storing or allocating data structures for the respective goal nodes in memory. The ego computing device may populate the data structures for different goal nodes with information about the goals of the corresponding goal nodes, such as the identification of the goal and any metadata about the goal (e.g., a node score for the goal node).

[0091] The ego computing device may generate one or more interaction nodes for each of one or more interaction layers of the hierarchical node graph. The ego computing device may generate interaction nodes based on the potential interactions and / or non - interactions determined by the ego computing device between the ego object and the detected proxy objects. The ego computing device may determine such interactions and / or non - interactions for each goal node among the goal nodes. For example, for a goal node associated with the goal of turning left into a far lane, the ego computing device may determine potential interactions or non - interactions with a pedestrian crossing the street in the lane and another vehicle approaching in the same lane. For example, the ego computing device may determine potential interactions by querying memory with the location and / or identification of what the object is, and identify the potential interactions corresponding to the scenario. In some cases, the ego computing device may execute a machine learning model (e.g., a neural network, a support vector machine, a random forest, etc.) to identify different potential interactions. The ego computing device may similarly query memory or execute a machine learning model to determine which proxy objects may participate in the potential interactions for the ego to achieve the goal. In some cases, the ego computing device similarly determines potential interactions with the identified proxy objects without first determining which proxy objects may participate in the potential interactions with the ego object for performing the goal. The ego computing device may generate interaction nodes that link the interaction nodes to the goal nodes in the hierarchical node graph. The ego computing device may include interaction identification and metadata about the interaction in each interaction node linked to the goal node (e.g., interaction type, node score of the interaction node for the interaction, speed of one or each of the objects in the interaction node, etc.). The ego computing device may link any number of interaction nodes to the goal node.

[0092] An autonomous computing device may sequentially link interaction nodes of different interaction layers that depend on each other. For example, one interaction with an agent object may only be able to occur after another interaction with the same agent object or a different agent object. Thus, the autonomous computing device may link subsequent interactions with earlier interactions in separate interaction layers of a hierarchical node graph. For example, an autonomous object may let a pedestrian cross a street and then let another vehicle pass by the autonomous object. The autonomous computing device may link an interaction node for letting the pedestrian pass to an interaction node for letting the vehicle pass in adjacent layers of the hierarchical node graph. The autonomous computing device may link any number of interaction nodes to respective interaction nodes in any number of layers. Thus, the autonomous computing device may generate a hierarchical node graph taking into account the different interactions or non-interactions that may occur and the purposes for which the autonomous object is implemented.

[0093] In some cases, the hierarchical node graph may be generated by a server (such as analytics server 110a) and deployed on the autonomous object. For example, the server may receive image data from different autonomous objects in an autonomous driving scenario, where the scenarios involve different agent objects around the autonomous objects within the scenario. The server may receive data for the autonomous driving scenario and generate a hierarchical node graph (such as a single hierarchical node graph) that covers each scenario in the scenario having purpose nodes for the purposes of the scenario and interaction nodes for interacting and non-interacting with the agent objects of the scenario. When the server receives data for a scenario from an autonomous object, the server may generate the hierarchical node graph by adding purpose nodes and interaction nodes for different scenarios. The server may avoid duplication of purpose nodes or interaction nodes, for example, by analyzing the hierarchical node graph for the same type of nodes (such as purpose nodes or interaction nodes) before adding a new node to the hierarchical node graph and adding a new node only in response to determining that the same node or a similar node (such as a node having metadata above a threshold) does not already exist in the hierarchical node graph. In some cases, such as to conserve processing resources, the analysis may be performed only on nodes in the same node branch to which the node will be added (such as nodes that are linked to each other). The server may generate such a hierarchical node graph of nodes over time. The server may deploy (such as send as a binary file to respective autonomous objects for use) the hierarchical node graph (such as as a master hierarchical node graph) to the autonomous objects for autonomous driving as described herein. After deployment, the server may continue to receive data and update the hierarchical node graph based on the received data. The server may deploy an updated version of the hierarchical node graph at set intervals in response to receiving an input to do so, or in response to determining that any other condition is met.

[0094] An autonomous computing device can generate node scores for nodes of a hierarchical node graph. The autonomous computing device can generate such node scores for interaction nodes and destination nodes. The autonomous computing device can generate node scores for individual nodes using an analysis protocol (such as a function, algorithm, machine learning model, or artificial intelligence model configured to generate a node score for a node). For example, the autonomous computing device can generate a node score for a node by retrieving data in a node data structure and executing a neural network trained to generate a score for the node. The neural network can output a node score for each node based on the data within the corresponding node. In another example, the autonomous computing device can generate a node score by performing a function (such as sum, median, average, weighted sum, weighted average, etc.) on the values within the node. The autonomous computing device can generate node scores for nodes in any way.

[0095] In some cases, the node score can correspond to factors such as comfort, physics-based constraints (such as collision checking), intervention likelihood (such as the likelihood of human intervention), and / or humanoid discriminators (such as a score indicating that a human driver would make the same decision). For example, when training a neural network (or another machine learning model), the neural network can be trained to generate scores related to each factor or subset of these factors, where a higher value for each factor can correspond to a higher score, and a lower level of the factor can correspond to a lower score. Thus, when the neural network generates a score for a node, the neural network can simulate determining a score representing or corresponding to these factors. In another example, different machine learning models (such as neural networks) or functions can be trained or configured to generate scores for different factors. In such a case, the machine learning model or function can be configured to process a specific type of data or the same type of data from the corresponding node. For each node, different machine learning models or functions can output scores for the corresponding factors. The different machine learning models or functions can process the scores for the factors to generate a node score for the node. The scores for the factors can be the node score. The autonomous computing device can store any node scores generated for the nodes within the corresponding nodes themselves. A server can similarly generate node scores and store them in the nodes of the hierarchical node graph generated by the server.

[0096] In a non-limiting example, now refer to Figure 3, depicts a roadway scenario 300. The roadway scenario 300 includes a host object 302 in lane 304 attempting to turn left into an intersection. In doing so, the host object 302's host computing device may collect image data of the environment around the host object 302. From the image data, the host computing device may detect lane 304 as well as proxy objects 306 and 308. The proxy object 306 may be a pedestrian crossing the road in lane 304. The proxy object 308 may be a vehicle turning right onto the same road as the host object 302 is attempting to turn, behind the pedestrian. The host computing device may generate a hierarchical node graph based on potential interactions with the proxy objects 306 and 308.

[0097] Referring again to Figure 2 , at step 206, the host computing device adds an interaction node (e.g., a second interaction node) to a subsequent layer of the interaction nodes of the hierarchical node graph. The host computing device may perform step 206 when generating the hierarchical node graph, e.g., at step 204, or after generating the hierarchical node graph to update the hierarchical node graph. The host computing device may add an interaction node in response to detecting or determining an interaction with a proxy node in the space around the hierarchical node graph. In one example, the host computing device may add an interaction node after generating the hierarchical node graph and detecting a new proxy node in the space around the host object. The host computing device may add an interaction node by determining one or more interaction nodes on which the new interaction node depends (e.g., based on whether the interaction of the new interaction node is after or based on the interaction of a previous interaction node). The host computing device may add the interaction node to the hierarchical node graph in an interaction layer that follows the interaction layer in which the previous interaction node is located.

[0098] The self-computing device may add an interaction node in response to determining that the node score for the interaction node exceeds a threshold. For example, before adding the interaction node to the hierarchical node graph, the self-computing device may determine the node score for the interaction node. The self-computing device may do so based on the data in the interaction node using the systems and methods described herein. The self-computing device may compare the node score to a threshold (e.g., a defined threshold). In response to determining that the node score exceeds the threshold, the self-computing device may add the interaction node to the hierarchical node graph. Otherwise, the self-computing device may discard the interaction node (e.g., remove the interaction node from memory or otherwise not add the interaction node to the hierarchical node graph), or add the interaction node to a hierarchical node graph with a flag that restricts linking any other interaction nodes to the interaction node. In some cases, the self-computing device may generate node scores for different factors for the interaction node and compare the node scores to thresholds (e.g., the same threshold or different thresholds for each factor). The self-computing device may add the interaction node to the hierarchical node graph in response to determining a number or combination (e.g., a defined number or combination) of scores that exceed the threshold. Otherwise, the self-computing device may discard the interaction node. A server may similarly add interaction nodes to a server-generated hierarchical node graph.

[0099] In some cases, the self-computing device may remove a node from the hierarchical node graph. For example, the self-computing device may identify the node scores of different nodes in the hierarchical node graph. The self-computing device may compare the node scores to a threshold. In response to determining that the node score for a node is less than the threshold, the self-computing device may remove the node from the hierarchical node graph. When removing a node from the hierarchical node graph, the self-computing device may identify any nodes that depend on the removed node in the hierarchical node graph. In some cases, the self-computing device may further remove any such nodes in response to determining that the corresponding removed node does not depend on another node in the hierarchical node graph (e.g., a node in a previous interaction layer or a destination layer). By doing so, the self-computing device may remove undesired nodes and branches from the hierarchical node graph that an administrator may never want to select as part of a selected trajectory, or may prevent a trajectory including the removed node from ever being selected. Thus, the self-computing device may avoid using the processing resources required to store the removed node or the removed branches in the hierarchical node graph storage and / or trajectory selection.

[0100] In another example, the self-computing device can maintain a defined size of the hierarchical node graph. For example, when adding an interaction node to the hierarchical node graph, the self-computing device can determine whether adding the node will cause the hierarchical node graph to have a size (e.g., number of nodes) exceeding a threshold. The self-computing device can instantiate and increment a counter for each node (e.g., each interaction node) of the hierarchical node graph and increment the counter for the node to be added. In response to determining that the new node causes or will cause the hierarchical node graph to have a size exceeding the threshold, the self-computing device can identify the node scores of different nodes (e.g., different interaction nodes) and remove the interaction node with the lowest interaction score or the interaction node with a node score below the threshold, including any nodes that depend on the selected node. The self-computing device can decrement the counter according to the number of removed nodes. Thus, the self-computing device can maintain a constant or consistent size to maintain the hierarchical node graph when the self-computing device detects other agent objects and avoid spike processing requirements during processing.

[0101] In a non-limiting example, referring to Figure 4A to 4F, the self-computing device can generate a hierarchical node graph 400 based on agent objects detected in the environment surrounding the self-object. The hierarchical node graph 400 can include a destination layer 402, a trajectory layer 404, an interaction layer 406, an interaction layer 408, and a destination layer 410. The interaction layer 406 can be the initial interaction layer. The interaction layer 408 can be a subsequent interaction layer. The self-computing device can generate the hierarchical node graph 400 based on a lane 412, occupancy 414, and moving objects (e.g., agent objects) 416 detected by the self-computing device from image data from a camera of the self-object.

[0102] To generate the hierarchical node graph 400, the self-computing device can use a step-based method. For example, the self-computing device can first generate the destination nodes 418, 420, 422, and 424, and add these destination nodes 418, 420, 422, and 424 to the hierarchical node graph 400. The self-computing device can generate node scores for each of the destination nodes 418, 420, 422, and 424. The self-computing device can compare the node scores for the destination nodes 418, 420, 422, and 424 with a threshold. The self-computing device can determine the destination nodes for each of the destination nodes 418, 420, and 422, but cannot determine the destination node for the destination node 424. Thus, the self-computing device can generate the trajectory nodes 426, 428, 430, and 432, which are correspondingly linked to different destination nodes 418, 420, and 422, but not linked to the destination node 424 because the node score for the destination node 424 is below the threshold. The trajectory nodes and trajectory layers may or may not be included in the hierarchical data structure created by the self-computing device when implementing the systems and methods described herein. The trajectory nodes 426, 428, 430, and 432 can correspond to different movements, paths, or trajectories that the self-object can take to achieve the purposes of the destination nodes 418, 420, 422 to which the trajectory nodes 426, 428, 430, and 432 are linked or dependent. The trajectory nodes can store data (such as movement speed and position) of the corresponding trajectories in the trajectory nodes 426, 428, 430, and 432.

[0103] The autonomous computing device can use functions or machine learning models on the data in the trajectory nodes 426, 428, 430, and 432 to generate node scores for the trajectory nodes 426, 428, 430, and 432. The autonomous computing device can compare the node scores of the trajectory nodes 426, 428, 430, and 432 with a threshold. The autonomous computing device can identify the trajectory nodes that exceed the threshold and generate interaction nodes for the initial interaction layer based on the identified trajectory nodes. For example, the autonomous computing device can determine that the node score for the trajectory node 430 exceeds the threshold and, in response to that determination, generate interaction nodes 434 and 436 that are linked to and / or dependent on the trajectory node 430. The interaction for the interaction node 434 can be driving in front of the pedestrian, as illustrated in image 448. The interaction for the interaction node 436 can be letting the pedestrian pass (e.g., yielding to the pedestrian) and turning left after the pedestrian, as illustrated in image 450. The autonomous computing device can determine the node scores for the interaction nodes 434 and 436 and compare the node scores with a threshold (e.g., the same threshold or a different threshold as the threshold used for the node scores of the trajectory nodes 426, 428, 430, and 432). The autonomous computing device can determine that the node score for the interaction node 436 exceeds the threshold, but the node score for the interaction node 434 does not exceed the threshold. Thus, the autonomous computing device can not link any other interaction nodes to the interaction node 434, but can add interaction nodes 440, 442, and 442 that are dependent on (e.g., linked to) the interaction node 436 to the hierarchical node graph 400. The interaction for the interaction node 440 can be driving behind the pedestrian but in front of another vehicle according to the interaction of the interaction node 436, as illustrated in image 452. The interaction for the interaction node 442 can be letting the pedestrian and the vehicle pass (e.g., yielding to the pedestrian and the vehicle) and turning left, as illustrated in image 454. The autonomous computing device can repeat this process for any number of interaction nodes and / or interaction layers.

[0104] In some cases, the autonomous computing device can generate destination nodes that are dependent on the interaction nodes. For example, if the autonomous computing device determines that the occurrence of the interaction of the interaction node 442 will trigger another destination, the autonomous computing device can generate a destination node 446 that is dependent on the interaction node 442. The autonomous computing device can make this determination by determining that the data of the interaction node 442 satisfies a condition stored in the memory. In one example, the interaction can be completing a left turn after letting an agent object pass the autonomous object. The autonomous computing device can analyze the new state after the interaction node 442 and generate a destination node 446 to drive straight on the new lane that the autonomous object is driving on. The new destination can correspond to a new trajectory or initiate repeating method 200 to generate a hierarchical node graph to select a trajectory.

[0105] Referring again toFigure 2 At step 208, the autonomous computing device determines a trajectory score for each of a plurality of trajectories of the hierarchical node graph. A trajectory can be or include a destination node and one or more interaction nodes that are linked to each other between interaction node layers. For example, a trajectory can include nodes corresponding to a scenario where the autonomous object turns left as a destination and the autonomous computing device detects a pedestrian passing through and another vehicle passing by. A trajectory can relate to nodes corresponding to a pedestrian crossing a road and then turning left after a vehicle passes by. Another trajectory in the scenario can be to turn left in front of the pedestrian. Another trajectory in the scenario can be to turn left behind the pedestrian but before the vehicle passes by. Another trajectory can be to completely avoid the pedestrian and the passing vehicle and instead turn right. There can be any number of trajectories corresponding to the nodes of the hierarchical node graph. The autonomous computing device can determine the trajectory score of a trajectory according to or otherwise based on the node scores of the trajectory nodes. For example, the autonomous computing device can perform a function (such as summation or aggregation techniques, median, average, weighted sum, weighted average, etc.) or a machine learning model on the node scores of each trajectory to generate a trajectory score for the corresponding trajectory.

[0106] In some cases, the autonomous computing device can determine multiple trajectory scores for individual trajectories. Different trajectory scores can correspond to different factors (such as physics-based constraints (such as collision checking), comfort analysis, intervention likelihood, humanoid discriminator, etc.). The autonomous computing device can use the above functions or machine learning models on the node scores for the corresponding factors to determine the trajectory scores for these factors. In some cases, the autonomous computing device can combine different trajectory scores for a trajectory to generate a single trajectory score, such as by using a function or a machine learning model to do so.

[0107] In some cases, the autonomous computing device can remove an entire trajectory from the hierarchical node graph. The autonomous computing device can remove individual trajectories from the hierarchical node graph by storing a flag or indication in memory indicating that the trajectory is not to be used or by deleting the nodes of the trajectory. The autonomous computing device can remove a trajectory in response to determining that the trajectory score for the trajectory is below a threshold or in response to determining that the trajectory score meets another condition (such as the lowest trajectory score of the trajectories of the hierarchical node graph). By doing so, the autonomous computing device can reduce the size of the hierarchical node graph or otherwise ensure that processing resources are not wasted on evaluating low-scoring trajectories.

[0108] At step 210, the autonomous computing device selects a trajectory for the autonomous object. The autonomous computing device may select a trajectory for the autonomous object based on the trajectory score of the trajectory determined by the autonomous computing device from the hierarchical node graph. The autonomous object may select a trajectory in response to determining that the trajectory score for the trajectory meets a condition. In one example, the autonomous object may select a trajectory in response to determining that the trajectory score for the trajectory is the highest score among the trajectory scores determined by the autonomous computing device. In response to selecting the trajectory, the autonomous computing device may control the autonomous object according to the selected trajectory.

[0109] The various illustrative logical blocks, modules, circuits, and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, computer software, or combinations of both. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps have been described generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the overall system. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present disclosure or the claims.

[0110] Embodiments implemented in computer software may be implemented in software, firmware, middleware, microcode, hardware description language, or any combination thereof. Code segments or machine-executable instructions may represent a procedure, function, subprogram, program, routine, subroutine, module, software package, class, or any combination of instructions, data structures, or program statements. By passing and / or receiving information, data, arguments, parameters, or memory contents, a code segment may be coupled to another code segment or hardware circuit. Information, arguments, parameters, data, etc. may be passed, forwarded, or transmitted via any suitable means including memory sharing, message passing, token passing, network transmission, etc.

[0111] The actual software code or specialized control hardware used to implement these systems and methods does not limit the claimed features or the present disclosure. Accordingly, the operation and behavior of the systems and methods are described without reference to the specific software code, it being understood that the software and control hardware may be designed to implement the systems and methods based on the description herein.

[0112] When implemented in software, functions can be stored as one or more instructions or code on a non-transitory computer-readable or processor-readable storage medium. The steps of the methods or algorithms disclosed herein can be implemented in processor-executable software modules, which can reside on a computer-readable or processor-readable storage medium. Non-transitory computer-readable or processor-readable media include both computer storage media and tangible storage media, and tangible storage media facilitate the transfer of a computer program from one place to another. A non-transitory processor-readable storage medium can be any available medium accessible by a computer. By way of example and not limitation, such non-transitory processor-readable media can include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other tangible storage medium that can be used to store the desired program code in the form of instructions or data structures and that can be accessed by a computer or a processor. As used herein, disk and optical disk include compact disk (CD), laser disk, optical disk, digital versatile disk (DVD), Blu-ray disk, and floppy disk, where "disk" generally magnetically reproduces data, while "optical disk" optically reproduces data with a laser. Combinations of the above should also be included within the scope of computer-readable media. Additionally, the operations of a method or algorithm can reside as code and / or instructions in one or any combination or collection on a non-transitory processor-readable medium and / or a computer-readable medium, which can be incorporated into a computer program product.

[0113] The foregoing description of the disclosed embodiments is provided to enable a person skilled in the art to make or use the embodiments described herein and their variations. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the principles defined herein can be applied to other embodiments without departing from the spirit or scope of the subject matter disclosed herein. Thus, the disclosure is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the following claims and the principles and novel features disclosed herein.

[0114] Although various aspects and embodiments have been disclosed, other aspects and embodiments are also contemplated. The various aspects and embodiments disclosed are for illustrative purposes and are not intended to be limiting, and the true scope and spirit are indicated by the following claims.

Claims

1. A method, comprising: The processor uses image data captured by a camera of the ego object to detect one or more proxy objects in the space around the ego object; The processor stores a hierarchical node graph, the hierarchical node graph including: A goal layer including one or more goal nodes corresponding to goals to be achieved by the ego object; Multiple interaction layers of interaction nodes after the goal layer, each interaction node corresponding to at least one of a plurality of trajectories for the ego object in view of the one or more proxy objects and corresponding to a node score, the multiple interaction layers including an initial interaction layer of interaction nodes and multiple subsequent interaction layers of interaction nodes after the initial interaction layer of interaction nodes, wherein each interaction node in the multiple subsequent interaction layers depends on at least one interaction node in a previous interaction layer among the multiple interaction layers; In response to determining that the node score of a first interaction node in the initial interaction layer of the interaction nodes exceeds a threshold, the processor adds a second interaction node to a subsequent interaction layer of the interaction nodes in the multiple subsequent interaction layers, the second interaction node being linked to the first interaction node; The processor determines a trajectory score for each of the plurality of trajectories based on one or more node scores of one or more nodes corresponding to the trajectory; and The processor selects a trajectory among the plurality of trajectories for the ego object based on the trajectory score for the trajectory.

2. The method according to claim 1, further comprising: The processor controls the ego object according to the selected trajectory.

3. The method according to claim 1, wherein determining the trajectory score for the trajectory comprises: The processor aggregates the one or more node scores of the one or more nodes of the trajectory.

4. The method according to claim 1, wherein selecting the trajectory comprises: In response to determining that the trajectory score for the trajectory is higher than the trajectory scores of other trajectories among the plurality of trajectories, the processor selects the trajectory.

5. The method according to claim 1, further comprising: The processor executes a neural network to determine the node score for each interaction node of the hierarchical node graph.

6. The method according to claim 1, wherein the node score for each interaction node corresponds to the comfort associated with the interaction node.

7. The method according to claim 1, wherein the node score for each interaction node corresponds to the comfort associated with the interaction node and the likelihood of intervention associated with the interaction node.

8. The method according to claim 1, further comprising: In response to detecting the one or more proxy objects in the space around the ego object using image data captured by a camera of the ego object, the processor generates the hierarchical node graph.

9. The method according to claim 1, further comprising: The processor executes an analysis protocol to determine the node score for each interaction node of the hierarchical node graph; The processor compares the node score with a threshold; And The processor removes each interaction node of the hierarchical node graph corresponding to a node score less than the threshold from the hierarchical node graph.

10. The method according to claim 1, further comprising: In response to determining that adding the second interaction node to the subsequent layer of the interaction nodes in the multiple subsequent layers causes the number of nodes in the hierarchical node graph to exceed a threshold, the processor removes a third node from the hierarchical node graph based on the node score for the third node.

11. An autologous object, comprising: A camera; A processor; And A non-transitory computer-readable medium configured to be executed by the processor, wherein the processor is configured to: Use image data captured by the camera to detect one or more proxy objects in the space around the ego object; A storage hierarchical node graph, the hierarchical node graph including: A destination layer, the destination layer including one or more destination nodes corresponding to destinations for the realization of the self-object; A plurality of interaction layers of interaction nodes after the destination layer, each interaction node corresponding to at least one of a plurality of trajectories for the self-object in view of the one or more agent objects and corresponding to a node score, the plurality of interaction layers including an initial interaction layer of interaction nodes and a plurality of subsequent interaction layers of interaction nodes after the initial interaction layer of interaction nodes, wherein each interaction node in the plurality of subsequent interaction layers depends on at least one interaction node in a previous interaction layer in the plurality of interaction layers; In response to determining that the node score of a first interaction node in the initial interaction layer of the interaction nodes exceeds a threshold, adding a second interaction node to a subsequent interaction layer of the interaction nodes in the plurality of subsequent interaction layers, the second interaction node being linked to the first interaction node; Determining a trajectory score for each of the plurality of trajectories based on one or more node scores of one or more nodes corresponding to the trajectory; and Selecting a trajectory from the plurality of trajectories for the self-object based on the trajectory score for the trajectory.

12. The self-object according to claim 11, wherein the processor is further configured to: Control the self-object according to the selected trajectory.

13. The self-object according to claim 11, wherein the processor is configured to determine a trajectory score for the trajectory by aggregating the one or more node scores of the one or more nodes of the trajectory.

14. The self-object according to claim 11, wherein the processor is configured to select the trajectory by selecting the trajectory in response to determining that the trajectory score for the trajectory is higher than the trajectory scores of other trajectories among the plurality of trajectories.

15. The self-object according to claim 11, wherein the processor is further configured to: Execute a neural network to determine the node score for each interaction node of the hierarchical node graph.

16. The self-object according to claim 11, wherein the node score for each interaction node corresponds to the comfort level associated with the interaction node.

17. The self-object according to claim 11, wherein the node score for each interaction node corresponds to the comfort level associated with the interaction node and the likelihood of intervention associated with the interaction node.

18. The self-object according to claim 11, wherein the processor is further configured to: Generate the hierarchical node graph in response to detecting one or more proxy objects in the space around the self-object using image data captured by a camera of the self-object.

19. The self-object according to claim 11, wherein the processor is further configured to: Execute an analysis protocol to determine the node score for each interaction node of the hierarchical node graph; Comparing the node score with the threshold; And Removing from the hierarchical node graph each interaction node of the hierarchical node graph corresponding to a node score less than the threshold.

20. The self-object according to claim 11, wherein the processor is further configured to: In response to determining that adding the second interaction node to the subsequent layer of interaction nodes in the plurality of subsequent layers causes the number of nodes in the hierarchical node graph to exceed a threshold, remove the third node from the hierarchical node graph based on the node score for the third node.

Images