Artificial intelligence modeling techniques for joint behavior planning and prediction.

Abstract

The method includes the steps of detecting one or more agent objects in a space surrounding the ego object using image data captured by a camera of the ego object; storing a hierarchical node graph having a goal layer having one or more goal nodes and a plurality of interaction layers of interaction nodes following the goal layer; adding interaction nodes to the interaction layers of interaction nodes of the plurality of interaction layers; determining a trajectory score for each of the plurality of trajectories based on one or more node scores of one or more nodes corresponding to the trajectories in the hierarchical node graph; and selecting one of the plurality of trajectories in the ego object based on the trajectory score for the trajectory.

Description

[Technical Field]

[0001] [Cross-reference to related patent applications] This application claims priority to U.S. Provisional Application No. 63 / 377,954, filed September 30, 2022, and U.S. Provisional Application No. 63 / 378,028, filed September 30, 2022, each of which is incorporated by reference herein in its entirety for all purposes.

[0002] The present disclosure generally relates to artificial intelligence-based modeling techniques for selecting suitable trajectories for an ego. [Background technology]

[0003] Due to rapid advances in computer technology, autonomous navigation techniques used for autonomous vehicles and robots (collectively, egos) have become widespread. These advances enable safer and more reliable autonomous navigation for egos. Egos often must navigate through complex and dynamic environments and terrain that may include vehicles, traffic, pedestrians, cyclists, and a variety of other static or dynamic obstacles. Understanding an ego's surroundings is necessary for making informed and appropriate decisions to avoid collisions. Summary of the Invention

[0004] For the foregoing reasons, a method and system that can analyze an ego's surroundings and select a trajectory that avoids collisions with objects in the ego's surroundings is desirable. A system (e.g., an ego's computing system) implementing the systems and methods herein can do so using a hierarchical node graph with layers of interaction nodes that represent potential interactions or non-interactions with one or more agent objects that the system detects in the ego's surrounding environment. The system can generate a score at each interaction node in the hierarchical node graph based on different variables, such as physics-based constraints, comfort, intervention potential, and / or a human-like discriminator. The system can combine the scores at individual nodes to determine a score at a node. The system can identify trajectories represented by the hierarchical node graph. Each trajectory can include different variations of interaction nodes at different layers, which can be linked together in the hierarchical node graph. The system can generate a trajectory score for the trajectory based on the node scores of the interaction nodes of each trajectory, such as by performing a function on each node score of each trajectory. The system can compare the trajectory scores among each other and select the trajectory with the highest trajectory score. The system can operate the ego using the selected trajectory. This process can greatly simplify trajectory selection techniques to allow the ego to make faster decisions using less processing power than conventional techniques that may attempt to determine possible trajectories for objects in the surrounding environment, which can require a significant amount of processing resources on a busy street.

[0005] In one embodiment, the method includes detecting, by a processor, one or more agent objects in a space surrounding the ego object using image data captured by a camera of the ego object; and storing, by the processor, a hierarchical node graph, the hierarchical node graph having a goal layer comprising one or more goal nodes corresponding to goals for the ego object to achieve; and a plurality of interaction layers of interaction nodes following the goal layer, each interaction node corresponding to at least one of a plurality of trajectories in the ego object considering the one or more agent objects and corresponding to a node score, the plurality of interaction layers comprising a first interaction layer of interaction nodes and a plurality of subsequent interaction layers of interaction nodes following the first interaction layer of interaction nodes, each interaction node in the plurality of subsequent interaction layers a plurality of interaction layers, each of which is subordinate to at least one interaction node in a previous interaction layer of the plurality of interaction layers; in response to determining that a node score of a first interaction node in a first interaction layer of the interaction nodes exceeds a threshold, adding, by the processor, a second interaction node to a subsequent interaction layer of the interaction nodes of the plurality of subsequent interaction layers, the second interaction node being linked to the first interaction node; determining, by the processor, a trajectory score for each of a plurality of trajectories based on one or more node scores of one or more nodes corresponding to the trajectory; and selecting, by the processor, one trajectory of the plurality of trajectories in the ego object based on the trajectory score for the trajectory.

[0006] The method may further include controlling, by the processor, the ego object according to the selected trajectory.

[0007] Determining a trajectory score for the trajectory may include aggregating, by the processor, one or more node scores of one or more nodes of the trajectory.

[0008] Selecting the trajectory may include selecting the trajectory in response to determining, by the processor, that a trajectory score for the trajectory is higher than trajectory scores of other trajectories of the plurality of trajectories.

[0009] The method may include running, by a processor, a neural network to determine a node score for each interaction node in the hierarchical node graph.

[0010] The node score for each interaction node may correspond to the comfort associated with the interaction node.

[0011] The node score at each interaction node may correspond to a comfort level associated with the interaction node and a likelihood of intervention associated with the interaction node.

[0012] The method may further include generating, by the processor, a hierarchical node graph in response to detecting one or more agent objects in a space surrounding the ego object using image data captured by a camera of the ego object.

[0013] The method may further include the steps of: executing, by the processor, an analysis protocol to determine a node score at each interaction node in the hierarchical node graph; comparing, by the processor, the node score to a threshold; and removing, by the processor, each interaction node in the hierarchical node graph that corresponds to a node score less than the threshold.

[0014] The method may further include, in response to determining that adding the second interaction node to a subsequent layer of the interaction node among the plurality of subsequent layers would cause the number of nodes in the hierarchical node graph to exceed a threshold, removing, by the processor, a third node from the hierarchical node graph based on a node score for the third node.

[0015] In another embodiment, the ego object may include a camera, a processor, and a non-transitory computer-readable medium configured to be executed by the processor, wherein the processor detects one or more agent objects in a space surrounding the ego object using image data captured by the camera, and stores a hierarchical node graph, the hierarchical node graph including a goal layer including one or more goal nodes corresponding to goals for the ego object to achieve, and a plurality of interaction layers of interaction nodes following the goal layer, each interaction node corresponding to at least one of a plurality of trajectories in the ego object considering the one or more agent objects and corresponding to a node score, the plurality of interaction layers including a first interaction layer of interaction nodes and a plurality of subsequent interaction layers of interaction nodes following the first interaction layer of interaction nodes, and each interaction node in the plurality of subsequent interaction layers and a plurality of interaction layers, each interaction node subordinate to at least one interaction node in a previous interaction layer of the plurality of interaction layers, and configured to, in response to determining that a node score of a first interaction node in a first interaction layer of the interaction nodes exceeds a threshold, add a second interaction node to a subsequent interaction layer of the interaction nodes of the plurality of subsequent interaction layers, the second interaction node being linked to the first interaction node, determine a trajectory score for each of the plurality of trajectories based on one or more node scores of the one or more nodes corresponding to the trajectories, and select one of the plurality of trajectories in the ego object based on the trajectory score for the trajectory.

[0016] The processor may be further configured to control the ego object according to the selected trajectory.

[0017] The processor may be configured to determine a trajectory score for the trajectory by aggregating one or more node scores of one or more nodes of the trajectory.

[0018] The processor may be configured to select the trajectory by selecting the trajectory in response to determining that a trajectory score for the trajectory is higher than trajectory scores of other trajectories of the plurality of trajectories.

[0019] The processor may be further configured to execute a neural network to determine a node score at each interaction node of the hierarchical node graph.

[0020] The node score for each interaction node may correspond to the comfort associated with the interaction node.

[0021] The node score at each interaction node may correspond to a comfort level associated with the interaction node and a likelihood of intervention associated with the interaction node.

[0022] The processor may be further configured to generate a hierarchical node graph in response to detecting one or more agent objects in a space surrounding the ego object using image data captured by a camera of the ego object.

[0023] The processor may be further configured to execute an analysis protocol to determine a node score at each interaction node in the hierarchical node graph, compare the node score to a threshold, and remove from the hierarchical node graph each interaction node in the hierarchical node graph that corresponds to a node score less than the threshold.

[0024] The processor may be further configured, in response to determining that adding the second interaction node to a layer subsequent to the interaction node among the plurality of subsequent layers causes the number of nodes in the hierarchical node graph to exceed a threshold, to remove, by the processor, a third node from the hierarchical node graph based on a node score for the third node. [Brief explanation of the drawings]

[0025] Non-limiting embodiments of the present disclosure are described by way of example with reference to the accompanying drawings, which are schematic and not intended to be drawn to scale, and unless indicated as representing background art, the figures represent aspects of the present disclosure.

[0026] [Figure 1A] 1 illustrates components of an AI-enabled visual data analysis system, according to one embodiment.

[0027] [Figure 1B] 1 illustrates various sensors associated with an ego, according to one embodiment.

[0028] [Figure 1C] 1 illustrates components of a vehicle, according to one embodiment.

[0029] [Figure 2] FIG. 1 illustrates a flow diagram of a process performed by an AI-enabled visual data analysis system, according to one embodiment.

[0030] [Figure 3] 1 illustrates a road scenario, according to one embodiment.

[0031] [Figure 4A] 1 illustrates a hierarchical node graph for a road scenario, according to one embodiment. [Figure 4B] 1 illustrates a hierarchical node graph for a road scenario, according to one embodiment. [Figure 4C]1 illustrates a hierarchical node graph for a road scenario, according to one embodiment. [Figure 4D] 1 illustrates a hierarchical node graph for a road scenario, according to one embodiment. [Figure 4E] 1 illustrates a hierarchical node graph for a road scenario, according to one embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0032] Reference will now be made to the exemplary embodiments illustrated in the drawings, and specific language will be used herein to describe the embodiments. It will nevertheless be understood that no limitation of the scope of the claims or this disclosure is intended thereby. Alterations and further modifications of the features of the invention shown herein, and further applications of the principles of the subject matter shown herein that will occur to those skilled in the art in possession of this disclosure, are to be considered within the scope of the subject matter disclosed herein. Other embodiments may be used and / or other changes may be made without departing from the spirit or scope of the present disclosure. The exemplary embodiments described in the detailed description are not meant to limit the presented subject matter.

[0033] An ego (e.g., an autonomous vehicle such as a car, truck, bus, motorcycle, all-terrain vehicle, cart, robot, or other automated device) traveling on a road must constantly monitor its surroundings for other objects on the road. The ego (e.g., the ego's processor) may detect various objects, such as pedestrians, other vehicles, or animals, and encounter scenarios in which the ego must navigate around the objects to reach a desired destination or goal, such as turning left into the right-most lane when a pedestrian is crossing the road. In such scenarios, the ego may navigate around the pedestrian by determining different potential trajectories for the pedestrian and any other objects in the area and potential trajectories for the ego. The ego may analyze each of the potential trajectories using an optimization function to determine a trajectory that will allow the ego to achieve the goal of turning left into the right-most lane while avoiding the pedestrian. This can require a large amount of processing power and time to make a decision, given the large number of variables involved. An ego may make such decisions many times per second (e.g., every 50 milliseconds) to autonomously navigate a road. Thus, an autonomous ego may use a large amount of processing power, and therefore energy and time, to make decisions about how to navigate a road. Using this processing power over time may cause the ego to arrive at its destination later and use greater amounts of energy.

[0034] An ego or system implementing the systems and methods described herein may overcome the aforementioned technical deficiencies. For example, a processor in the system or ego may implement a hierarchical node graph including different layers of nodes for scenarios the ego will encounter. The hierarchical node graph may include goal nodes corresponding to goals the ego will achieve (e.g., avoiding pedestrians, reaching a target lane or parking lot, etc.) and interaction nodes corresponding to interactions and / or non-interactions between the ego and agent objects (e.g., objects that are moving or may move in the environment around the ego). Examples of interactions may include the ego passing in front of or behind the agent object or contacting the agent object, although interaction does not require contact with the agent object. Examples of non-interactions may include the ego taking action to avoid the agent object entirely. The processor may identify such nodes in the hierarchical node graph for the ego and agent objects detected by the ego using the ego's camera. The processor may determine a score for each interaction node and / or goal node based on intervention likelihood (e.g., human intervention likelihood), a human-like discriminator, and / or physics-based constraints for each interaction node or goal node. The processor may sequentially identify different trajectories of the hierarchical node graph, each trajectory including a goal node and one or more interaction nodes, within the hierarchical node graph. The processor may determine a trajectory score for each trajectory. The processor may select the trajectory corresponding to the highest trajectory score. The processor may use the selected trajectory to control the ego. In this way, the ego can determine an optimal trajectory for the ego without determining the trajectories of objects in the surrounding environment or using complex cost functions, thereby reducing processing costs and time for selecting trajectories to use for autonomous driving.

[0035] A hierarchical node graph can have hierarchical layers of nodes. A processor can use the hierarchical configuration of the layers to determine how to traverse the hierarchical node graph to determine potential trajectories to use to control the ego. For example, the hierarchical node graph can include a goal layer including one or more goal nodes. Each goal node can be the start of one or more trajectories in the ego. The hierarchical node graph can then include multiple interaction layers, a first interaction layer, and one or more subsequent interaction layers. Each interaction layer can include one or more interaction nodes. Each interaction node in the first interaction layer can be linked to at least one goal node. Each interaction node in the first subsequent interaction layer can be linked to at least one interaction node in the first interaction layer. Each interaction node in a subsequent interaction layer relative to the first subsequent interaction layer can be linked to at least one interaction node in the preceding interaction layer. The processor may traverse different trajectories using links between nodes and select a trajectory for the ego by combining the scores of the interaction nodes and / or goal nodes of the trajectories.

[0036] To avoid generating a hierarchical node graph that is too large and may require a large amount of computing resources to maintain and evaluate different trajectories, the processor may prune the hierarchical node graph over time. For example, the processor may compare the node scores of the nodes in the hierarchical node graph with a threshold. In response to determining that the node score of the node is less than the threshold, the processor may remove the node from the hierarchical node graph. In one example, the processor may remove nodes with undesirable characteristics, such as the ego touching an object, to avoid using processing resources for trajectories that the processor does not select. The processor may remove such nodes and expand other higher-scoring nodes over time to maintain a hierarchical node graph that the processor can use to efficiently control the ego.

[0037] 1A is a non-limiting example of system components capable of implementing the methods and systems discussed herein. FIG. 1A illustrates components of an artificial intelligence (AI)-enabled visual data analysis system 100. System 100 may include analysis server 110a, system database 110b, administrator computing device 120, ego 140a-b (collectively ego 140), ego computing devices 141a-c (collectively ego computing devices 141), and server 160. System 100 is not limited to the components described herein and may include additional or other components not shown for the sake of brevity, which components should be considered within the scope of the embodiments described herein.

[0038] The above components may be connected via a network 130. Examples of network 130 may include, but are not limited to, a private or public LAN, a WLAN, a MAN, a WAN, and the Internet. Network 130 may include wired and / or wireless communications according to one or more standards and / or over one or more transport media.

[0039] Communications over network 130 may occur according to various communication protocols, such as Transmission Control Protocol / Internet Protocol (TCP / IP), User Datagram Protocol (UDP), IEEE communications protocols, etc. In one example, network 130 may include wireless communications according to the Bluetooth® set of specifications or another standard or proprietary wireless communications protocol. In another example, network 130 may also include communications over cellular networks, including, for example, Global System for Mobile Communications (GSM), Code Division Multiple Access (CDMA), or Enhanced Data for Global Evolution (EDGE) networks.

[0040] System 100 illustrates an example of a system architecture and components that may be used to implement one or more AI models, such as AI model 110c and hierarchical node graph 110d. Specifically, as depicted in FIG. 1A and described herein, analytics server 110a can use methods discussed herein to generate and use hierarchical node graph 110d for autonomous navigation using data retrieved from ego 140 (e.g., by using data streams 172 and 174). In one example, AI model 110c can detect occupancy of different voxels representing an area surrounding ego 140 based on image data captured by a camera of ego 140. Based on the occupancy data, analytics server 110a or ego 140 can detect different agent objects (e.g., moving objects) within the area surrounding ego 140. Ego 140 or analytics server 110a can use the detected agent objects to generate hierarchical node graph 110d to include goal nodes and / or interaction nodes. The goal nodes can indicate individual goals for the ego 140 to achieve, and the interaction nodes can indicate potential interactions or non-interactions between the ego 140 and the agent object or between the agent object itself. The ego 140 or the analysis server 110a can generate node scores for the nodes of the hierarchical node graph 110d. The ego 140 can use the node scores to generate trajectory scores for different trajectories that include different variations of linked nodes in the hierarchical node graph 110d. The ego 140 can select a trajectory based on the trajectory's trajectory score (e.g., based on the trajectory with the highest trajectory score). The ego 140 can operate autonomously according to the selected trajectory. Thus, the system 100 depicts a method of navigation using a hierarchical node graph that is faster and requires fewer processing resources than conventional methods of trajectory generation and / or selection.

[0041] In FIG. 1A, the AI ​​model 110c and the hierarchical node graph 110d are shown as components of the system database 110b, however, the AI ​​model 110c and the hierarchical node graph 110d may be stored in different or separate components, such as cloud storage or any other data repository accessible to the analytics server 110a or ego 140.

[0042] The analytics server 110a may also be configured to display an electronic platform showing various training attributes for training the AI ​​model 110c. The electronic platform may be displayed on the administrator computing device 120 to allow an analyst to monitor the training of the AI ​​model 110c. An example of an electronic platform generated and hosted by the analytics server 110a may be a web-based application or website configured to display the training dataset collected from the ego 140 and / or the training status / metrics of the AI ​​model 110c.

[0043] Analysis server 110a may be any computing device equipped with a processor and non-transitory machine-readable storage capable of performing the various tasks and processes described herein. Non-limiting examples of such computing devices may include workstation computers, laptop computers, server computers, etc. Although system 100 includes a single analysis server 110a, system 100 may include any number of computing devices operating in a distributed computing environment, such as a cloud environment.

[0044] Ego 140 may represent various electronic data sources that transmit data associated with a previous or current navigation session to analytics server 110a. Ego 140 may be any device configured for navigation, such as vehicle 140a and / or truck 140c. Ego 140 is not limited to being a vehicle and may include robotic devices as well. For example, ego 140 may include robot 140b, which may represent a general-purpose, bipedal, autonomous humanoid robot capable of navigating various terrains. Robot 140b may be provided with software that enables balance, navigation, perception, or interaction with the physical world. Robot 140b may also include various cameras configured to transmit visual data to analytics server 110a.

[0045] Although referred to herein as “ego,” ego 140 may or may not be an autonomous device configured for automatic navigation. For example, in some embodiments, ego 140 may be controlled by a human operator or by a remote processor. ego 140 may include various sensors, such as those depicted in FIG. 1B . The sensors may be configured to collect data as ego 140 navigates various terrains (e.g., roads). Analytics server 110a may collect data provided by ego 140. For example, analytics server 110a may obtain navigation session and / or road / terrain data (e.g., images of ego 140 navigating roads) from various sensors, such that the collected data is ultimately used by AI model 110c for training purposes.

[0046] As used herein, a navigation session corresponds to a trip in which ego 140 travels a route, regardless of whether the trip was autonomous or controlled by a human. In some embodiments, the navigation session may be for data collection and model training purposes. However, in some other embodiments, ego 140 may refer to a vehicle purchased by a consumer, and the purpose of the trip may be classified as daily use. A navigation session may begin when ego 140 travels more than a threshold distance (e.g., 0.1 miles, 100 feet) from a non-moving location or exceeds a threshold speed (e.g., greater than 0 mph, greater than 1 mph, greater than 5 mph). A navigation session may end when ego 140 is returned to a non-moving location and / or turned off (e.g., when the driver exits the vehicle).

[0047] Ego 140 may correspond to a group of egos monitored by analytics server 110a to generate hierarchical node graph 110d. For example, the driver of vehicle 140a may authorize analytics server 110a to monitor data associated with their respective vehicle. As a result, analytics server 110a can collect sensor / camera data and detect agent objects in the environment of ego 140 from which it collects sensor / camera data using various methods discussed herein. Analytics server 110a can gradually build hierarchical node graph 110d from the data by adding goal nodes and interaction nodes linked to each other in successive layers from different scenarios for which ego 140 provides data. Analytics server 110a can generate node scores for different nodes and prune nodes with otherwise low node scores or node scores below a threshold. Analytics server 110a can deploy or transmit the hierarchical node graph to different egos 140 for use in autonomous driving.

[0048] Over time, ego 140 can send additional data regarding different scenarios to analytics server 110a. Analytics server 110a can update hierarchical node graph 110d based on the data over time and send the updated version of hierarchical node graph 110d to ego 140 for use in autonomous driving. Thus, system 100 performs a loop in which navigation data received from ego 140 can be used to update hierarchical node graph 110d. Ego 140 may include a processor that processes hierarchical node graph 110d for navigation purposes (e.g., to select a trajectory to use for navigation).

[0049] Egos 140 may be equipped with various technologies that enable them to gather data from their surroundings and (potentially) navigate autonomously. For example, ego 140 may be equipped with an inference chip for running self-driving software.

[0050] Various sensors for each ego 140 may monitor and transmit collected data associated with different navigation sessions to analytics server 110a. FIGS. 1B-1C show block diagrams of sensors incorporated within ego 140, according to one embodiment. The number and location of each sensor discussed with respect to FIGS. 1B-1C may depend on the type of ego discussed in FIG. 1A. For example, robot 140b may include different sensors than vehicle 140a or truck 140c. For example, robot 140b may not include airbag activation sensor 170q. Additionally, the sensors for vehicle 140a and truck 140c may be in different locations than those shown in FIG. 1C.

[0051] As discussed herein, various sensors incorporated within each ego 140 may be configured to measure various data associated with each navigation session. Analytics server 110a may periodically collect the data monitored and collected by these sensors, which is processed according to methods described herein and used to generate hierarchical node graph 110d and / or execute AI model 110c to generate an occupancy map and detect agent objects in the space around ego 140. Further, trajectory recommendations for ego 140 can be generated by executing hierarchical node graph 110d and / or AI model 110c.

[0052] Ego 140 may include user interface 170a. User interface 170a may refer to the user interface of an ego computing device (e.g., ego computing device 141 of FIG. 1A). User interface 170a may be implemented as a display screen integrated with or coupled to a vehicle's interior, a head-up display, a touchscreen, etc. User interface 170a may include input devices such as a touchscreen, knobs, buttons, a keyboard, a mouse, a gesture sensor, a steering wheel, etc. In various embodiments, user interface 170a may be adapted to provide user input (e.g., as types of signals and / or sensor information) to other devices or sensors of ego 140 (e.g., the sensors shown in FIG. 1B), such as controller 170c.

[0053] User interface 170a may also be implemented with one or more logic devices that may be adapted to execute instructions, such as software instructions, that implement any of the various processes and / or methods described herein. For example, user interface 170a may be adapted to form a communication link, send and / or receive communications (e.g., sensor signals, control signals, sensor information, user input, and / or other information), or perform various other processes and / or methods. In another example, a driver may use user interface 170a to control the temperature of ego 140 or activate its features (e.g., autonomous driving or steering system 170o). Accordingly, user interface 170a may monitor and collect driving session data in conjunction with other sensors described herein. User interface 170a may also be configured to display various data generated / predicted by analytics server 110a and / or AI model 110c.

[0054] Orientation sensor 170b may be implemented as one or more of a compass, float, accelerometer, and / or other digital or analog device capable of measuring the orientation of ego 140 (e.g., the magnitude and direction of roll, pitch, and / or yaw relative to one or more reference orientations, such as gravity and / or magnetic north). Orientation sensor 170b may be adapted to provide orientation measurements at ego 140. In other embodiments, orientation sensor 170b may be adapted to provide roll, pitch, and / or yaw rate of ego 140 using a time series of orientation measurements. Orientation sensor 170b may be positioned and / or adapted to provide orientation measurements relative to a particular coordinate frame of ego 140.

[0055] Controller 170c may be implemented as any suitable logic device (e.g., a processing device, microcontroller, processor, application specific integrated circuit (ASIC), field programmable gate array (FPGA), memory storage device, memory reader, or other device or combination of devices) that can be adapted to execute, store, and / or receive appropriate instructions, such as software instructions that implement control loops for controlling various operations of ego 140. Such software instructions may also process sensor signals, determine sensor information, provide user feedback (e.g., via user interface 170a), interrogate devices regarding operating parameters, select operating parameters for devices, or implement methods for performing any of the various operations described herein.

[0056] Communications module 170e may be implemented as any wired and / or wireless interface configured to communicate sensor data, configuration data, parameters, and / or other data and / or signals to any feature shown in FIG. 1A (e.g., analytics server 110a). As described herein, in some embodiments, communications module 170e may be implemented in a distributed manner, such that portions of communications module 170e are implemented within one or more elements and sensors shown in FIG. 1B. In some embodiments, communications module 170e may delay communication of sensor data. For example, when ego 140 does not have network connectivity, communications module 170e may store sensor data in temporary data storage and transmit the sensor data when ego 140 is identified as having adequate network connectivity.

[0057] Speed ​​sensor 170d may be implemented as an electronic pitot tube, a metering gear or wheel, a water speed sensor, a wind speed sensor, a wind speed sensor (e.g., direction and magnitude), and / or other device capable of measuring or determining the linear velocity of ego 140 (e.g., within the surrounding medium and / or aligned with the longitudinal axis of ego 140) and providing such measurement as a sensor signal that can be communicated to various devices.

[0058] Gyroscope / accelerometer 170f may be implemented as an electronic sextant, a semiconductor device, an integrated chip, an accelerometer sensor, or other system or device capable of measuring angular velocity / acceleration and / or linear acceleration (e.g., direction and magnitude) of ego 140 and providing such measurements as sensor signals that can be communicated to other devices, such as analytics server 110a. Gyroscope / accelerometer 170f may be positioned and / or adapted to make such measurements relative to a particular coordinate frame of ego 140. In various embodiments, gyroscope / accelerometer 170f may be mounted in a common housing and / or module with other elements shown in FIG. 1B to ensure a common frame of reference or known transformations between frames of reference.

[0059] Global Navigation Satellite System (GNSS) 170h may be implemented as a global positioning satellite receiver and / or another device capable of determining the absolute and / or relative position of ego 140 based on, for example, radio signals received from space-born and / or terrestrial sources and providing such measurements as sensor signals that can be communicated to various devices. In some embodiments, GNSS 170h may be adapted to determine the velocity, speed, and / or yaw rate of ego 140 (e.g., using a time series of position measurements), such as the absolute velocity and / or yaw component of the angular velocity of ego 140.

[0060] Temperature sensor 170i may be implemented as a thermistor, an electrical sensor, an electrical thermometer, and / or other device capable of measuring a temperature associated with ego 140 and providing such measurement as a sensor signal. Temperature sensor 170i may be configured to measure an environmental temperature associated with ego 140, such as a cockpit or dash temperature, which may be used to estimate the temperature of one or more elements of ego 140.

[0061] Humidity sensor 170j may be implemented as a relative humidity sensor, an electrical sensor, an electrical relative humidity sensor, and / or another device capable of measuring the relative humidity associated with ego 140 and providing such measurement as a sensor signal.

[0062] Steering sensor 170g may be adapted to physically adjust the orientation of ego 140 according to one or more control signals provided by a logic device, such as controller 170c, and / or user input. Steering sensor 170g may include one or more actuators and control surfaces of ego 140 (e.g., rudders or other types of steering or trim mechanisms) and may be adapted to physically adjust the control surfaces to various positive and / or negative steering angles / positions. Steering sensor 170g may also be adapted to sense the current steering angles / positions of such steering mechanisms and provide such measurements.

[0063] Propulsion system 170k may be implemented as a propeller, turbine, or other thrust-based propulsion system, a mechanical wheeled and / or tracked propulsion system, a wind / sail-based propulsion system, and / or other type of propulsion system that may be used to provide motive power to ego 140. Propulsion system 170k may also monitor the direction of motive power and / or thrust of ego 140 relative to a coordinate frame of reference of ego 140. In some embodiments, propulsion system 170k may be coupled to and / or integrated with steering sensor 170g.

[0064] Occupant restraint sensor 170l may monitor the seat belt detection and lock / unlock assembly, as well as other occupant restraint subsystems. Occupant restraint sensor 170l may include various environmental and / or status sensors, actuators, and / or other devices that facilitate operation of safety mechanisms associated with operation of ego 140. For example, occupant restraint sensor 170l may be configured to receive movement and / or status data from other sensors shown in FIG. 1B. Occupant restraint sensor 170l may determine whether a safety mechanism (e.g., a seat belt) is engaged.

[0065] Camera 170m may refer to one or more cameras integrated into ego 140, as depicted in FIG. 1C, or may include multiple cameras integrated into (or retrofitted to) ego 140. Camera 170m may be an inward-facing or outward-facing camera of ego 140. For example, as depicted in FIG. 1C, ego 140 may include one or more inward-facing cameras that can monitor and collect video of occupants of ego 140. Ego 140 may include eight outward-facing cameras. For example, ego 140 may include front camera 170m-1, forward-looking side camera 170m-2, forward-looking side camera 170m-3, rearward-looking side camera 170m-4 on each front fender, camera 170m-5 on each side (e.g., integrated into the B-pillar), and rear camera 170m-6.

[0066] 1B, radar 170n and ultrasonic sensor 170p may be configured to monitor the distance of ego 140 to other objects, such as other vehicles or immovable objects (e.g., trees or garage doors). Ego 140 may also include an automatic driving or steering system 170o configured to autonomously navigate ego 140 using data collected via various sensors (e.g., radar 170n, speed sensor 170d, and / or ultrasonic sensor 170p).

[0067] Thus, automated driving or steering system 170o may analyze various data collected by one or more sensors described herein to identify driving data. For example, automated driving or steering system 170o may calculate the risk of a forward collision based on ego 140's speed and its distance to another vehicle on the road. Autonomous driving or steering system 170o may also determine whether the driver is touching the steering wheel. Autonomous driving or steering system 170o may transmit the analyzed data to various features discussed herein, such as an analytics server.

[0068] Airbag deployment sensor 170q may predict or detect a crash and cause one or more airbags to deploy or inflate. Airbag deployment sensor 170q may transmit data regarding the deployment of the airbags, including data associated with the event that caused the deployment.

[0069] 1A , administrator computing device 120 may represent a computing device operated by a system administrator. Administrator computing device 120 may be configured to display data obtained or generated by analytics server 110 a (e.g., various analytics metrics and risk scores), allowing the system administrator to monitor various models utilized by analytics server 110 a, review feedback, and / or facilitate the training of AI models 110 c and / or the generation of hierarchical node graphs 110 d maintained by analytics server 110 a and / or individual egos 140.

[0070] Ego 140 may be any device configured to navigate various routes, such as vehicle 140a or robot 140b. As discussed with respect to FIGS. 1B-1C, ego 140 may include various telemetry sensors. Ego 140 may also include ego computing device 141. Specifically, each ego may have its own ego computing device 141. For example, truck 140c may have ego computing device 141c. For brevity, ego computing devices are collectively referred to as ego computing devices 141. Ego computing device 141 may control the presentation of content on ego 140's infotainment system, process process commands related to the infotainment system, aggregate sensor data, manage communication of data to electronic data sources, receive updates, and / or send messages. In one configuration, ego computing device 141 communicates with an electronic control unit. In another configuration, ego computing device 141 is an electronic control unit. Ego computing device 141 may include a processor and non-transitory machine-readable storage media capable of performing the various tasks and processes described herein. For example, AI model 110c described herein may be stored and executed (or directly accessed) by ego computing device 141. Non-limiting examples of ego computing device 141 may include a vehicle multimedia and / or display system.

[0071] In one example of how ego computing device 141 of ego 140 can generate and / or use hierarchical node graph 110d for navigation, when ego computing device 141 controls ego 140 for autonomous driving, a camera of ego 140 can generate image data of a space around ego 140. Ego computing device 141 can execute AI model 110c to automatically detect agent objects within the space, for example, by generating and analyzing distinct voxels representing the space in occupancy characteristics from the image data. Ego computing device 141 can determine a task for ego 140 to perform, such as making a left turn. In response to determining the task, ego computing device 141 can generate or use hierarchical node graph 110d to determine a trajectory to use to control ego 140 to perform the task.

[0072] For example, to generate hierarchical node graph 110d, ego computing device 141 may generate one or more goal nodes corresponding to the task. The goal nodes may correspond to different goals or targets for the task, such as making a left turn, making a safe left turn, avoiding exceeding a certain speed, etc. The ego computing device 141 may generate one or more interaction nodes. The interaction nodes may correspond to different interactions or non-interactions between ego 140 and detected agent objects in the space surrounding ego 140. For example, ego computing device 141 may determine how ego 140 can interact with the agent object by determining the agent object's identification or classification (e.g., pedestrian or vehicle), the agent object's current location, the agent object's current location relative to a task target location, the agent object's current location relative to ego 140, and / or the agent object's current state (e.g., whether to move, its speed, its direction of movement, its size, etc.). The ego computing device 141 can determine such identification or classification and characteristics of the agent object using machine learning techniques, using one or more features, or by querying memory with sensor data generated about the agent object. The ego computing device 141 can use the determined characteristics to determine different interactions that may occur by using machine learning models, using one or more features, or by querying memory. The ego computing device 141 can generate interaction nodes for interactions or non-interactions that may occur in attempting to achieve each goal of the goal nodes. The ego computing device 141 can link the interaction nodes to the goal nodes on which the interaction nodes each depend.

[0073] The ego computing device 141 may determine node scores for the nodes of the hierarchical node graph 110d. The ego computing device 141 may do so using a function or machine learning technique on the data for each node. For example, the ego computing device 141 may input the data for each interaction node into a machine learning model (e.g., a neural network, a support vector machine, a random forest, etc.) and run the machine learning model for each interaction node. The machine learning model may output a node score for the interaction node based on the run. The ego computing device 141 may store the score for each node for which a score was generated.

[0074] The machine learning model may be trained to output a node score based on factors such as comfort, physics-based constraints (e.g., likelihood of collision), human-like discriminators (e.g., likelihood of a human performing the same action), and / or intervention likelihood. The machine learning model may be trained to do so using, for example, labeling techniques that indicate scores for individual factors. The machine learning model may also be trained to aggregate scores for individual nodes to generate a node score for a node. In some cases, multiple machine learning models may be used to generate scores for separate factors for each node in a hierarchical data structure.

[0075] The ego computing device 141 can expand the hierarchical node graph 110d over time. The ego computing device 141 can do so based on the scores of the nodes in the hierarchical node graph 110d. For example, the ego computing device 141 can identify any interaction nodes with a node score (e.g., at least one node score) below a threshold. The ego computing device 141 can decide not to expand the identified interaction nodes and accordingly insert a label into such interaction nodes or memory or remove low-scoring nodes from the hierarchical node graph.

[0076] For interaction nodes with node scores above a threshold, the ego computing device 141 may determine another interaction that is subordinate to (e.g., follows or depends on) the transaction. In one example, if a pedestrian is crossing a road, the interaction of the initial interaction node may be to turn left after the pedestrian crosses the road. However, the ego computing device 141 may detect an oncoming vehicle that may be traveling in the same direction toward the space where the ego 140 is intended to enter. Thus, the ego computing device 141 may generate an interaction node that corresponds to having the pedestrian cross the road but that is linked to the initial interaction node that corresponds to letting the oncoming vehicle pass. The ego computing device 141 may generate another interaction node that is linked to the initial interaction node but that corresponds to passing in front of the oncoming vehicle. The ego computing device 141 can also generate interaction nodes for non-interaction with pedestrians and oncoming vehicles, such as waiting for pedestrians and oncoming vehicles to exit the space or turning (e.g., turning right) in a direction that does not involve the ego 140 interacting with pedestrians or oncoming vehicles. The ego computing device 141 can repeat the node scoring and expansion process over time for any number of agent objects in the space surrounding the ego 140.

[0077] The ego computing device 141 can generate trajectory scores for different trajectories outlined by the hierarchical node graph 110d. For example, the hierarchical node graph 110d can include one or more trajectories, each beginning with a goal node and including interaction nodes linked to each other by the goal node. In some cases, the goal node may not be included in the trajectory. The ego computing device 141 can determine a trajectory score for each trajectory based on the node scores of the nodes that make up the respective trajectory. The ego computing device 141 can retrieve the node scores from each node and perform a function on the retrieved node scores (e.g., using aggregation or summation techniques, determining an average or weighted average, determining a median, etc.) or use machine learning techniques to generate a trajectory score for each trajectory. In some cases, the ego computing device 141 can determine multiple trajectory scores for each trajectory based on the node scores for different factors.

[0078] The ego computing device 141 can select a trajectory from multiple trajectories based on the trajectory score. The ego computing device 141 can compare the trajectory scores together to determine a trajectory with a trajectory score that satisfies a condition. For example, the ego computing device 141 can identify the trajectory corresponding to the highest trajectory score. In another example, the ego computing device 141 can determine a combination of trajectory scores in a trajectory that satisfies a condition or has the highest mean, weighted mean, sum, weighted sum, or median (e.g., a combination of trajectory scores that is closest to the solution space in the combined factors). The ego computing device 141 can control the ego 140 according to the trajectory identified or selected from the hierarchical node graph 110d.

[0079] When analysis server 110a generates hierarchical node graph 110d, it can use techniques similar to those of ego computing device 141 to generate hierarchical node graph 110d. Analysis server 110a can do so using data from multiple egos 140 in different scenarios. In some cases, analysis server 110a can remove trajectories from hierarchical node graph 110d that have low scores (e.g., instruct or flag not to perform further calculations on or remove the nodes of the trajectory) so that ego 140 does not waste processing resources deciding whether to implement the low-scoring trajectory. Analysis server 110a can send hierarchical node graph 110d to ego 140, which can use hierarchical node graph 110d by identifying interaction nodes and / or agent objects that correspond to or match the scenario facing ego 140. Ego 140 can update hierarchical node graph 110d with new interaction nodes and / or goal nodes, and therefore new trajectories, based on the data ego 140 collects about each scenario.

[0080] FIG. 2 illustrates a flow diagram of a method 200 performed in an AI-enabled visual data analysis system, according to one embodiment. Method 200 may include steps 202-210. However, other embodiments may include additional or alternative steps, or may omit one or more steps. Method 200 may be performed by an ego computing device (e.g., a computer similar to the processor of ego computing device 141 or ego 140). However, one or more steps of method 200 may be performed by any number of computing devices operating in the distributed computing system (e.g., analysis server 110a) described in FIGS. 1A-1C. For example, one or more computing devices of an ego may locally perform some or all of the steps described in FIG. 2.

[0081] Using method 200, the ego computing device can implement a node hierarchical data structure for trajectory selection in the ego. To do so, the ego computing device can detect one or more agent objects (e.g., objects that are moving or may move) in a space or environment surrounding the ego (e.g., the ego object). The ego computing device can store a node data structure including goal nodes indicating goals for the ego to achieve and / or interaction nodes corresponding to different potential interactions and / or non-interactions between the ego and the agent object. The ego computing device can determine trajectory scores for different trajectories following different permutations of goal nodes and interaction nodes based on the node scores of the trajectories' nodes. The trajectory scores can correspond to the comfort, physics-based constraints, intervention potential, and / or discriminator (e.g., a human-like discriminator) of each trajectory. The ego computing device can compare the trajectory scores to identify the highest trajectory score. The ego computing device can select the trajectory with the highest trajectory score. The ego computing device can control the ego using the selected trajectory.

[0082] In step 202, the ego computing device detects one or more agent objects in the space surrounding the ego object. The ego computing device may detect the one or more agent objects using image data captured by the ego object's camera. For example, the ego object's camera may generate image data (e.g., images or video) of the environment surrounding the ego object over time and / or as the ego object is driving. The ego computing device may process the image data as the camera generates the images using object recognition techniques, such as by running a machine learning or artificial intelligence model, to detect different objects in the image data. The ego computing device may identify the location of the detected objects relative to the ego object. In one example, the ego computing device may detect objects in the image data by detecting the occupancy of different voxels representing the space surrounding the ego object.

[0083] The ego computing device may determine the type of object it detects. The type of object may be, for example, a stationary object or an agent object. The ego computing device may determine the type of object using a lookup technique in memory. For example, the ego computing device may detect an object from image data. In response to detecting the object, the ego computing device may use a lookup in memory to match the detected object with objects stored in memory. The objects stored in memory may have an association with an object type. The ego computing device may determine the type of the detected object based on the match with the objects stored in memory. In some cases, a machine learning or artificial intelligence model that the ego computing device executes to detect the object may further determine the type of the object. In some cases, the ego computing device may determine an identification or classification of the object (e.g., determine whether the object is a sign or a pedestrian) and determine the type of the object based on the determination (e.g., use a lookup in memory for identification or classification). The ego computing device may detect objects and determine the type of object in any manner.

[0084] In step 204, the ego computing device stores a hierarchical node graph. The hierarchical node graph may include a goal layer with one or more goal nodes corresponding to goals for the ego object to achieve. The hierarchical node graph may also include one or more interaction layers of interaction nodes following the goal layer. Each interaction node may correspond to an interaction or non-interaction between the ego object and at least one agent object (e.g., pedestrian, passing vehicle, etc.) that the ego computing device detects in a space surrounding the ego object. A goal node may correspond to a goal for the ego to achieve (e.g., turn into the leftmost lane, avoid hitting pedestrians, etc.). Each node may be a data structure (e.g., a table or Strapi model) that stores data specific to the node. The multiple interaction layers may include an initial interaction layer of interaction nodes and / or one or more subsequent interaction layers of interaction nodes following the initial layer of interaction nodes.

[0085] Nodes in adjacent layers in the hierarchy of a hierarchical node graph may be linked (e.g., store identifiers of the linked nodes) with one or more nodes in previous and / or subsequent layers in the hierarchy. For example, each goal node in a goal layer may be linked to at least one interaction node in a first interaction layer; each interaction node in a first interaction layer may be linked to at least one interaction node in a first subsequent interaction layer of the hierarchy nodes, etc. The links may indicate dependencies or causal relationships between nodes. For example, a first interaction node (e.g., an interaction node in a first interaction layer) may be linked to a subsequent interaction node (e.g., an interaction node in a subsequent interaction layer). The first interaction node may correspond to waiting for a pedestrian to pass before making a left turn. The subsequent interaction node may correspond to waiting for a passing vehicle to pass before making a left turn. A subsequent interaction node may depend on the initial interaction node, since the subsequent interaction node's interaction may only be possible if the initial interaction node's interaction occurs first. In some cases, the dependency may indicate the sequential nature of the interactions. For example, the subsequent interaction may occur after the interaction of the initial interaction node. A hierarchical node graph may include any number of interaction nodes in different interaction layers that are linked in this manner with interaction nodes in other interaction layers of the hierarchical node graph.

[0086] Links between goal nodes in the goal layer and interaction nodes in the initial interaction layer may have dependencies similar to the dependencies between interaction nodes. For example, a goal may be to make a far left turn. Performing a far left turn may allow for different interactions with agent objects (e.g., vehicles or pedestrians) than if the goal were to turn in a different direction. Thus, an interaction node linked with a goal node for making a left turn may take into account agent objects that may be affected or influence whether and / or how the ego object makes the left turn.

[0087] An interaction node may also correspond to a lack of interaction with one or more other agent objects. For example, a goal node may correspond to a goal to perform a left turn. The ego computing device may detect a pedestrian in the lane for the turn and another vehicle approaching from the right. There may be interaction nodes in the hierarchical node graph for interactions by the ego object, such as turning right to avoid the pedestrian and car, or standing still until the path is clear. Each such interaction node may be an interaction node in the hierarchical node graph.

[0088] In some cases, the ego computing device may generate a hierarchical node graph. The ego computing device may generate the hierarchical node graph in response to determining at least one task for the ego object to accomplish (e.g., determining to make a left turn to follow a predetermined or configured path). The task may be a goal. The ego computing device may generate the hierarchical node graph further in response to detecting one or more agent objects in a space surrounding the ego object.

[0089] For example, in response to determining to achieve a goal and / or detecting one or more agent objects in the space surrounding the ego object, the ego computing device can generate one or more goal nodes in a goal layer of the hierarchical node graph. The ego computing device can generate one or more goal nodes by querying memory for different goals for the ego computing device to achieve based on the task. Examples of such goals for a left turn task may be avoiding hitting pedestrians, avoiding hitting other vehicles, making sure the turn is not too sharp, ensuring the ego object's speed remains below a threshold, etc. The ego computing device can generate one or more goal nodes by storing or allocating data structures at individual goal nodes in memory. The ego computing device can populate the data structures at different goal nodes with information about the goal of each goal node, such as the identity of the goal and any metadata about the goal (e.g., the goal node's node score).

[0090] The ego computing device can generate one or more interaction nodes in each of one or more interaction layers of the hierarchical node graph. The ego computing device can generate interaction nodes based on potential interactions and / or non-interactions it determines between the ego object and a detected agent object. The ego computing device can determine such interactions and / or non-interactions for each goal node. For example, for a goal node associated with a goal making a left turn into a distant lane, the ego computing device can determine potential interactions or non-interactions with a pedestrian crossing the street in the lane and another vehicle approaching in the same lane. The ego computing device can determine potential interactions by, for example, querying a memory using the location and / or identity of the object to identify potential interactions corresponding to the scenario. In some cases, the ego computing device can implement a machine learning model (e.g., a neural network, a support vector machine, a random forest, etc.) to identify different potential interactions. The ego computing device can similarly query memory or run machine learning models to determine which agent objects may be involved in potential interactions for the ego to achieve its goal. In some cases, the ego computing device similarly determines potential interactions with identified agent objects without first determining which agent objects may be involved in potential interactions with the ego object for the execution of the goal. The ego computing device can generate interaction nodes that link the interaction nodes to goal nodes in a hierarchical node graph.The ego computing device may include an identification of the interaction with metadata about the interaction at each interaction node linked to the goal node (e.g., the type of interaction, the node score of the interaction's interaction node, the velocity of the object at one or each of the interaction nodes, etc.) The ego computing device may link any number of interaction nodes to a goal node.

[0091] The ego computing device can sequentially link interaction nodes in different interaction layers that are dependent on each other. For example, one interaction with an agent object can occur only following another interaction with the same or a different agent object. Thus, the ego computing device can link subsequent interactions with previous interactions in separate interaction layers of a hierarchical node graph. For example, an ego object may cause a pedestrian to cross a street and then cause another vehicle to pass the ego object. The ego computing device may link an interaction node for passing the pedestrian to an interaction node for passing the vehicle in an adjacent layer of the hierarchical node graph. The ego computing device can link any number of interaction nodes to individual interaction nodes in any number of layers. Thus, the ego computing device can generate a hierarchical node graph taking into account the different interactions or non-interactions that may occur and the goals that the ego object may achieve.

[0092] In some cases, the hierarchical node graph may be generated by a server (e.g., analysis server 110a) and deployed on the ego object. For example, the server may receive image data from different ego objects in an autonomous driving scenario including different agent objects surrounding the ego object in the scenario. The server may receive the data for the autonomous driving scenarios and generate a hierarchical node graph (e.g., a single hierarchical node graph) covering each scenario with goal nodes for the scenario's goal and interaction nodes for interactions and non-interactions with the scenario's agent object. The server may generate the hierarchical node graph by adding goal nodes and interaction nodes in different scenarios as the server receives the data for the scenarios from the ego objects. The server may, for example, analyze the hierarchical node graph for nodes of the same type (e.g., goal nodes or interaction nodes) before adding a new node to the hierarchical node graph and avoid duplicating goal or interaction nodes by only adding a new node in response to determining that the same or similar node (e.g., a node with similar metadata above a threshold) is not already present in the hierarchical node graph. In some cases, to conserve processing resources, analysis may only be performed on nodes within the same branch of the node to which the node is added (e.g., nodes linked to each other). A server can generate such a hierarchical node graph over time. The server can deploy (e.g., send, such as a binary file, to individual ego bodies for use in) the hierarchical node graph (e.g., as a master hierarchical node graph) to the ego bodies for use in autonomous driving, as described herein. After deployment, the server can continue to receive data and update the hierarchical node graph based on the received data. The server can deploy an updated version of the hierarchical node graph at set intervals, in response to receiving input to do so, or in response to determining that any other condition is met.

[0093] The ego computing device may generate node scores for nodes in a hierarchical node graph. The ego computing device may generate such node scores for interaction nodes and goal nodes. The ego computing device may generate node scores for individual nodes using an analysis protocol (e.g., a function, algorithm, machine learning model, or artificial intelligence model configured to generate node scores for nodes). For example, the ego computing device may generate node scores for nodes by searching data in a node's data structure and running a neural network trained to generate scores for the nodes. The neural network may output node scores for individual nodes based on the data in each node. In another example, the ego computing device may generate node scores by performing a function (e.g., sum, median, mean, weighted sum, weighted average, etc.) on values ​​in the nodes. The ego computing device may generate node scores for nodes in any manner.

[0094] In some cases, the node score may correspond to factors such as comfort, physics-based constraints (e.g., collision checks), intervention potential (e.g., likelihood of human intervention), and / or human-like discriminator (e.g., a score indicating that the same decision would be made by a human driver). For example, when a neural network (or another machine learning model) is trained, the neural network may be trained to generate scores that correlate with each or a subset of these factors, where higher values ​​of individual factors may correspond to higher scores and lower levels of factors may correspond to lower scores. Thus, when the neural network generates a score at a node, the neural network may simulate determining a score that represents or corresponds to the factor. In another example, different machine learning models (e.g., neural networks) or functions may be trained or configured to generate scores for different factors. In such cases, the machine learning models or functions may be configured to process specific types of data or the same type of data from each node. For each node, a different machine learning model or function may output a score for each factor. The different machine learning models or functions may process the scores of the factors to generate a node score at the node. The scores of the factors may be node scores. The ego computing device may store any node scores generated for its own nodes, and the server may similarly generate and store node scores for nodes in the hierarchical node graph that it generates.

[0095] In a non-limiting example, and referring now to FIG. 3 , a road scenario 300 is depicted. The road scenario 300 includes an ego object 302 attempting to make a left turn into lane 304 of an intersecting road. In doing so, the ego computing device of the ego object 302 can collect image data of the environment surrounding the ego object 302. From the image data, the ego computing device can detect the lane 304 and agent objects 306 and 308. The agent object 306 may be a pedestrian crossing the road across lane 304. The agent object 308 may be a vehicle making a right turn behind the pedestrian on the same road onto which the ego object 302 is attempting to turn. The ego computing device can generate a hierarchical node graph based on potential interactions with the agent objects 306 and 308.

[0096] Referring again to FIG. 2 , in step 206, the ego computing device adds an interaction node (e.g., a second interaction node) to a subsequent layer of interaction nodes in the hierarchical node graph. The ego computing device may perform step 206, for example, when generating the hierarchical node graph in step 204 or following generating the hierarchical node graph to update the hierarchical node graph. The ego computing device may add the interaction node in response to detecting or determining an interaction with an agent node in the space surrounding the hierarchical node graph. In one example, the ego computing device may add the interaction node after generating the hierarchical node graph and detecting a new agent node in the space surrounding the ego object. The ego computing device may add the interaction node by determining one or more interaction nodes to which the new interaction node depends (e.g., based on whether the new interaction node's interaction occurs subsequent to or based on the interaction of a previous interaction node). The ego computing device can add an interaction node to a hierarchical node graph in an interaction layer subsequent to the interaction layer in which the previous interaction node is located.

[0097] The ego computing device can add an interaction node in response to determining that a node score for the interaction node exceeds a threshold. For example, before adding the interaction node to the hierarchical node graph, the ego computing device can determine a node score for the interaction node. The ego computing device can do this using the systems and methods described herein based on the data in the interaction node. The ego computing device can compare the node score to a threshold (e.g., a defined threshold). In response to determining that the node score exceeds the threshold, the ego computing device can add the interaction node to the hierarchical node graph. Otherwise, the ego computing device can discard the interaction node (e.g., remove the interaction node from memory or otherwise not add the interaction node to the hierarchical node graph) or can add the interaction node to the hierarchical node graph with a flag that restricts the linking of any further interaction nodes from the interaction node. In some cases, the ego computing device may generate node scores for different factors in an interaction node and compare the node scores to a threshold (e.g., the same threshold or a different threshold for each factor). In response to determining that a number or combination of scores (e.g., a defined number or combination) exceeds a threshold, the ego computing device may add the interaction node to the hierarchical node graph. Otherwise, the ego computing device may discard the interaction node. The server may similarly add the interaction node to a hierarchical node graph that it generates.

[0098] In some cases, the ego computing device may remove nodes from the hierarchical node graph. For example, the ego computing device may identify node scores of different nodes in the hierarchical node graph. The ego computing device may compare the node scores to a threshold. In response to determining that the node score for a node is less than the threshold, the ego computing device may remove the node from the hierarchical node graph. In removing a node from the hierarchical node graph, the ego computing device may identify any nodes in the hierarchical node graph that depend on the removed node. In some cases, the ego computing device may remove any such nodes further in response to determining that each removed node does not depend on another node in the hierarchical node graph (e.g., a node in a previous interaction layer or a goal layer). In doing so, the ego computing device may remove undesirable nodes and branches from the hierarchical node graph that an administrator may never want selected as part of a selected trajectory or that may prevent a trajectory including the removed node from ever being selected. Thus, the ego computing device may avoid using processing resources required to store the removed nodes or removed branches in hierarchical node graph storage and / or trajectory selection.

[0099] In another example, the ego computing device can maintain a defined size of the hierarchical node graph. For example, when adding an interaction node to the hierarchical node graph, the ego computing device may determine whether adding the node will cause the hierarchical node graph to have a size (e.g., number of nodes) that exceeds a threshold. The ego computing device can instantiate and increment a counter at each node (e.g., each interaction node) of the hierarchical node graph and increment the counter for the added node. In response to determining that the new node causes the hierarchical node graph to have or will have a size that exceeds the threshold, the ego computing device can identify node scores of different nodes (e.g., different interaction nodes) and remove interaction nodes with node scores below the threshold, including the interaction node with the lowest interaction score or any nodes subordinate to the selected node. The ego computing device can decrement the counter according to the number of nodes removed. Thus, the ego computing device can maintain the hierarchical node graph as the ego computing device detects additional agent objects and maintain a constant or consistent size to maintain processing requirements and avoid processing spikes.

[0100] 4A-4F, an ego computing device can generate a hierarchical node graph 400 based on detected agent objects in an environment surrounding the ego object. The hierarchical node graph 400 can include a goal layer 402, a trajectory layer 404, an interaction layer 406, an interaction layer 408, and a goal layer 410. The interaction layer 406 can be a first interaction layer. The interaction layer 408 can be a subsequent interaction layer. The ego computing device can generate the hierarchical node graph 400 based on lanes 412, occupancies 414, and moving objects (e.g., agent objects) 416 that the ego computing device detects from image data from the ego object's camera.

[0101] To generate hierarchical node graph 400, the ego computing device can use a step-based approach. For example, the ego computing device can first generate goal nodes 418, 420, 422, and 424 and add goal nodes 418, 420, 422, and 424 to hierarchical node graph 400. The ego computing device can generate a node score for each goal node 418, 420, 422, and 424. The ego computing device can compare the node scores at goal nodes 418, 420, 422, and 424 to a threshold. The ego computing device can determine a goal node at each of goal nodes 418, 420, and 422, but not at goal node 424. Thus, the ego computing device may generate trajectory nodes 426, 428, 430, and 432 that are linked to different goal nodes 418, 420, and 422, respectively, but are not linked to goal node 424 because the node score of goal node 424 is below a threshold. Trajectory nodes and trajectory layers may or may not be included in the hierarchical data structure created by the ego computing device when implementing the systems and methods described herein. Trajectory nodes 426, 428, 430, and 432 may correspond to different movements, paths, or trajectories that the ego object can take to achieve the goals of goal nodes 418, 420, and 422 to which trajectory nodes 426, 428, 430, and 432 link or depend. The trajectory nodes may store data (e.g., movement speed and location) for each trajectory within trajectory nodes 426, 428, 430, and 432.

[0102] The ego computing device can use a function or machine learning model on the data in trajectory nodes 426, 428, 430, and 432 to generate node scores for trajectory nodes 426, 428, 430, and 432. The ego computing device can compare the node scores for trajectory nodes 426, 428, 430, and 432 to a threshold. The ego computing device can identify trajectory nodes that exceed the threshold and generate interaction nodes for the first interaction layer based on the identified trajectory nodes. For example, the ego computing device can determine that the node score for trajectory node 430 exceeds a threshold and, in response to the determination, generate interaction nodes 434 and 436 linked to and / or dependent on trajectory node 430. The interaction at interaction node 434 can be running in front of a pedestrian, as shown in image 448. The interaction at interaction node 436 may be to let the pedestrian pass (e.g., yield to the pedestrian) and turn left after the pedestrian, as shown in image 450. The ego computing device may determine node scores at interaction nodes 434 and 436 and compare the node scores to a threshold (e.g., the same threshold as the threshold used for the node scores of trajectory nodes 426, 428, 430, and 432, or a different threshold). The ego computing device may determine that the node score of interaction node 436 exceeds the threshold, but that the node score of interaction node 434 does not. Thus, the ego computing device may not link any additional interaction nodes to interaction node 434, but may add interaction nodes 440, 442, and 443 to the hierarchical node graph 400 that are subordinate to (e.g., linked to) interaction node 436. The interaction at interaction node 440 may be to drive behind a pedestrian but in front of another vehicle per the interaction at interaction node 436, as shown in image 452.The interaction at interaction node 442 may be to let pedestrians and vehicles pass (e.g., yield to pedestrians and vehicles) and make a left turn, as shown in image 454. The ego computing device may repeat this process for any number of interaction nodes and / or interaction layers.

[0103] In some cases, the ego computing device may generate a goal node dependent on the interaction node. For example, the ego computing device may generate a goal node 446 dependent on the interaction node 442 if the ego computing device determines that the occurrence of the interaction of the interaction node 442 triggers another goal. The ego computing device may make such a determination by determining that the data of the interaction node 442 satisfies a condition stored in memory. In one example, the interaction may be to have the agent object complete a left turn after passing the ego object. The ego computing device may analyze the new state after the interaction node 442 and generate a goal node 446 for the ego object to proceed straight in the new lane it is traveling in. The new goal may generate a hierarchical node graph corresponding to a new trajectory or to initiate an iteration of the method 200 to select a trajectory.

[0104] Referring again to FIG. 2 , in step 208, the ego computing device determines a trajectory score for each of a plurality of trajectories in the hierarchical node graph. A trajectory can be or include a goal node and one or more interaction nodes linked to each other between layers of interaction nodes. For example, a trajectory may include a node corresponding to a scenario in which the ego object has a goal of turning left and the ego computing device detects a crossing pedestrian and another passing vehicle. The trajectory may include a node corresponding to a pedestrian crossing a road and turning left after a vehicle passes. Another trajectory in the scenario may be turning left in front of the pedestrian. Another trajectory in the scenario may be turning left behind the pedestrian before the vehicle passes. Another trajectory may be completely avoiding the pedestrian and the passing vehicle and turning right instead. There may be any number of trajectories corresponding to nodes in the hierarchical node graph. The ego computing device may determine a trajectory score for a trajectory according to or otherwise based on the node scores of the trajectory's nodes. For example, the ego computing device may perform a function (e.g., sum or aggregation techniques, median, mean, weighted sum, weighted average, etc.) or a machine learning model on the node scores of the nodes in each trajectory to generate a trajectory score for each trajectory.

[0105] In some cases, the ego computing device can determine multiple trajectory scores for each trajectory. Different trajectory scores can correspond to different factors (e.g., physics-based constraints (e.g., collision checking), comfort analysis, intervention potential, human-like discriminators, etc.). The ego computing device can determine the trajectory scores for the factors using a function or machine learning model as described above for the node scores for each factor. In some cases, the ego computing device can combine different trajectory scores for a trajectory to generate a single trajectory score, such as by using a function or machine learning model to do so.

[0106] In some cases, the ego computing device may remove all trajectories from the hierarchical node graph. The ego computing device may remove individual trajectories from the hierarchical node graph by storing a flag or indication in memory indicating not to use the trajectory or by removing the trajectory's nodes. The ego computing device may remove a trajectory in response to determining that the trajectory score is below a threshold or in response to determining that the trajectory score meets another condition, such as the lowest trajectory score of the trajectories in the hierarchical node graph. In doing so, the ego computing device may reduce the size of the hierarchical node graph or otherwise avoid wasting processing resources evaluating low-scoring trajectories.

[0107] In step 210, the ego computing device selects a trajectory for the ego object. The ego computing device may select a trajectory for the ego object based on a trajectory score for the trajectory determined by the ego computing device from the hierarchical node graph. The ego object may select a trajectory in response to determining that the trajectory score for the trajectory satisfies a condition. In one example, the ego object may select a trajectory in response to determining that the trajectory score for the trajectory is the highest of the trajectory scores determined by the ego computing device. In response to selecting the trajectory, the ego computing device may control the ego object according to the selected trajectory.

[0108] The various illustrative logic blocks, modules, circuits, and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, computer software, or a combination of both. To clearly illustrate this interchangeability between hardware and software, the various illustrative components, blocks, modules, circuits, and steps have been described generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the particular application and design constraints imposed on the overall system. Those skilled in the art may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present disclosure or the claims.

[0109] Computer software-implemented embodiments may be implemented in software, firmware, middleware, microcode, hardware description languages, or any combination thereof. A code segment or machine-executable instruction may represent a procedure, a function, a subprogram, a program, a routine, a subroutine, a module, a software package, a class, or any combination of instructions, data structures, or program statements. A code segment may be coupled to another code segment or a hardware circuit by passing and / or receiving information, data, arguments, parameters, or memory contents. Information, arguments, parameters, data, etc. may be passed, forwarded, or transmitted via any suitable means including memory sharing, message passing, token passing, network transmission, etc.

[0110] The actual software code or specialized control hardware used to implement these systems and methods is not a limitation of the claimed features or this disclosure. Accordingly, the operation and behavior of the systems and methods are described without reference to specific software code, with the understanding that software and control hardware can be designed to implement the systems and methods based on the description herein.

[0111] If implemented in software, the functions may be stored as one or more instructions or code on a non-transitory, computer-readable, or processor-readable storage medium. The steps of a method or algorithm disclosed herein may be embodied in a processor-executable software module, which may reside on a computer-readable or processor-readable storage medium. Non-transitory computer-readable or processor-readable media includes both computer storage media and tangible storage media that facilitate transfer of a computer program from one place to another. Non-transitory processor-readable storage media may be any available medium that can be accessed by a computer. By way of example, and not limitation, such non-transitory processor-readable media may include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other tangible storage medium that can be used to store desired program code in the form of instructions or data structures and that can be accessed by a computer or processor. As used herein, disk and disc include compact discs (CDs), laser discs, optical discs, digital versatile discs (DVDs), Blu-ray discs, and floppy disks, with a "disc" typically reproducing data magnetically and a "disc" reproducing data optically using a laser. Combinations of the above should also be included within the scope of computer-readable media. Furthermore, the operations of a method or algorithm may reside as one or any combination or set of code and / or instructions on a non-transitory processor-readable medium and / or computer-readable medium, which may be incorporated into a computer program product.

[0112] The previous description of the disclosed embodiments is provided to enable any person skilled in the art to make or use the embodiments described herein and variations thereof. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the principles defined herein may be applied to other embodiments without departing from the spirit or scope of the subject matter disclosed herein. Thus, the present disclosure is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the following claims and the principles and novel features disclosed herein.

[0113] While various aspects and embodiments have been disclosed, other aspects and embodiments are contemplated. The various disclosed aspects and embodiments are for purposes of illustration and are not intended to be limiting, with the true scope and spirit being indicated by the following claims.

Claims

1. detecting, by a processor, one or more agent objects in a space around the ego object using image data captured by a camera of the ego object; storing, by the processor, a hierarchical node graph, the hierarchical node graph comprising: a goal layer comprising one or more goal nodes corresponding to goals for the ego object to achieve; a plurality of interaction layers of interaction nodes following the goal layer, each interaction node corresponding to at least one of a plurality of trajectories in the ego object taking into account the one or more agent objects and corresponding to a node score, the plurality of interaction layers comprising a first interaction layer of interaction nodes and a plurality of subsequent interaction layers of interaction nodes following the first interaction layer of interaction nodes, each interaction node in the plurality of subsequent interaction layers being dependent on at least one interaction node in a previous interaction layer of the plurality of interaction layers; and adding, by the processor, a second interaction node to a subsequent interaction layer of interaction nodes of the plurality of subsequent interaction layer interaction nodes in response to determining that a node score of a first interaction node in the initial interaction layer of interaction nodes exceeds a threshold, wherein the second interaction node is linked to the first interaction node; determining, by the processor, a trajectory score for each of a plurality of trajectories based on one or more node scores of one or more nodes corresponding to the trajectory; selecting, by the processor, one trajectory of the plurality of trajectories for the ego object based on the trajectory score for the trajectory; A method comprising:

2. The method of claim 1 further comprising the step of controlling, by the processor, the ego object according to the selected trajectory.

3. The method of claim 1 , wherein determining the trajectory score for the trajectory comprises aggregating, by the processor, the one or more node scores of the one or more nodes of the trajectory.

4. 2. The method of claim 1 , wherein selecting the trajectory comprises selecting the trajectory in response to the processor determining that the trajectory score for the trajectory is higher than trajectory scores of other trajectories in the plurality of trajectories.

5. The method of claim 1 , further comprising: executing, by the processor, a neural network to determine the node score at each interaction node in the hierarchical node graph.

6. The method of claim 1 , wherein the node score for each interaction node corresponds to a pleasantness associated with the interaction node.

7. The method of claim 1 , wherein the node score for each interaction node corresponds to a comfort associated with the interaction node and a likelihood of intervention associated with the interaction node.

8. 2. The method of claim 1, further comprising generating, by the processor, the hierarchical node graph in response to detecting the one or more agent objects in a space surrounding the ego object using image data captured by a camera of the ego object.

9. executing, by the processor, an analysis protocol to determine the node score at each interaction node in the hierarchical node graph; comparing, by the processor, the node score to a threshold; removing, by the processor, from the hierarchical node graph, each interaction node in the hierarchical node graph that corresponds to a node score below the threshold; The method of claim 1 further comprising:

10. 2. The method of claim 1, further comprising: in response to determining that adding the second interaction node to the subsequent layer of interaction nodes in the plurality of subsequent layers would cause the number of nodes in the hierarchical node graph to exceed a threshold, removing, by the processor, a third node from the hierarchical node graph based on a node score of the third node.

11. It is an ego object, A camera and a processor; a non-transitory computer-readable medium configured to be executed by the processor; Equipped with The processor: Detecting one or more agent objects in a space around the ego object using image data captured by the camera; storing a hierarchical node graph, the hierarchical node graph comprising: a goal layer comprising one or more goal nodes corresponding to goals for the ego object to achieve; a plurality of interaction layers of interaction nodes following the goal layer, each interaction node corresponding to at least one of a plurality of trajectories in the ego object taking into account the one or more agent objects and corresponding to a node score, the plurality of interaction layers comprising a first interaction layer of interaction nodes and a plurality of subsequent interaction layers of interaction nodes following the first interaction layer of interaction nodes, each interaction node in the plurality of subsequent interaction layers being dependent on at least one interaction node in a previous interaction layer of the plurality of interaction layers; Equipped with in response to determining that a node score of a first interaction node in the initial interaction tier of interaction nodes exceeds a threshold, adding a second interaction node to a subsequent interaction tier of interaction nodes of the plurality of subsequent interaction tiers, the second interaction node being linked to the first interaction node; determining a trajectory score for each of the plurality of trajectories based on one or more node scores of one or more nodes corresponding to the trajectory; selecting a trajectory from the plurality of trajectories in the ego object based on the trajectory score for the trajectory; The ego object is constructed in this way.

12. The ego object of claim 11 , wherein the processor is further configured to control the ego object according to the selected trajectory.

13. The ego object of claim 11 , wherein the processor is configured to determine the trajectory score for the trajectory by aggregating the one or more node scores of the one or more nodes of the trajectory.

14. 12. The ego object of claim 11, wherein the processor is configured to select the trajectory by selecting the trajectory in response to determining that the trajectory score for the trajectory is higher than trajectory scores of other trajectories of the plurality of trajectories.

15. The ego object of claim 11 , wherein the processor is further configured to execute a neural network to determine the node score at each interaction node of the hierarchical node graph.

16. The ego object of claim 11 , wherein the node score at each interaction node corresponds to a pleasantness associated with the interaction node.

17. The ego object of claim 11 , wherein the node score at each interaction node corresponds to a pleasantness associated with the interaction node and a likelihood of intervention associated with the interaction node.

18. The ego object of claim 11 , wherein the processor is further configured to generate the hierarchical node graph in response to detecting the one or more agent objects in a space surrounding the ego object using image data captured by a camera of the ego object.

19. The processor: executing an analysis protocol to determine the node score at each interaction node of the hierarchical node graph; comparing the node score to a threshold; removing from the hierarchical node graph each interaction node in the hierarchical node graph that corresponds to a node score below the threshold; The ego object of claim 11 further configured to:

20. 12. The ego object of claim 11, wherein the processor is further configured to, in response to determining that adding the second interaction node to the subsequent layer of interaction nodes in the plurality of subsequent layers would cause a number of nodes in the hierarchical node graph to exceed a threshold, remove a third node from the hierarchical node graph based on a node score of the third node.

Images