AI inference compiler and runtime toolchain

Abstract

Embodiments include systems and methods for processing sensor data and generating operational instructions for ego (e.g., autonomous vehicle, robot) hardware. The ego includes any number of machine learning architectures, often neural network architectures, for processing the sensor data, perceiving the ego's surrounding environment, and making decisions regarding the ego's behavior. The ego's neural network architecture ingests the sensor data and uses the sensor data to perform any number of operations related to a particular domain or task, such as object recognition or path planning. A graph partitioner is trained to assign functions in the neural network's software and sensor data to specific hardware processing units. Some compilers are used to generate instructions based on the assigned type of processing unit.

Description

[Technical Field]

[0001] [CROSS-REFERENCE TO RELATED APPLICATIONS] This application claims priority to U.S. Provisional Application No. 63 / 377,954, filed September 30, 2022, which is incorporated by reference herein in its entirety for all purposes.

[0002] This application relates generally to implementing neural network architectures for autonomous vehicles or other autonomous electronic devices, and more particularly to systems and methods for remotely and efficiently compiling and deploying such neural network architectures. [Background technology]

[0003] Autonomous navigation technology used in autonomous vehicles and robots (sometimes called "egos") has become widespread due to rapid advances in computer technology. These advances enable safer and more reliable autonomous navigation for egos. Egos often must navigate through complex and dynamic environments and terrain that may include vehicles, traffic, pedestrians, cyclists, and a variety of other static or dynamic obstacles. Making informed and appropriate decisions to avoid collisions requires understanding the ego's surroundings. This involves developing and deploying complex neural network architectures on the ego.

[0004] Increasing data volumes and feature sophistication naturally raise resource demand issues and require solutions to improve computational efficiency in both the hardware and software components of the ego. This requires sophisticated mechanisms for compiling and deploying neural network architectures on the ego. In some situations, the challenge becomes heightened, for example, by remote deployment from a software development source to a remote ego, and / or by deploying software updates of neural network architectures to the ego's predetermined execution hardware, among other things. Summary of the Invention

[0005] The embodiments described herein include systems and methods that address various shortcomings in the art and may also provide various additional or alternative advantages. The embodiments include hardware and software configurations that improve performance in processing sensor data by software components, as well as computational hardware for an ego (e.g., an autonomous vehicle, a robot). The ego includes any number of machine learning architectures, often neural network architectures, for processing the sensor data, perceiving the ego's surrounding environment, and making decisions regarding the ego's behavior. The ego's neural network architecture ingests the sensor data and uses the sensor data to perform any number of operations related to a particular domain or task, such as object recognition or path planning. Any number of compilers translate the software functions of the neural network architecture and the sensor data into machine code executable instructions for execution by the hardware components.

[0006] An embodiment may include a method including the steps of: obtaining, by a computer, software programming comprising a plurality of functions of a plurality of sub-neural networks of a neural network architecture; assigning, by the computer, one or more of the sub-neural networks to a plurality of processing units of an ego, wherein for each sub-neural network, the computer assigns a processing unit for computing the plurality of functions of the sub-neural network; and generating, by the computer, a plurality of execution instructions for the plurality of processing units by running a plurality of compilers for the plurality of functions of each sub-neural network, wherein, for each execution instruction, the computer uses one of the plurality of compilers according to the processing units of the ego assigned to the plurality of functions of the sub-neural network.

[0007] The method may include generating, by the computer, a computer file including a plurality of execution instructions for a plurality of processing units to execute the plurality of sub-neural networks.

[0008] The method may include the step of transmitting, by the computer, a computer file to the ego.

[0009] At least one execution instruction may cause the ego's circuitry, which comprises multiple chips, to operate in an enhanced computation mode for parallel execution of multiple execution instructions.

[0010] The at least one execution instruction may instruct an ego circuit having multiple chips including multiple processing units to operate in a redundant mode for primary execution of the multiple execution instructions by a primary chip of the multiple chips.

[0011] The method may include applying, by the computer, a schedule optimizer engine to the execution instructions to generate an execution schedule for the execution instructions, the schedule optimizer engine comprising a neural network layer trained to generate the execution schedule to minimize latency.

[0012] The one or more processing units may include at least one of a GPU, a CPU, or an accelerator device.

[0013] The one or more processing units may be heterogeneous, including at least two types of processing units.

[0014] The computer may assign processing units of the ego's multiple processing units to apply the sub-neural networks according to one or more graphs representing the ego's circuit architecture having multiple processing units.

[0015] The method may include training, by a computer, one or more sub-neural networks for quantization recognition training based on one or more processing units by applying each sub-neural network to a training data set including data with intended quantization characteristics.

[0016] An embodiment may include a system including a computer with a processor, the computer configured to: acquire software programming comprising a plurality of functions of a plurality of sub-neural networks of a neural network architecture; assign one or more of the sub-neural networks to a plurality of processing units of an ego; for each sub-neural network, the computer assigns a processing unit for computing the plurality of functions of the sub-neural network; and generate a plurality of execution instructions for the plurality of processing units by running a plurality of compilers for the plurality of functions of each sub-neural network. For each execution instruction, the computer uses one of the plurality of compilers according to the processing unit of the ego assigned to the plurality of functions of the sub-neural network.

[0017] The computer may be further configured to generate a computer file including a plurality of execution instructions for the plurality of processing units to execute the plurality of sub-neural networks.

[0018] The computer may be further configured to transmit the computer file to the ego.

[0019] At least one execution instruction may cause multiple chips of the ego to operate in an enhanced computation mode for parallel execution of multiple execution instructions.

[0020] The at least one execution instruction may cause the plurality of chips of the ego to operate in a redundant mode for primary execution of the plurality of execution instructions by a primary chip of the plurality of chips.

[0021] The computer may be further configured to apply a schedule optimizer engine to the execution instructions to generate an execution schedule for the execution instructions, the schedule optimizer engine comprising a neural network layer trained to generate instructions to minimize latency.

[0022] The multiple processing units may include at least one of a GPU, a CPU, or an accelerator device.

[0023] The plurality of processing units may be heterogeneous, including at least two types of processing units.

[0024] The computer may assign processing units of the ego's multiple processing units to the sub-neural networks according to one or more graphs representing the ego's circuit architecture having multiple processing units.

[0025] The computer may be further configured to train one or more sub-neural networks for quantization recognition training based on the one or more processing units by applying each sub-neural network to a training data set stored in the database, the training data set including data with intended quantization characteristics.

[0026] An embodiment may include an ego comprising a circuit board comprising a plurality of system-on-chip (SOC) devices and a plurality of microcontrollers corresponding to the SOC devices; and a processor of the SOC devices, wherein the processor is configured to send an initial timing message to a first microcontroller at an initial time according to a kernel clock of the processor; receive a response message from the first microcontroller indicating a response time at a first controller clock of the first microcontroller; determine a completion time according to the kernel clock of the processor in response to receiving the response message from the first microcontroller; calculate an error rate in the kernel clock representing a difference between the kernel clock of the processor and the controller clock of the first microcontroller based on the initial clock time, the completion time, and the response time; and adjust a frequency of the kernel clock based on the error rate to reduce the error rate between the kernel clock and the first controller clock.

[0027] The processor of the SOC device is configured to send an initial timing message in response to receiving a wake-up command from the first microcontroller.

[0028] The first microcontroller coupled to the processor may be configured to calculate a second error rate representing a difference between a first controller clock of the first microcontroller and a second controller clock of the second microcontroller based on an initial clock time between the first microcontroller and the second microcontroller, a completion time between the first microcontroller and the second microcontroller, and a response time between the first microcontroller and the second microcontroller, and adjust a second frequency of the second controller clock based on the second error rate to reduce the second error rate between the first controller clock and the second controller clock.

[0029] The first microcontroller is configured to determine whether a difference between a first controller clock and a second controller clock of the first microcontroller meets a threshold difference.

[0030] The first microcontroller is configured to transmit an initial timing message in response to executing a wake-up function of the first microcontroller.

[0031] The second microcontroller is configured to execute a restart function, and a second controller clock of the second microcontroller is updated to match a first controller clock of the first microcontroller indicated in a timing message received at the second microcontroller from the first microcontroller for the restart function.

[0032] An embodiment may include a method including: sending, by a processor of a system-on-chip (SOC), an initial timing message to a first microcontroller at an initial time according to a kernel clock of the processor; receiving, by the processor, a response message from the first microcontroller indicating a response time at a first controller clock of the first microcontroller; in response to receiving the response message, determining, by the processor, a completion time according to the kernel clock of the processor; calculating, by the processor, an error rate at the kernel clock representing a difference between the kernel clock of the processor and the first controller clock of the first microcontroller based on the initial clock time, the completion time, and the response time; and adjusting, by the processor, a frequency of the kernel clock based on the error rate to reduce the error rate between the kernel clock and the first controller clock.

[0033] The method may include receiving, by a processor, a wake-up command from the first microcontroller, the processor of the SOC device being configured to transmit an initial timing message in response to receiving the wake-up command.

[0034] The processor and the first microcontroller exchange one or more timing messages at startup according to a boot loader function of the first microcontroller.

[0035] The method may include calculating, by the first microcontroller, a second error rate representing a difference between a first controller clock of the first microcontroller and a second controller clock of the second microcontroller based on an initial clock time between the first microcontroller and the second microcontroller, a completion time between the first microcontroller and the second microcontroller, and a response time between the first microcontroller and the second microcontroller; and adjusting, in the second microcontroller, a second frequency of the second controller clock based on the second error rate to reduce the second error rate between the first controller clock and the second controller clock.

[0036] An embodiment may include a circuit board comprising a plurality of system-on-chip (SOC) devices and a plurality of microcontrollers corresponding to the SOC devices; and an ego comprising a first microcontroller, wherein the first microcontroller is configured to: send an initial timing message to a second microcontroller at an initial time according to a first controller clock of the first microcontroller; receive a response message from the second microcontroller indicating a response time at the second controller clock of the second microcontroller; determine a completion time according to the first controller clock of the first microcontroller in response to receiving the response message from the second microcontroller; calculate an error rate representing a difference between the first controller clock of the first microcontroller and the second controller clock of the second microcontroller based on the initial clock time, the completion time, and the response time; and adjust a frequency of the second controller clock based on the error rate to reduce the error rate between the first controller clock and the second controller clock.

[0037] The first microcontroller may be configured to determine whether a difference between a first controller clock and a second controller clock of the first microcontroller meets a threshold difference.

[0038] The first microcontroller may be further configured to send a wake-up signal to a kernel of a processor of a SOC coupled to the first microcontroller.

[0039] The processor of the SOC may be configured to calculate a second error rate representing a difference between a kernel clock of a kernel of the processor and a first controller clock of the first microcontroller based on an initial clock time between the SOC and the first microcontroller, a completion time between the SOC and the first microcontroller, and a response time between the SOC and the first microcontroller, and adjust a kernel frequency of the kernel clock based on the error rate to reduce the error rate between the kernel clock and the first controller clock.

[0040] The second microcontroller may be configured to send a second wake-up signal to a second kernel of a second processor of a second SOC coupled to the second microcontroller.

[0041] The second microcontroller may perform a restart function, and the second controller clock is updated to match the first controller clock indicated in the timing message received at the second microcontroller from the first microcontroller.

[0042] An embodiment may include a method including: sending, by a first microcontroller coupled to a first SOC, an initial timing message to a second microcontroller coupled to a second SOC at an initial time according to a first controller clock of the first microcontroller; receiving, by the first microcontroller, a response message from the second microcontroller indicating a response time in a second controller clock of the second microcontroller; determining, by the first microcontroller, a completion time according to the first controller clock of the first microcontroller in response to receiving the response message from the second microcontroller; calculating, by the first microcontroller, an error rate representing a difference between the first controller clock of the first microcontroller and the second controller clock of the second microcontroller based on the initial clock time, the completion time, and the response time; and adjusting, by the first microcontroller, a frequency of the second controller clock based on the error rate to reduce the error rate between the first controller clock and the second controller clock.

[0043] The first microcontroller may be configured to determine whether a difference between a first controller clock and a second controller clock of the first microcontroller meets a threshold difference.

[0044] The method may include sending, by the first microcontroller, a wake-up signal to a kernel of a processor of a SOC coupled to the first microcontroller.

[0045] The method may include calculating, by a processor of the first SOC, a second error rate representing a difference between a kernel clock of a kernel of the processor and a first controller clock of the first microcontroller based on an initial clock time between the SOC and the first microcontroller, a completion time between the SOC and the first microcontroller, and a response time between the SOC and the first microcontroller; and adjusting, by the processor of the first SOC, a kernel frequency of the kernel clock based on the error rate to reduce the error rate between the kernel clock and the first controller clock.

[0046] It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory and are intended to provide further explanation of the invention as claimed. [Brief explanation of the drawings]

[0047] Non-limiting embodiments of the present disclosure are described by way of example with reference to the accompanying drawings, which are schematic and not intended to be drawn to scale, and unless indicated as representing background art, the figures represent aspects of the present disclosure.

[0048] [Figure 1A] 1 illustrates components of an AI-enabled visual data analysis system for Ego, according to one embodiment.

[0049] [Figure 1B] 1 illustrates various sensors associated with a vehicle (or other type of ego), according to one embodiment.

[0050] [Figure 1C] 1 illustrates components of an ego, according to one embodiment.

[0051] [Figure 1D] 1 illustrates certain hardware and software components of an ego for performing fully or partially autonomous driving (SD) operations, according to one embodiment.

[0052] [Figure 1E] 1 illustrates certain hardware and software components of an ego for clock synchronization, according to one embodiment.

[0053] [Figure 2A] 1 illustrates data flow between hardware and software computing components of an ego's computing system, according to one embodiment. [Figure 2B] 1 illustrates data flow between hardware and software computing components of an ego's computing system, according to one embodiment.

[0054] [Figure 2C] 1 illustrates execution instructions placed on an execution schedule for execution by IC hardware components of a system, according to one embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0055] Reference will now be made to the exemplary embodiments illustrated in the drawings, and specific language will be used herein to describe the embodiments. It will nevertheless be understood that no limitation of the scope of the claims or this disclosure is intended thereby. Alterations and further modifications of the features of the invention shown herein, and further applications of the principles of the subject matter shown herein that will occur to those skilled in the art in possession of this disclosure, are to be considered within the scope of the subject matter disclosed herein. Other embodiments may be used and / or other changes may be made without departing from the spirit or scope of the present disclosure. The exemplary embodiments described in the detailed description are not meant to limit the presented subject matter.

[0056] The embodiments described herein include systems and methods that address various shortcomings in the art and may also provide various additional or alternative advantages. The embodiments include hardware and software configurations that improve performance in processing sensor data by software components, as well as computational hardware for an ego (e.g., an autonomous vehicle, a robot). The ego includes any number of machine learning architectures, often neural network architectures, for processing the sensor data, perceiving the ego's surrounding environment, and making decisions regarding the ego's behavior.

[0057] At the software level, ego's local or remote computing device may execute various software routines of a neural network architecture (or other machine learning architecture). The software defines the layers and functions of the neural network architecture, or defines a hierarchical parent neural network architecture and one or more hierarchical child neural network architectures (sometimes called child networks or sub-networks). ego's neural network architecture ingests sensor data and uses the sensor data to perform any number of operations related to a particular domain or task, such as object recognition or path planning.

[0058] At the hardware level, the ego includes various types of computing hardware resources, including, among other things, various integrated circuit (IC) components and associated firmware or controllers. The computing hardware components realize and execute the software functions of the neural network architecture using the sensor data. Any number of compilers translate the software functions of the neural network architecture and the sensor data into machine code executable instructions for execution by the hardware components.

[0059] Embodiments include various software routines for improving the performance and efficiency of processing sensor data within computing hardware by partitioning sensor data into multiple data partitions or data portions and partitioning the architecture structure in sub-networks. The software routines may include a machine learning architecture including a neural network layer that defines a graph partitioner. The graph partitioner is configured and trained to assign sensor data portions to specific functions of the sub-network and then assign one or more hardware processing units (e.g., GPUs, CPUs, dedicated hardware AI accelerator devices) to perform the functions using the sensor data portions. The neural network of the graph partitioner is trained to identify and assign functions to apply to the sensor data based on, for example, the type of sensor data or the type of sensor feature. In some embodiments, the graph partitioner may include hard-coded or pre-configured mappings between the type of sensor data or feature and the software functions of the neural network architecture.

[0060] The graph partitioner may be configured and trained to identify and assign functional sensor data portions to specific hardware processing units. The graph partitioner may be trained to select hardware processing units to perform functions based on desired performance behavior or outcomes, such as optimizing computing hardware efficiency, maximizing a computing hardware performance metric, or minimizing a computing hardware performance metric. For example, the graph partitioner may be trained to assign specific advanced functions to specially designed AI accelerator devices.

[0061] Embodiments may include a heterogeneous collection of hardware processing units. The graph partitioner may be trained to identify and assign compilers capable of generating executable instructions having machine code compatible with the assigned processing units. The graph partitioner may be trained, for example, to optimize the functionality of the computing hardware and assign processing units to perform functions to maximize or minimize a particular performance behavior of the computing hardware. By training the graph partitioner to dynamically assign heterogeneous processing units to perform the specified functions of the neural network architecture, the graph partitioner optimizes the execution of the neural network function in hardware. This improves the accuracy and performance of the neural network architecture when analyzing sensor data.

[0062] Embodiments include a schedule optimizer pre-configured or trained to identify relationships between different partitions of sensor data and neural network architectures. The schedule optimizer is used to combine multiple compiled pieces of code (e.g., execution instructions) into one or more executable files. In some cases, linking generates a sequence or configuration for executing instructions by hardware components. In these cases, the schedule optimizer may generate an execution schedule that minimizes latency by, for example, applying the linking engine's trained neural network architecture to determine dependencies between data, functions to be performed by processing units, and processing units assigned to perform the functions. The linking engine may then identify an execution schedule to optimize performance against dependency constraints. In this way, the schedule optimizer ensures that execution instructions are organized to reduce delay and latency, thereby enabling improved real-time processing of sensor data.

[0063] The ego's downstream hardware and software may incorporate outputs generated by the SD circuitry implementing the neural network architecture, such as the trajectory, speed, and other navigation decisions of the path planning network, to operate or manipulate the ego within the environment.

[0064] The ego hardware includes an SD circuit (e.g., an integrated circuit (IC) substrate) having two (or more) system-on-chip (SOC) chips or similar types of IC devices. The SOC chips may perform the functions of a particular neural network architecture by executing executable instructions compiled as an execution library from the source code of the particular neural network architecture. A server in the development system trains the neural network architecture on historical and / or current sensor data and the output of other neural network architectures. The server then applies the compiler toolchain described herein to the source code and data of the trained neural network architecture to generate an execution library. A neural network architecture in the graph partitioner is trained to assign portions of the execution library to corresponding processing units of the SOC chips. In some cases, a first neural network architecture is programmed to request or expect data input from a second neural network architecture. When the first SOC chip loads and executes instructions for execution in a first neural network architecture and the second SOC chip loads and executes instructions for execution in a second neural network architecture, the second SOC chip requests or expects data input from the first SOC chip. The SD circuitry includes a bus that allows the SOC chips to communicate data signals.

[0065] Embodiments include hardware and / or software components within the ego's circuitry that enable the SOC chips on the SD circuit board to operate as if they were functioning on a single synchronized clock. In some embodiments, the SD circuitry includes two (or more) SOC chips, each coupled to an SOC microcontroller, which may be any microcontroller device or similar processing circuit unit that maintains the chip clocks of the respective SOC chips. The SOC chips are simultaneously powered by the ego's computer processor so that the SOC clocks of the SOCs are as close to each other as possible. Each SOC chip includes an operating system (OS) kernel that manages the specific operations of its respective SOC chip. At predetermined intervals, each SOC chip exchanges time messages with its respective microcontroller to verify that the SOC's kernel clock is within a threshold distance from the SOC's controller clock and to correct or adjust the SOC's microcontroller as necessary. In this manner, each SOC chip maintains clock synchronization between its respective OS kernel and its respective microcontroller. Additionally, at preset intervals, the SOC microcontrollers exchange time messages to verify that the first controller-clock of the first SOC microcontroller is within a threshold distance of the second SOC controller-clock of the second SOC microcontroller. If the threshold distance is exceeded, the second microcontroller may be corrected or adjusted to reduce the controller-clock distance. In this manner, the microcontrollers maintain clock synchronization between the SOC microcontrollers, and by extension, between the SOC chips.

[0066] In some embodiments, the SD circuitry includes a controller or other processing unit that maintains clock synchronization between the SOC chips based on interpreting the timestamps of the sensor inputs (e.g., the timestamps of the camera inputs) and converting the timestamps between the SOC chips according to the difference between the current chip clocks of each SOC chip.

[0067] FIG. 1A is a non-limiting example of components of a system 100 capable of implementing the methods and systems discussed herein. For example, an analytics server may train an AI model and use the trained AI model to generate occupancy datasets and / or maps for one or more egos. FIG. 1A illustrates components of an AI-enabled visual data analytics system 100. System 100 may include analytics server 110a, system database 110b, administrator computing device 120, egos 140a-b (collectively ego 140), ego computing devices 141a-c (collectively ego computing devices 141), and server 160. System 100 is not limited to the components described herein and may include additional or other components not shown for purposes of brevity, which components should be considered within the scope of the embodiments described herein.

[0068] The above components may be connected via a network 130. Examples of network 130 may include, but are not limited to, a private or public LAN, a WLAN, a MAN, a WAN, and the Internet. Network 130 may include wired and / or wireless communications according to one or more standards and / or over one or more transport media.

[0069] Communications over network 130 may occur according to various communication protocols, such as Transmission Control Protocol / Internet Protocol (TCP / IP), User Datagram Protocol (UDP), IEEE communications protocols, etc. In one example, network 130 may include wireless communications according to the Bluetooth® set of specifications or another standard or proprietary wireless communications protocol. In another example, network 130 may also include communications over cellular networks, including, for example, Global System for Mobile Communications (GSM), Code Division Multiple Access (CDMA), or Enhanced Data for Global Evolution (EDGE) networks.

[0070] System 100 illustrates an example of a system architecture and components that may be used to train and execute one or more AI models, such as AI model 110c. Specifically, as depicted in FIG. 1A and described herein, analytics server 110a may train AI model 110c using data retrieved from ego 140 (e.g., by using data streams 172 and 174) using methods described herein. Once AI model 110c is trained, each ego 140 may access the trained AI model 110c and execute it. For example, vehicle 141a with ego computing device 140a may send its camera feed to trained AI model 110c and determine the occupancy status of its surroundings (e.g., data stream 174). Furthermore, data captured and / or predicted by AI model 110c for ego 140 (during inference) may be used to improve AI model 110c. Thus, system 100 describes a continuous loop that can periodically improve the accuracy of AI model 110c. Furthermore, the system 100 loops in such a way that the data received by the ego 140 can be used in the learning phase as well as the inference phase.

[0071] Analytics server 110a may be configured to collect, process, and analyze navigation data (e.g., images captured while navigating) and various sensor data collected from ego 140. The collected data may then be processed and prepared into a training dataset. The training dataset may then be used to train one or more AI models, such as AI model 110c. Analytics server 110a may also be configured to collect visual data from ego 140. Using AI model 110c (trained using the methods and systems discussed herein), analytics server 110a may generate a dataset and / or an occupancy map for ego 140. Analytics server 110a may display the occupancy map on ego 140 and / or transmit the occupancy map / dataset to ego computing device 141, administrator computing device 120, and / or server 160.

[0072] Although FIG. 1A shows AI model 110c as a component of system database 110b, AI model 110c may be stored in a different or separate component, such as cloud storage or any other data repository accessible to analytics server 110a.

[0073] The analytics server 110a may also be configured to display an electronic platform showing various training attributes for training the AI ​​model 110c. The electronic platform may be displayed on the administrator computing device 120 to allow an analyst to monitor the training of the AI ​​model 110c. An example of an electronic platform generated and hosted by the analytics server 110a may be a web-based application or website configured to display the training dataset collected from the ego 140 and / or the training status / metrics of the AI ​​model 110c.

[0074] Analysis server 110a may be any computing device equipped with a processor and non-transitory machine-readable storage capable of performing the various tasks and processes described herein. Non-limiting examples of such computing devices may include workstation computers, laptop computers, server computers, etc. Although system 100 includes a single analysis server 110a, system 100 may include any number of computing devices operating in a distributed computing environment, such as a cloud environment.

[0075] Ego 140 may represent various electronic data sources that transmit data associated with a previous or current navigation session to analytics server 110a. Ego 140 may be any device configured for navigation, such as vehicle 140a and / or truck 140c. Ego 140 is not limited to being a vehicle and may include robotic devices as well. For example, ego 140 may include robot 140b, which may represent a general-purpose, bipedal, autonomous humanoid robot capable of navigating various terrains. Robot 140b may be provided with software that enables balance, navigation, perception, or interaction with the physical world. Robot 140b may also include various cameras configured to transmit visual data to analytics server 110a.

[0076] Although referred to herein as “ego,” ego 140 may or may not be an autonomous device configured for automatic navigation. For example, in some embodiments, ego 140 may be controlled by a human operator or by a remote processor. ego 140 may include various sensors, such as those depicted in FIG. 1B . The sensors may be configured to collect data as ego 140 navigates various terrains (e.g., roads). Analytics server 110a may collect data provided by ego 140. For example, analytics server 110a may obtain navigation session and / or road / terrain data (e.g., images of ego 140 navigating roads) from various sensors, such that the collected data is ultimately used by AI model 110c for training purposes.

[0077] As used herein, a navigation session corresponds to a journey in which ego 140 travels a route, regardless of whether the journey was autonomous or controlled by a human. In some embodiments, the navigation session may be for data collection and model training purposes. However, in some other embodiments, ego 140 may refer to a vehicle purchased by a consumer, and the purpose of the journey may be classified as daily use. A navigation session may begin when ego 140 travels more than a threshold distance (e.g., 0.1 miles, 100 feet) from a non-moving location or exceeds a threshold speed (e.g., greater than 0 mph, greater than 1 mph, greater than 5 mph). A navigation session may end when ego 140 is returned to a non-moving location and / or turned off (e.g., when the driver exits the vehicle).

[0078] Ego 140 may correspond to a group of egos monitored by analytics server 110a to train AI model 110c. For example, drivers of vehicles 140a may authorize analytics server 110a to monitor data associated with their respective vehicles. As a result, analytics server 110a may collect sensor / camera data using various methods discussed herein and generate a training dataset for training AI model 110c accordingly. Analytics server 110a may then apply the trained AI model 110c to analyze data associated with ego 140 and predict an occupancy map for ego 140. Furthermore, additional / ongoing data associated with ego 140 may be processed and added to the training dataset, allowing analytics server 110a to recalibrate AI model 110c accordingly. Thus, system 100 performs a loop in which navigation data received from ego 140 can be used to train AI model 110c. Ego 140 may include a processor that executes the trained AI model 110c for navigation purposes. During navigation, ego 140 can collect additional data about their navigation session and can use the additional data to calibrate AI model 110c. That is, ego 140 corresponds to an ego that can be used to train, run / use, and recalibrate AI model 110c. In a non-limiting example, ego 140 corresponds to a vehicle purchased by a customer that can navigate autonomously using AI model 110c and simultaneously improve AI model 110c.

[0079] Egos 140 may be equipped with various technologies that enable them to gather data from their surroundings and (potentially) navigate autonomously. For example, ego 140 may be equipped with an inference chip for running self-driving software.

[0080] Various sensors for each ego 140 may monitor and transmit collected data associated with different navigation sessions to analytics server 110a. FIGS. 1B-1C show block diagrams of sensors incorporated within ego 140, according to one embodiment. The number and location of each sensor discussed with respect to FIGS. 1B-1C may depend on the type of ego discussed in FIG. 1A. For example, robot 140b may include different sensors than vehicle 140a or truck 140c. For example, robot 140b may not include airbag activation sensor 170q. Additionally, the sensors for vehicle 140a and truck 140c may be in different locations than those shown in FIG. 1C.

[0081] As discussed herein, various sensors incorporated within each ego 140 may be configured to measure various data associated with each navigation session. Analytics server 110a may periodically collect the data monitored and collected by these sensors, which is processed according to methods described herein and used to train and / or execute AI model 110c to generate an occupancy map.

[0082] Ego 140 may include user interface 170a. User interface 170a may refer to the user interface of an ego computing device (e.g., ego computing device 141 of FIG. 1A). User interface 170a may be implemented as a display screen integrated with or coupled to a vehicle's interior, a head-up display, a touchscreen, etc. User interface 170a may include input devices such as a touchscreen, knobs, buttons, a keyboard, a mouse, a gesture sensor, a steering wheel, etc. In various embodiments, user interface 170a may be adapted to provide user input (e.g., as types of signals and / or sensor information) to other devices or sensors of ego 140 (e.g., the sensors shown in FIG. 1B), such as controller 170c.

[0083] User interface 170a may also be implemented with one or more logic devices that may be adapted to execute instructions, such as software instructions, that implement any of the various processes and / or methods described herein. For example, user interface 170a may be adapted to form a communication link, send and / or receive communications (e.g., sensor signals, control signals, sensor information, user input, and / or other information), or perform various other processes and / or methods. In another example, a driver may use user interface 170a to control the temperature of ego 140 or activate its features (e.g., autonomous driving or steering system 170o). Accordingly, user interface 170a may monitor and collect driving session data in conjunction with other sensors described herein. User interface 170a may also be configured to display various data generated / predicted by analytics server 110a and / or AI model 110c.

[0084] Orientation sensor 170b may be implemented as one or more of a compass, float, accelerometer, and / or other digital or analog device capable of measuring the orientation of ego 140 (e.g., the magnitude and direction of roll, pitch, and / or yaw relative to one or more reference orientations, such as gravity and / or magnetic north). Orientation sensor 170b may be adapted to provide orientation measurements at ego 140. In other embodiments, orientation sensor 170b may be adapted to provide roll, pitch, and / or yaw rate of ego 140 using a time series of orientation measurements. Orientation sensor 170b may be positioned and / or adapted to provide orientation measurements relative to a particular coordinate frame of ego 140.

[0085] Controller 170c may be implemented as any suitable logic device (e.g., a processing device, microcontroller, processor, application specific integrated circuit (ASIC), field programmable gate array (FPGA), memory storage device, memory reader, or other device or combination of devices) that can be adapted to execute, store, and / or receive appropriate instructions, such as software instructions that implement control loops for controlling various operations of ego 140. Such software instructions may also process sensor signals, determine sensor information, provide user feedback (e.g., via user interface 170a), interrogate devices regarding operating parameters, select operating parameters for devices, or implement methods for performing any of the various operations described herein.

[0086] Communications module 170e may be implemented as any wired and / or wireless interface configured to communicate sensor data, configuration data, parameters, and / or other data and / or signals to any feature shown in FIG. 1A (e.g., analytics server 110a). As described herein, in some embodiments, communications module 170e may be implemented in a distributed manner, such that portions of communications module 170e are implemented within one or more elements and sensors shown in FIG. 1B. In some embodiments, communications module 170e may delay communication of sensor data. For example, when ego 140 does not have network connectivity, communications module 170e may store sensor data in temporary data storage and transmit the sensor data when ego 140 is identified as having adequate network connectivity.

[0087] Speed ​​sensor 170d may be implemented as an electronic pitot tube, a metering gear or wheel, a water speed sensor, a wind speed sensor, a wind speed sensor (e.g., direction and magnitude), and / or other device capable of measuring or determining the linear velocity of ego 140 (e.g., within the surrounding medium and / or aligned with the longitudinal axis of ego 140) and providing such measurement as a sensor signal that can be communicated to various devices.

[0088] Gyroscope / accelerometer 170f may be implemented as an electronic sextant, a semiconductor device, an integrated chip, an accelerometer sensor, or other system or device capable of measuring angular velocity / acceleration and / or linear acceleration (e.g., direction and magnitude) of ego 140 and providing such measurements as sensor signals that can be communicated to other devices, such as analytics server 110a. Gyroscope / accelerometer 170f may be positioned and / or adapted to make such measurements relative to a particular coordinate frame of ego 140. In various embodiments, gyroscope / accelerometer 170f may be mounted in a common housing and / or module with other elements shown in FIG. 1B to ensure a common frame of reference or known transformations between frames of reference.

[0089] Global Navigation Satellite System (GNSS) 170h may be implemented as a global positioning satellite receiver and / or another device capable of determining the absolute and / or relative position of ego 140 based on, for example, radio signals received from space-born and / or terrestrial sources and providing such measurements as sensor signals that can be communicated to various devices. In some embodiments, GNSS 170h may be adapted to determine the velocity, speed, and / or yaw rate of ego 140 (e.g., using a time series of position measurements), such as the absolute velocity and / or yaw component of the angular velocity of ego 140.

[0090] Temperature sensor 170i may be implemented as a thermistor, an electrical sensor, an electrical thermometer, and / or other device capable of measuring a temperature associated with ego 140 and providing such measurement as a sensor signal. Temperature sensor 170i may be configured to measure an environmental temperature associated with ego 140, such as a cockpit or dash temperature, which may be used to estimate the temperature of one or more elements of ego 140.

[0091] Humidity sensor 170j may be implemented as a relative humidity sensor, an electrical sensor, an electrical relative humidity sensor, and / or another device capable of measuring the relative humidity associated with ego 140 and providing such measurement as a sensor signal.

[0092] Steering sensor 170g may be adapted to physically adjust the orientation of ego 140 according to one or more control signals provided by a logic device, such as controller 170c, and / or user input. Steering sensor 170g may include one or more actuators and control surfaces of ego 140 (e.g., rudders or other types of steering or trim mechanisms) and may be adapted to physically adjust the control surfaces to various positive and / or negative steering angles / positions. Steering sensor 170g may also be adapted to sense the current steering angles / positions of such steering mechanisms and provide such measurements.

[0093] Propulsion system 170k may be implemented as a propeller, turbine, or other thrust-based propulsion system, a mechanical wheeled and / or tracked propulsion system, a wind / sail-based propulsion system, and / or other type of propulsion system that may be used to provide motive power to ego 140. Propulsion system 170k may also monitor the direction of motive power and / or thrust of ego 140 relative to a coordinate frame of reference of ego 140. In some embodiments, propulsion system 170k may be coupled to and / or integrated with steering sensor 170g.

[0094] Occupant restraint sensor 170l may monitor the seat belt detection and lock / unlock assembly, as well as other occupant restraint subsystems. Occupant restraint sensor 170l may include various environmental and / or status sensors, actuators, and / or other devices that facilitate operation of safety mechanisms associated with operation of ego 140. For example, occupant restraint sensor 170l may be configured to receive movement and / or status data from other sensors shown in FIG. 1B. Occupant restraint sensor 170l may determine whether a safety mechanism (e.g., a seat belt) is engaged.

[0095] Camera 170m may refer to one or more cameras integrated into ego 140, as depicted in FIG. 1C, or may include multiple cameras integrated into (or retrofitted to) ego 140. Camera 170m may be an inward-facing or outward-facing camera of ego 140. For example, as depicted in FIG. 1C, ego 140 may include one or more inward-facing cameras that can monitor and collect video of occupants of ego 140. Ego 140 may include eight outward-facing cameras. For example, ego 140 may include front camera 170m-1, forward-looking side camera 170m-2, forward-looking side camera 170m-3, rearward-looking side camera 170m-4 on each front fender, camera 170m-5 on each side (e.g., integrated into the B-pillar), and rear camera 170m-6.

[0096] 1B, radar 170n and ultrasonic sensor 170p may be configured to monitor the distance of ego 140 to other objects, such as other vehicles or immovable objects (e.g., trees or garage doors). Ego 140 may also include an automatic driving or steering system 170o configured to autonomously navigate ego 140 using data collected via various sensors (e.g., radar 170n, speed sensor 170d, and / or ultrasonic sensor 170p).

[0097] Thus, automated driving or steering system 170o may analyze various data collected by one or more sensors described herein to identify driving data. For example, automated driving or steering system 170o may calculate the risk of a forward collision based on ego 140's speed and its distance to another vehicle on the road. Autonomous driving or steering system 170o may also determine whether the driver is touching the steering wheel. Autonomous driving or steering system 170o may transmit the analyzed data to various features discussed herein, such as an analytics server.

[0098] Airbag deployment sensor 170q may predict or detect a crash and cause one or more airbags to deploy or inflate. Airbag deployment sensor 170q may transmit data regarding the deployment of the airbags, including data associated with the event that caused the deployment.

[0099] 1A , administrator computing device 120 may represent a computing device operated by a system administrator. Administrator computing device 120 may be configured to display data retrieved or generated by analytics server 110a (e.g., various analytics metrics and risk scores), allowing the system administrator to monitor various models utilized by analytics server 110a, review feedback, and / or facilitate training of AI models 110c maintained by analytics server 110a.

[0100] Ego 140 may be any device configured to navigate various routes, such as vehicle 140a or robot 140b. As discussed with respect to FIGS. 1B-1C, ego 140 may include various telemetry sensors. Ego 140 may also include ego computing device 141. Specifically, each ego may have its own ego computing device 141. For example, truck 140c may have ego computing device 141c. For simplicity, ego computing devices are collectively referred to as ego computing device 141. Ego computing device 141 may control the presentation of content on ego 140's infotainment system, process commands related to the infotainment system, aggregate sensor data, manage communication of data to electronic data sources, receive updates, and / or send messages. In one configuration, ego computing device 141 communicates with an electronic control unit. In another configuration, ego computing device 141 is an electronic control unit. Ego computing device 141 may include a processor and non-transitory machine-readable storage media capable of performing the various tasks and processes described herein. For example, AI model 110c described herein may be stored and executed (or directly accessed) by ego computing device 141. Non-limiting examples of ego computing device 141 may include a vehicle multimedia and / or display system.

[0101] In one example of how AI model 110c can be trained, analytics server 110a may collect data from ego 140 to train AI model 110c. Before running AI model 110c to generate / predict occupancy data sets, analytics server 110a may train AI model 110c using various methods. Through training, AI model 110c can capture data from one or more cameras of one or more egos 140 (without having to receive radar data) and predict occupancy data in the ego's surroundings. The operations described in this example may be performed by any number of computing devices (e.g., processors of ego 140) operating in the distributed computing system described in FIGS. 1A-1D .

[0102] The analysis server 110a may use the sensors of the ego 140 to generate a first dataset having a first set of data points, where each data point in the first set of data points corresponds to a location and a sensor attribute of at least one voxel in a space surrounding the ego 140, and the sensor attribute indicates whether the at least one voxel is occupied by an object having mass.

[0103] To train the AI ​​model 110c, the analytics server 110a may first use one or more egos 140 to drive a particular route. While driving, the egos 140 may generate navigation session data using one or more of their sensors (including one or more cameras). For example, one or more egos 140 equipped with various sensors may navigate a specified route. As the one or more egos 140 traverse the terrain, their sensors may capture continuous (or periodic) data about their surroundings. The sensors may indicate occupancy around the one or more egos 140. For example, the sensor data may indicate various objects having mass around the one or more egos 140 as they navigate their route.

[0104] The analysis server 110a may generate a first dataset using sensor data received from one or more of the egos 140. The first dataset may indicate the occupancy of different voxels within the perimeter of one or more of the egos 140. As used herein, a voxel is a three-dimensional pixel that forms the building block of the perimeter of one or more of the egos 140. Within the first dataset, each voxel may encapsulate sensor data indicating whether a mass has been identified for that particular voxel. As used herein, mass may refer to or correspond to any object identified using a sensor. For example, in some embodiments, the ego 140 may be equipped with a LiDAR that identifies mass by emitting laser pulses and measuring the time it takes for these pulses to travel to an object (having mass) and return. The LiDAR sensor system may operate based on the principle of measuring the distance between the LiDAR sensor and objects within its field of view. This information, combined with other sensor data, may be analyzed to identify and characterize different masses or objects within the perimeter of one or more of the egos 140.

[0105] Various additional data may be used to indicate whether voxels surrounding one or more egos 140 are occupied by objects having mass. For example, in some embodiments, a digital map of the surroundings of one or more egos 140 (e.g., a digital map of the route being traveled by the ego) may be used to determine the occupancy status of each voxel.

[0106] In operation, as one or more egos 140 navigate, their sensors collect data and transmit the data to analytics server 110a, as depicted in data stream 176. For example, computing device 141 of ego 140 may transmit sensor data to analytics server 110a using data stream 176.

[0107] Analysis server 110a may use ego 140's camera to generate a second dataset having a second set of data points, where each data point in the second set of data points corresponds to the location and image attributes of at least one voxel in the space surrounding ego 140.

[0108] The analysis server 110a may receive camera feeds from one or more egos 140 navigating the same route as the first step. In some embodiments, the analysis server 110a may perform the first step and the second step simultaneously (or contemporaneously). Alternatively, two (or more) different egos 140 may navigate the same route, with one ego transmitting its sensor data and the second ego 140 transmitting its camera feed.

[0109] In some embodiments, one or more egos 140 may include one or more high-resolution cameras that capture a continuous stream of visual data from around the one or more egos 140 as the one or more egos 140 navigate through a route. Analytics server 110a may then use the camera feeds to generate a second dataset, with visual elements / representations of different voxels around the one or more egos 140 included in the second dataset.

[0110] In operation, as one or more egos 140 navigate, their cameras collect data and transmit the data to analytics server 110a, as depicted in data stream 172. For example, ego computing device 141 may use data stream 172 to transmit image data to analytics server 110a.

[0111] The analytics server 110a may train an AI model using the first and second data sets, whereby the AI ​​model 110c trains itself by correlating each data point in the first set of data points with a corresponding data point in the second set of data points using the respective positions of each data point, where once trained, the AI ​​model 110c is configured to receive a camera feed from the new ego 140 and predict the occupancy state of at least one voxel in the camera feed.

[0112] Using the first and second data sets, analytics server 110a may train AI model 110c such that AI model 110c can correlate different visual attributes of a voxel (in the camera feed in the second data set) to the occupancy state of that voxel (in the first data set). Once trained in this manner, AI model 110c may receive a camera feed (e.g., from new ego 140) without receiving sensor data and then determine the occupancy state of each voxel for new ego 140.

[0113] The analysis server 110a may generate a training dataset including a first and a second dataset. The analysis server 110a may use the first dataset as ground truth. For example, the first dataset may indicate different locations of voxels and their occupancy states. The second dataset may include a visual (e.g., camera feed) illustration of the same voxels. Using the first dataset, the analysis server 110a may label the data such that data records associated with each voxel corresponding to an object are indicated as having a positive occupancy state.

[0114] The labeling of the occupancy states of different voxels may be performed automatically and / or manually. For example, in some embodiments, the analysis server 110a may label the data using a human reviewer. For example, as discussed herein, camera feeds from one or more cameras on a vehicle may be presented to a human reviewer on an electronic platform for labeling. Additionally or alternatively, the entire data may be ingested by the AI ​​model 110c, which identifies corresponding voxels, analyzes the first digital map, and correlates the image of each voxel with its respective occupancy state.

[0115] Using the ground truth, the AI ​​model 110c may be trained so that the visual elements of each voxel are analyzed and correlated with whether the voxel is occupied by mass. Thus, the AI ​​model 110c may look up the occupancy state of each voxel (using the first data set) and use that information as ground truth. The AI ​​model 110c may also look up the visual attributes of the same voxels using the second data set.

[0116] In some embodiments, analytics server 110a may use a supervised training method. For example, using ground truth and received visual data, AI model 110c may train itself to be able to predict occupancy at a voxel using only images of that voxel. As a result, once trained, AI model 110c may receive camera feeds, analyze the camera feeds, and determine the occupancy at each voxel in the camera feed (without having to use radar).

[0117] The analytics server 110a may provide a series of training data sets to the AI ​​model 110c and obtain a set of predicted outputs (e.g., predicted occupancy states). The analytics server 110a may then train the AI ​​model 110c by comparing the predicted data with ground truth data to determine differences and adjusting the internal weights and parameters of the AI ​​model 110c proportional to the determined differences according to a loss function. The analytics server 110a may train the AI ​​model 110c in a similar manner until the predictions of the trained AI model 110c closely meet a particular threshold (e.g., recall or precision).

[0118] Additionally or alternatively, the analytics server 110a may use unsupervised methods in which the training dataset is unlabeled. Because labeling the data in the training dataset can be time-consuming and require excessive computational power, the analytics server 110a may utilize unsupervised training techniques to train the AI ​​model 110c.

[0119] After AI model 110c is trained, the AI ​​model may be used by ego 140 to predict occupancy data around one or more egos 140. For example, AI model 110c may divide the ego's surroundings into different voxels and predict occupancy for each voxel. In some embodiments, AI model 110c (or analytics server 110a using data predicted using AI model 110c) may generate an occupancy map or occupancy network representing the surroundings of one or more egos 140 at any given time.

[0120] In another example of how AI model 110c may be used, after training AI model 110c, analytics server 110a (or a local chip of ego 140) may collect data from the egos (e.g., one or more of the egos 140) to predict occupancy data sets for one or more egos 140. This example describes how AI model 110c can be used to predict occupancy data in real time or near real time for one or more egos 140. This configuration may have a processor, such as analytics server 110a, execute the AI ​​model. However, one or more actions may be executed locally, for example, via a chip located within one or more egos 140. In operation, AI model 110c may be executed locally via ego 140 such that it can use the results to navigate itself autonomously.

[0121] The processor may input image data of the space surrounding ego object 140 to AI model 110c using cameras on ego object 140. The processor may collect and / or analyze data received from various cameras (e.g., outward-facing cameras) on one or more of ego 140. In another example, the processor may collect and aggregate footage recorded by one or more cameras on ego 140. The processor may then send the footage to AI model 110c trained using the methods discussed herein.

[0122] The processor may predict occupancy attributes of multiple voxels by executing AI model 110c, which may use methods discussed herein to predict occupancy states in different voxels surrounding one or more egos 140 using the received image data.

[0123] The processor may generate a dataset based on the plurality of voxels and their corresponding occupancy attributes. The analysis server 110a may generate a dataset including the occupancy states of different voxels according to their coordinate values. The dataset may be a queryable dataset that can be used to send predicted occupancy states to different software modules.

[0124] In operation, one or more ego 140 may collect image data from their cameras and transmit the image data to a processor (located locally on one or more ego 140) and / or analytics server 110a, as depicted in data stream 172. The processor may then execute AI model 110c to predict occupancy data at one or more ego 140. If the prediction is performed by analytics server 110a, the occupancy data may be transmitted to one or more ego 140 using data stream 174. If a processor is located locally within one or more ego 140, the occupancy data is transmitted to ego computing device 141 (not shown in FIG. 1A ).

[0125] Using the methods discussed herein, training of AI model 110c can be performed such that execution of AI model 110c (at inference time) can be performed locally on any ego 140. Collected data (e.g., navigation data collected during navigation of ego 140, such as journey image data) can then be fed back to AI model 110c so that the additional data can improve AI model 110c.

[0126] 1D illustrates certain hardware and software components of ego 140 for performing fully or partially autonomous driving (SD) operations, according to one embodiment. Ego 140 includes SD circuitry 150 and ego computing device 141, which may include the same or different components of SD circuitry 150. SD circuitry 150 includes SD chips 152a-152b (commonly referred to as SD chips 152), such as system-on-chip (SoC) integrated circuit chips. Each SD chip 152 includes non-transitory machine-readable memory, such as DRAM 190a-190b (commonly referred to as DRAM 190) and SRAM. SD chip 152 further includes various types of processing units, including GPU 191, CPUs 193a-193c (commonly referred to as CPUs 193), and specially designed AI accelerator devices 192a-192b (commonly referred to as AI accelerator devices 192). SD chip 152 includes chip-to-chip interfaces 194a-194b (commonly referred to as chip interfaces 194), such as Peripheral Component Interconnect (PCI) or PCI-Express (PCIe). SD chip 152 communicates signals over inter-chip bus 199 according to the protocols and programming of chip interface 194.

[0127] As described with respect to FIG. 1A , analysis server 110 (or other computing device) may compile and download compiled executable binaries of software for neural network architectures to ego 140 and / or ego computing device 141. Ego computing device 141 may generate and / or execute various software programming operations and executable binaries for managing the operation of SD circuit 150 (or other hardware), which may include executable instructions for applying the neural network architecture to types of sensor data from sensors in ego 140. The executable instructions received, generated, and / or executed by ego computing device 141 may include executable instructions for managing the operation of components of SD circuit 150. For example, the compiled executable binaries may include instructions indicating a destination SD chip 152 for transferring data signals between SD chips 152 via bus 199, or indicating an executing SD chip 152 to perform functions of a particular neural network architecture, for example. For example, instructions generated and compiled for a neural network architecture for recognizing traffic sign objects using data from camera 170m may be loaded into and executed by components of first SD chip 152a, while instructions compiled for a neural network architecture for path planning may be loaded and executed by components of second SD chip 152b. If the path planning neural network architecture is trained to use data outputs generated by the traffic sign neural network architecture, the instructions in first SD chip 152a instruct it to transfer output data signals to second SD chip 152b via bus 199, and the instructions for second SD chip 152b instruct it to use such data signals received from first SD chip 152a.

[0128] In an exemplary embodiment, SD circuit 150 includes two SD chips 152a-152b. Often, SD chips 152 function in a redundant or failover mode of operation, with first SD chip 152a acting as a primary chip and second SD chip 152b acting as a secondary chip. For example, first SD chip 152a is prioritized to execute the majority of executable instructions, and second SD chip 152b is called upon to operate as a failover or redundancy in the event of a problem with first SD chip 152a.

[0129] SD circuitry 150 may operate in an enhanced computing mode that balances the execution instruction pipeline among SD chips 152. As an example, ego computing device 141 executes software routines to compile execution instructions to be executed by processing units 191-193 of SD chip 152 and distribute the execution instructions to the optimal hardware components of SD circuitry 150.

[0130] SD chip 152 includes inter-chip memory sequencers 195a-195b (commonly referred to as inter-chip memory sequencer 195). Inter-chip memory sequencer 195 includes a hardware IC device used to coordinate communication of signals between systems-on-chip (SoCs), such as SD chip 152. In some implementations, inter-chip memory sequencer 195 may include a non-transitory storage location that provides a shared memory space accessible by SD chip 152. In some implementations, inter-chip memory sequencer 195 performs operations to coordinate data signal transfers between SD chips 152, for example, by generating various control signals. Inter-chip memory sequencer 195 may implement one or more inter-chip communication protocols, such as PCIe, SPI, or I2C, among others.

[0131] The hardware and software components of the runtime system of ego 140 (e.g., ego computing device 141, SD circuit board 150, controller 180) receive a compiled program schedule (e.g., execution instructions 218a-218h in execution schedule 217 of FIGS. 2A-2C ), which may be in the form of executable binaries 216, and execute the instructions across the various chips 152 of circuit 150 on the various types of heterogeneous cores (e.g., processing units 190-193 of each chip 152). The runtime system (represented as being executed by controller 180) includes software components for an inter-chip computation scheduler, a heterogeneous hardware scheduler (e.g., CPU accelerator, GPU accelerator, AI accelerator), an inter-chip memory sequencer 195 for scheduling and managing inter-chip signals via chip interface 194 and bus 199, and a clock synchronizer (e.g., OS kernel). In some implementations, the runtime system programming supports model parallelism across multiple SD chips 152. In this manner, the ego 140 may review and go deeper into the clock synchronizer.

[0132] In some embodiments, ego 140 includes controller 180, which performs various operations for managing SD circuit 150. Controller 180 may perform various functions, for example, according to instructions from ego computing device 141 (or other components of ego 140) or configuration input from an administrative user. For example, controller 180 toggles, configures, or directs SD circuit 150 to operate in various operational modes. In some situations, for example, controller 180 directs SD circuit 150 to operate in an enhanced computing mode, in which first SD chip 152a executes a first instruction segment of executable instructions and second SD chip 152b executes a second instruction segment. As another example, in some situations, controller 180 directs SD circuit 150 to operate in a failover mode, in which second SD chip 152b executes executable instructions when first SD chip 152a fails.

[0133] The SD chip 152 includes one or more DRAMs 190 or other types of non-transitory memory for storing data input for the SD chip 152. The data input may be stored in the DRAM 190 for reference by the processing unit for various calculations. In some configurations, the AI ​​accelerator device 192 includes SRAM such that the SD chip 152 moves data from the DRAM 190 to the SRAM of the AI ​​accelerator device 192 for storage. The AI ​​accelerator device 192 performs calculations according to the execution instructions and returns the data to the DRAM 190 or other destination in the SD circuitry 150.

[0134] SD chip 152 includes various types of processing units, which may include any hardware integrated circuit (IC) processor device capable of performing the various processes and tasks described herein. Non-limiting examples of types of processing units include GPU 191, CPU 193, AI accelerator device 192, microcontroller, ALU, ASIC, and FPGA, among others. The processing units may perform the computational functions of a programming layer that defines a neural network architecture or sub-architecture. A compiler outputs executable instructions that represent the operations of the neural network architecture to be executed by ego computing device 141 (or other components of ego 140).

[0135] The AI ​​accelerator device 192 is a hardware accelerator designed specifically for neural network operations, beneficially focusing on improvements to optimize power and performance (e.g., low latency). The AI ​​accelerator device 192 includes hardware IC devices (e.g., microcontrollers, ALUs, ASICs, FPGAs, processor devices) designed for high-speed operation when processing neural network architectures. For example, as transformer neural network architectures (e.g., GPTs) and other types of neural network modeling techniques become more prevalent, other types of processing units (e.g., CPUs 193, GPUs 191) may become slower due to design considerations intended for broader implementation use cases. For example, neural network architectures, sub-neural networks (e.g., the moving object network 206b in FIG. 2B), or child neural networks perform computer vision or object recognition by implementing various GPTs (or other types of transformers) on image sensor data, beneficially replacing previous techniques for post-processing of vision neural networks. AI accelerator device 192 is specifically designed for neural network operations, allowing GPT transformers to operate natively within the computing components of ego 140, such that AI accelerator device 192 provides faster and more efficient processing than conventional GPU 191 or CPU 193 performing similar GPT transforms. In this manner, AI accelerator device 192 reduces or eliminates latency, improving overall efficiency and contributing to ego 140's ability to make real-time decisions. Furthermore, the structural design and design rationale of AI accelerator device 192 consumes relatively less power than conventional GPU 191 or CPU 193 when performing the more sophisticated and complex functions of neural network architectures, such as transformer networks (e.g., transformers).

[0136] In some embodiments, a transformer (e.g., GPT) may be adapted for execution on the ego 140, thereby improving the overall performance of the computing components for an autonomous or semi-autonomous ego 140. For example, typical transformers are often resource-intensive, consume a lot of power, and / or cause significant latency in processing the output, thus hindering the overall performance of the ego 140. Therefore, transformers are often powerful neural network architectures that are not deployed on the ego 140. To address this issue, the transformers for the ego 140 described herein may be deployed without the attention module softmax commonly found in traditional transformers. Embodiments described herein may include transformers with attention without softmax and may perform ReLU operations. Replacing softmax with ReLU in such transformers makes it feasible to deploy the transformer architecture on an autonomous or semi-autonomous ego 140.

[0137] In some embodiments, the SD circuitry includes a controller or other processing unit that maintains clock synchronization between the SOC chips based on interpreting the timestamps of the sensor inputs (e.g., the timestamps of the camera inputs) and converting the timestamps between the SOC chips according to the difference between the current chip clocks of each SOC chip.

[0138] 1E illustrates certain hardware and software components of ego 140 for maintaining clock synchronization between SD chips 152, according to one embodiment. In such an embodiment, SD circuitry 150 includes two (or more) SD chips 152 and two (or more) microcontrollers 155a-155b (commonly referred to as microcontrollers 155) coupled to corresponding microcontrollers 155. For example, a first SD chip 152a is coupled to the first microcontroller 155a, and a second SD chip 152b is coupled to the second microcontroller 155b. Each SD chip 152 includes and executes an OS kernel 153 that manages the operation of the SD chip 152, including executing various executable instructions of the neural network architecture and managing the operation of the SD chip 152.

[0139] OS kernel 153 may include any type of OS capable of performing the various processes and tasks described herein, including managing the execution of executable instructions and maintaining a software-based OS clock. Examples of OSes for OS kernel 153 include Linux, Unix, etc.

[0140] Microcontroller 155 comprises any type of processing circuitry capable of performing the various processes and tasks described herein. Non-limiting examples include a microcontroller, a controller, an ALU, an FPGA, an ASIC, etc. SOC chip 152 comprises one or more processing units running an OS kernel 153 (e.g., Linux®) and one or more smaller microcontrollers 155. Microcontroller 155 includes low-level programming for performing various low-level functions. For example, the functionality of microcontroller 155 includes a boot loader function whereby microcontroller 155 boots various components of SD chip 152, including booting the processing unit running OS kernel 153. In some embodiments, microcontroller 155 or other devices in SD circuitry 150 may include or be coupled to a clock oscillator or counter that oscillates or increments a monotonic clock at a given frequency.

[0141] The microcontroller 155 may communicate with the processing unit running the OS kernel 153 via a given interface (e.g., mailbox, Ethernet, UART, PCIe) to exchange data signals such as time messages or correction instructions. Similarly, the first microcontroller 155a may communicate with the second microcontroller 155b via another interface (e.g., mailbox, Ethernet, UART, PCIe) to exchange time messages or correction instructions.

[0142] Generally, microcontroller 155 executes a boot loader function to boot microcontroller 155 and OS kernel 153 simultaneously, at a relatively early point after booting ego 140. At boot time, microcontroller 155 and OS kernel 153 communicate various timing messages to synchronize with each other. In this way, the boot and synchronization functions of microcontroller 155 logically form a common monotonic time clock for SD circuit 150 before the processor unit of OS kernel 153 has a chance to begin execution.

[0143] When the OS kernel 153 (e.g., Linux®) programming begins booting on the processor, early in the boot of the OS kernel 153, the OS kernel 153 resets its monotonic kernel clock. The OS kernel 153 may reset the kernel clock to match the controller clock of the corresponding microcontroller 155. This reset may occur before any or significant operations have had a chance to start on the SD chip 152, or before any or significant operations have been started in the OS kernel 153. When the OS kernel 153 boots, the OS kernel 153 synchronizes the kernel clock to the corresponding microcontroller 155.

[0144] In some implementations, at a preset resynchronization period or threshold time, the SD chip 152 and the microcontroller 155 actively maintain synchronized kernel and controller clocks through a control loop operation in which the SD chip 152 and the microcontroller 155 exchange timing messages. For example, at a synchronization interval (e.g., once per second), the OS kernel 153 measures the time error with respect to the corresponding microcontroller 155, and in some circumstances, the OS kernel 153 instructs the connected microcontroller 155 to initiate a small adjustment to correct the controller clock.

[0145] In some embodiments, clock synchronization operations include a redundancy fallback feature. In certain situations, an SD chip 152 (e.g., the second SD chip 152b) may suffer a fatal error and must be restarted, while the other SD chip 152 (e.g., the first SD chip 152a) may continue to operate until the restarted SD chip 152 (e.g., the second SD chip 152b) recovers. In such situations, the operational SD chip 152 (e.g., the first SD chip 152a) continues to maintain the kernel clock of the OS kernel 153 (e.g., the first OS kernel 153a) and the controller clock of the corresponding microcontroller 155 (e.g., the first microcontroller 155a), and thus, by extension, maintains the overall logical synchronization clock of the SD circuit 150. Thus, once the restarted SD chip 152b recovers, overall synchronization may continue, for example, via synchronization messages between the first microcontroller 155a and the second microcontroller 155b. The restarted SD chip 152b does not need to restart a new kernel clock and a new controller clock at zero or some other initialized time point. The microcontroller 155b and the OS kernel 153b may start the kernel clock and the controller clock at the current monotonic time of the global synchronized clock in the SD circuitry 150. The microcontroller 155b may perform a recovery process, a restart process, or a startup process that includes a synchronization process with the operational microcontroller 155a, in which case the microcontrollers 155a may exchange time messages to indicate the current time of the operational microcontroller 155a's controller clock, which reflects the global synchronized clock of the SD circuitry 150. The recovered microcontroller 155b and the recovered OS kernel 153b may then exchange time messages indicating the current time of the microcontroller 155a's controller clock, which reflects the global synchronized clock of the SD circuitry 150. In this way, components of the SD circuitry 150 do not need to calculate, distribute, or convert the time difference between the discrete clocks of the SD chip 152.After reboot, application software running in the recovered SD chip 152b and recovered OS kernel 153b can begin executing and participating almost immediately, with limited delay to reconstruct pre-failure state and / or without ongoing latency due to continuous calculations to convert what becomes a discontinuous clock. This beneficially improves fault tolerance and supports failover redundancy for the ego 140.

[0146] The OS kernel 153 and / or microcontroller 155 may perform an error correction function that adjusts the frequency of the controller clock by a relatively small amount, such that the microcontroller 155 or OS kernel 153 increases or decreases the frequency and clock time (e.g., controller clock, kernel clock) by a certain amount.

[0147] As previously mentioned, each SD chip 152 has several types of synchronization operations, including kernel-controller synchronization operations between the OS kernel 153 and the corresponding microcontroller 155, and SOC synchronization or inter-controller synchronization operations between the microcontrollers 155 of the SD chip 152.

[0148] In a first type of synchronization operation (e.g., synchronizing OS kernel 153a with a corresponding microcontroller 155a), OS kernel 153 sends timing messages to microcontroller 155. OS kernel 153 and microcontroller 155 of SD chip 152 each include a communication interface (e.g., mailbox, Ethernet, UART, PCI) for exchanging timing messages (or other types of messages) over signal connections, wires, or buses according to the protocol of the particular interface.

[0149] At startup and / or at preset intervals, the OS kernel 153 sends timing messages to the microcontroller 155 and receives return timing messages from the microcontroller 155, referencing the associated clock times to determine whether one or more clocks have drifted beyond a threshold distance. The OS kernel 153 sends an initial timing message to the microcontroller 155 at a first time (T1). The OS kernel 153 references the kernel clock to retrieve the current time and assigns the current time as the initial message time (T1) in the initial timing message. The microcontroller 155 receives the initial timing message and references the current time of the microcontroller 155's controller clock. The microcontroller 155 sends a response timing message to the OS kernel 153 at a response time (T2). The OS kernel 153 assigns a response time to the response timing message according to the current time of the controller clock. The OS kernel 153 receives the response timing message indicating the response time (T2) and references the kernel clock to retrieve the current time of the kernel clock. When the OS kernel 153 receives the response message, it assigns the current time of the kernel clock as the completion time (T3) to the response timing message. The OS kernel 153 calculates an average time based on an average of kernel times including the initial message time (T1) and the completion time (T3). The OS kernel 153 then compares this average time with the response time (T2) received from the microcontroller 155, and calculates and outputs the difference between the average time and the response time. In an exemplary configuration, the first OS kernel 153a may calculate an offset representing an estimate of the time error between the first monotonic kernel clock of the first OS kernel 153a and the first monotonic controller clock of the first microcontroller 155a. In this exemplary configuration, the offset is calculated as the difference between the average time and the response time.

[0150] In a second type of synchronization operation (e.g., synchronizing a first microcontroller 155a with a second microcontroller 155b), the OS kernel 153 sends timing messages to the microcontrollers 155. Each microcontroller 155 includes a communication interface (e.g., mailbox, Ethernet, UART, PCI) for exchanging timing messages (or other types of messages) with other microcontrollers 155 via signal connections, wires, or buses according to the protocol of the particular interface.

[0151] At startup time and / or at preset intervals, the microcontrollers 155 automatically begin exchanging timing messages with each other. The microcontrollers 155 may send timing messages without having to establish a handshake or exchange any prior or predicate communications. Each microcontroller 155 uses the timing messages to capture and determine the message time (T1) and completion time (T3) and receives the response time (T2) returned from the other microcontrollers 155. Each microcontroller 155 may then calculate the offset as described above with respect to the OS kernel 153. Thus, each microcontroller 155 can estimate the offset relative to its peer microcontrollers 155, and the microcontrollers do this automatically.

[0152] In some situations, a particular microcontroller 155 is restarted and recovered. In such situations, the restart or recovery function of the recovered microcontroller 155b may function as a slave to the operational microcontroller 155a, which acts as the master. The first microcontroller 155a and the second microcontroller 155b treat the controller clock of the first microcontroller 155a as the master controller clock. Upon restart and recovery, the first microcontroller 155a and the second microcontroller 155b exchange timing messages that directly map the controller clock of the first microcontroller 155a to the controller clock of the second microcontroller 155b.

[0153] In some implementations, one or more offsets representing the difference between the two clocks are sent as error signals, for example, to a proportional-integral (PI) controller or a phase-locked loop (PLL) controller. Using known algorithmic techniques, the PLL or PI controller may calculate an error rate based on the offsets, where the input is the offset, considered as the phase, and the output is the rate. The error rate is the rate that the OS kernel 153 (or microcontroller 155) must correct for. At every instance or interval (e.g., every second), a new measurement of the rate, the rate difference (offset) between the two clocks, is calculated, and the OS kernel 153 (or microcontroller 155) may adjust the kernel clock (or controller clock) by applying that rate to the kernel clock (or controller clock). As an example, if the calculated rate indicates that the kernel clock of the first OS kernel 153a is 2 ppm faster than the first microcontroller 155a, the first OS kernel 153a adjusts to a new frequency that matches the 2 ppm slower frequency in the first OS kernel 153a to satisfy the frequency oscillations of the first microcontroller 155a.

[0154] The first microcontroller 155a sends a timing message to the second microcontroller 155b, receives a return timing message from the second microcontroller 155b, and references the associated time to determine whether one or more clocks have drifted beyond a threshold distance. The first microcontroller 155a sends an initial timing message to the second microcontroller 155b at a first time (T1). The OS kernel 153 references the kernel clock to retrieve the current time and assigns the current time as the initial message time (T1) in the initial timing message. The microcontroller 155 receives the initial timing message and references the current time of the controller clock of the microcontroller 155. The microcontroller 155 sends a response timing message to the OS kernel 153 at a response time (T2). The OS kernel 153 assigns a response time to the response timing message according to the current time of the controller clock. The OS kernel 153 receives the response timing message indicating the response time (T2) and references the kernel clock to retrieve the current time of the kernel clock. When the OS kernel 153 receives the response message, it assigns the current time of the kernel clock as the completion time (T3) to the response timing message. The OS kernel 153 calculates an average time based on an average of kernel times including the initial message time (T1) and the completion time (T3). The OS kernel 153 then compares this average time with the response time (T2) received from the microcontroller 155, and calculates and outputs the difference between the average time and the response time. In an exemplary configuration, the first OS kernel 153a may calculate an offset representing an estimate of the time error between the first monotonic kernel clock of the first OS kernel 153a and the first monotonic controller clock of the first microcontroller 155a. In this exemplary configuration, the offset is calculated as the difference between the average time and the response time.

[0155] 2A-2B depict data flow between hardware and software computing components of a system 200 for developing and compiling executable instructions 218a-218h (commonly referred to as execution instructions 218) in a development system 201 to be loaded into an ego 202 as executable binaries 216, according to one embodiment. FIG. 2C shows the execution instructions 218 generated and organized into an execution schedule 217 for execution by circuit hardware components of the system 200, according to an embodiment.

[0156] One or more computing devices of development system 201 (e.g., analytics server 110) may execute software programming that defines one or more neural network architectures 204, hardware model training engine 207, compiler 210, and execution scheduler 212, among other types of software programming routines. Additionally, computing devices of development system 201 may execute software programming to train, retrain, and tune neural network architecture 204 or portions of neural network architecture 204 (e.g., parameters, hyperparameters, weights, layers, functions) with various forms of historical and / or current sensor data from any number of egos 202 for prediction accuracy and consistency. Additionally or alternatively, the computing device of development system 201 may execute software programming for training, retraining, and tuning neural network architecture 204 or portions of neural network architecture 204 (e.g., parameters, hyperparameters, weights, layers, functions) to various types of input data, output data, or predicted data of neural network architecture 204 that are relative to, scaled, or optimized for data sizes and formats implemented by hardware components of ego 202, for example.

[0157] During training or inference time, the computing device of development system 201 extracts features or tensors from input data, such as historical or current sensor data retrieved from a database of development system 201 containing historical data collected from sensors of ego 202 or captured by ego 202. The computing device of development system 201 feeds the input data to neural network architecture 204 or sub-architectures for various operations (e.g., computer vision, object recognition), applies neural network architecture 204 to the input data to generate predicted outputs, and adjusts or retrains portions of neural network architecture 204 (e.g., parameters, hyperparameters, weights, layers) during training.

[0158] The computing device of development system 201 applies a graph partitioner to the sensor data to generate data partitions or data portions. Ego computing device 141 applies a set of compilers (not shown), which may logically form a compiler toolchain for ego 202's neural network architecture, to compile and debug code for executing layers of the neural network architecture for sensor data interpretation. Each compiler is used to convert a high-level programming language into machine code containing executable instructions executed by the SD circuit 150's hardware. The compilers may be configured or optimized to compile the programming code according to the specific architecture or type of the SD chip's processing unit (e.g., CPU 193, GPU 191, or dedicated AI accelerator device 192 hardware). A schedule optimizer of the execution scheduler may combine multiple compiled pieces of code (e.g., executable instructions) into one or more executable files or data streams for an execution schedule (not shown).

[0159] The schedule optimizer and execution scheduler obtain a set of execution instructions and map the execution instructions to hardware components of the SD circuit (e.g., GPU 191, AI accelerator device 192, CPU 193) to execute the particular execution instructions. In some implementations, the schedule optimizer of the execution scheduler is trained to optimize operations to be performed in the hardware components of the SD circuit. The schedule optimizer is trained to determine or pre-configure the time or latency requirements for the hardware components to perform the operations of the execution instructions. This is often possible because such performance timing or latency metrics are known, static in nature, quickly calculated, or pre-stored. In this way, the schedule optimizer maps the execution instructions to the components of the SD circuit according to the minimized or optimized latency. Additionally or alternatively, the schedule optimizer determines which hardware components of the SD circuit should execute which execution instructions (e.g., which compiler generated the machine code for the execution instructions) based on characteristics of the execution instructions. In this manner, the schedule optimizer maps execution instructions to processing units based on the compiler that generated the particular execution instructions.

[0160] 2A, system 200 includes software programming for executing neural network architecture 204 on a computing device of development system 201, including various domain-specific or task-specific subnetworks 206a-206e (commonly referred to as subnetworks 206), although other types of machine learning architectures may be included. The source code of the software programming defines aspects (e.g., parameters, hyperparameters, weights, layers, functions) of neural network architecture 204, including source code defining any number of subnetworks 206, including traffic sign network 206a, moving object network 206b, lane network 206c, occupancy network 206d, and path planning network 206e, for performing operations for a particular domain or task, although embodiments may include additional or alternative types of subnetworks. The software components of development system 201 may further include compiler 210, hardware model training engine 207, and execution scheduler 212, which includes functionality for defining schedule optimizer 214.

[0161] During training of neural network architecture 204 and subnetwork 206, development system 201 executes hardware model training engine 207 to train subnetwork 206 on model or representation data in ego 202's hardware components. In this manner, hardware model training engine 207 provides quantized awareness training (QAT) for neural network architecture 204 so that subnetwork 206 can be optimized for ego 202's hardware components and quantization resilience. Beneficially, the QAT functionality of hardware model training engine 207 trains neural network architecture 204 and subnetwork 206 more efficiently and in a smaller size. The QAT functionality of hardware model training engine 207 trains subnetwork 206 by applying various or desired quantized weights (e.g., 16-bit floating-point values; 8-bit floating-point values; 8-bit integers) and activations (e.g., 16-bit floating-point values; 8-bit floating-point values; 8-bit integers). The QAT functionality of the hardware model training engine 207 trains the neural network architecture 204 and / or each sub-network 206 to operate with lower precision numbers. For example, the hardware model training engine 207 may train a sub-network 206 to take in or generate 8-bit floating-point or integer values, rather than taking in or generating 16-bit floating-point or integer values.

[0162] Beneficially, the hardware model training engine 207 can improve efficiency and reduce demands on computing resources on the ego 202 by operating with smaller quantized data sizes. This, in turn, can reduce the power consumption required by the ego 202 hardware. For example, by training the neural network architecture 204 to be quantization-aware or resilient, the hardware executing the neural network architecture 204 and the compiled neural network architecture 204 can operate well on lower-precision data values ​​or primitives. In this manner, the ego 202 hardware can execute a neural network architecture 204 trained and compiled with a lower precision using relatively less power than would be required if the ego 202 hardware were to execute a neural network architecture 204 trained and compiled with a higher precision.

[0163] The neural network architecture 204 may include or be connected to a neural network of a graph partitioner 208. The graph partitioner 208 may partition the sensor data received via the ingestion layer 202 into sub-networks 206 to process the sensor data. The neural network architecture 204 is logically partitioned into sub-networks 206. The neural network layers of the graph partitioner 208 parse the sensor data into data portions and then map the data portions to sub-networks 206 that are trained to perform the function of the particular sub-network 206. The graph partitioner 208 maps the sensor data portions to the sub-networks 206 according to, for example, the type of data or features in the sensor data used by the sub-networks 206.

[0164] After assigning the sensor data and functions to the subnetworks 206, the graph partitioner 208 may assign which hardware components of the SD circuit 201 should implement and execute the functions of the subnetworks 206. The neural network layers of the graph partitioner 208 are trained to assign the sensor data portions and functions of particular subnetworks 206 to particular processing units (e.g., CPUs, GPUs, dedicated hardware AI accelerator devices) of the chips 203a-203b (commonly referred to as chips 203) (e.g., SoCs, SD chips) of the SD circuit 201. For example, the graph partitioner 208 is configured and trained to assign relatively simple functions of a subnetwork 206 that use a portion of the sensor data to the CPU of the chip 203, and assign relatively complex functions of another subnetwork 206 that use a different portion of the sensor data to the AI ​​accelerator device of the chip 203.

[0165] 2B , domain-specific subnetworks 206 include a traffic sign network 206 a, a moving object network 206 b, a lane network 206 c, an occupancy network 206 d, and a path planning network 206 e. The subnetworks 206 perform various types of domain-specific or task-related functions for a given purpose (e.g., object recognition, path planning) according to the software programming of the neural network layers of a particular subnetwork 206. As one example, the function of traffic sign network 206 a includes recognizing specific object image data (or other types of sensor data) such as traffic regulations, stop signs, yield signs, speed signs, and topology signs. As another example, the function of occupancy network 206 d includes determining per-voxel occupancy, per-voxel speed, and 3D surface shape semantics, among other image-related metrics, in the image data (or other types of sensor data). As another example, the functions of the route planning network 206e include generating a trajectory or path for navigating the ego and using image data or other types of sensor data to adjust the path for, among other things, collision avoidance.

[0166] The sub-networks 206 perform various operations or functions to compute sensor data and generate outputs for the particular sub-network 206. In some cases, these functions include procedural computations or operations. In some cases, these functions include child neural networks of the particular sub-network 206. Non-limiting examples of types of child networks of a sub-network 206 include an intake layer or incorporation layer (sometimes referred to as a "head layer"), a correction layer, a regularization neural network (RegNet) layer, a transformer layer, and a multi-layer perceptron (MLP) layer, among others.

[0167] In some cases, graph partitioner 208 is further configured and trained to assign portions of input sensor data to specific subnetworks 206 based on the type of child neural network in subnetwork 206. Additionally or alternatively, graph partitioner 208 is trained to assign functions and sensor data to hardware in SD circuit 201 based on the capabilities of the hardware. For example, graph partitioner 208 may be trained to optimize hardware efficiency and / or reduce hardware latency, or achieve any additional or alternative performance behavior of the hardware. As one example, graph partitioner 208 may assign a series of interrelated or dependent functions of child networks in subnetwork 206 to one or more CPUs on the same first chip 203a, thereby improving efficiency. As another example, graph partitioner 208 may assign complex functions of subnetwork 206 to AI accelerator devices on chip 203a, thereby improving computational speed. The graph partitioner 208 may be trained to maximize or minimize a performance metric, or the graph configuration may be trained to balance and optimize according to multiple performance metrics.

[0168] Returning to FIG. 2A , system 200 includes a compiler toolchain comprising a set of compilers 210a-210c (commonly referred to as compilers 210). Compilers 210 include software programming configured to convert high-level programming languages ​​of layers and functions of neural network architecture 204 and sensor data into machine code of executable instructions 218 that can be executed by hardware in SD circuit 201. System 200 includes a heterogeneous collection of processing units and compilers 210, where compilers 210 are configured to convert from a given high-level programming language (e.g., functions of subnetwork 206 and sensor data portions) to a given machine code (e.g., executable instructions 218) compatible with an assigned processing unit. In some embodiments, software routines in compilers 210 may selectively compile machine code for multiple types of processing units such that compiler 210 may generate executable instructions 218 for multiple types of processing units (e.g., CPU, GPU, AI accelerator device).

[0169] Graph partitioner 208 (or other components of system 200) is configured and trained to identify which compiler 210 should be assigned to compile which functions of subnetwork 206. For example, continuing with the previous example, graph partitioner 208 is configured and trained to assign relatively simple functions of subnetwork 206 to a first compiler 210a programmed to generate executable instructions 218 for a CPU, and relatively complex functions of other subnetworks 206 to a second compiler 210b programmed to generate executable instructions 218 for an AI accelerator device. The output of compiler 210 is executable instructions 218 compiled from the sensor data portions and software programming for the functions of subnetwork 206.

[0170] System 200 includes an execution scheduler 212 neural network that includes a layer that defines a schedule optimizer 214 (sometimes referred to as a "linker"). Schedule optimizer 214 of execution scheduler 212 may combine multiple compiled pieces of code (e.g., executable instructions 218) generated by compiler 210 of a compiler toolchain into one or more executable binaries 216 that include one or more executable files or data streams of execution instructions 218. Execution scheduler 212 may arrange or queue execution instructions 218 for ordered execution by hardware of SD circuit 201.

[0171] Executable binary 216 is downloaded from the software of system 200 into non-transitory memory of the hardware of SD circuit 201. In SD circuit 201, a software- or firmware-based controller component of SD circuit 201 parses the executable instructions 218 of executable binary 216 and loads the executable instructions 218 into one or more non-transitory memories (not shown) accessible to assigned processing units (or other hardware components) of chip 203. The processing units then execute the executable instructions 218 to perform the functions of neural network architecture 204.

[0172] Referring to FIG. 2C , in some cases, the execution instructions 218 generated by the compiler 210 may be logically represented, arranged as an execution schedule 217. The output of the compiler toolchain includes execution instructions 218 for the functions of the subnetwork 206. For ease of understanding, the execution instructions 218 of FIG. 2C indicate the chip 203 assigned to perform the operations, the processing unit (e.g., GPU, CPU, AI accelerator device) assigned to perform the operations, and which neural network architecture functions are performed. However, the execution instructions 218 of potential embodiments may include additional or alternative types of information, such as input data sources or interfaces, output data destinations or interfaces, and computational instructions.

[0173] As an example, the first execution instruction 218a indicates that the first GPU (gpu0) of the second chip 203b (SoC1) is assigned to execute a function. The first execution instruction 218a instructs the first GPU of the second chip 203b to execute a function of the corrective neural network architecture in the occupancy network 206d (i.e., run gpu0, soC1).

[0174] Hardware and software downstream of the ego may incorporate outputs generated by the SD circuitry 201 executing the neural network architecture 204, such as trajectory, speed, and other navigation decisions of the path planning network 206e, to operate or manipulate the ego within the environment.

[0175] The various illustrative logic blocks, modules, circuits, and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, computer software, or a combination of both. To clearly illustrate this interchangeability between hardware and software, the various illustrative components, blocks, modules, circuits, and steps have been described generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the particular application and design constraints imposed on the overall system. Those skilled in the art may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present invention.

[0176] Computer software-implemented embodiments may be implemented in software, firmware, middleware, microcode, hardware description languages, or any combination thereof. A code segment or machine-executable instructions may represent a procedure, a function, a subprogram, a program, a routine, a subroutine, a module, a software package, a class, or any combination of instructions, data structures, or program statements. A code segment may be coupled to another code segment or a hardware circuit by passing and / or receiving information, data, arguments, attributes, or memory contents. Information, arguments, attributes, data, etc. may be passed, forwarded, or transmitted via any suitable means including memory sharing, message passing, token passing, network transmission, etc.

[0177] The actual software code or specialized control hardware used to implement these systems and methods is not a limitation of the present invention. Accordingly, the operation and behavior of the systems and methods are described without reference to specific software code, with the understanding that software and control hardware can be designed to implement the systems and methods based on the description herein.

[0178] If implemented in software, the functions may be stored as one or more instructions or code on a non-transitory computer-readable or processor-readable storage medium. The steps of a method or algorithm disclosed herein may be embodied in a processor-executable software module, which may reside on a computer-readable or processor-readable storage medium. Non-transitory computer-readable or processor-readable media includes both computer storage media and tangible storage media that facilitate transfer of a computer program from one place to another. Non-transitory processor-readable storage media may be any available medium that can be accessed by a computer. By way of example, and not limitation, such non-transitory processor-readable media may include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other tangible storage medium that can be used to store desired program code in the form of instructions or data structures and that can be accessed by a computer or processor. As used herein, disk and disc include compact discs (CDs), laser discs, optical discs, digital versatile discs (DVDs), floppy disks, and Blu-ray discs, where disks typically reproduce data magnetically while discs reproduce data optically using lasers. Combinations of the above should also be included within the scope of computer-readable media. Furthermore, the operations of a method or algorithm may reside as one or any combination or set of code and / or instructions on a non-transitory processor-readable medium and / or computer-readable medium, which may be incorporated into a computer program product.

[0179] The previous description of the disclosed embodiments is provided to enable any person skilled in the art to make or use the present invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments without departing from the spirit or scope of the present invention. Thus, the present disclosure is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the following claims and the principles and novel features disclosed herein.

[0180] While various aspects and embodiments have been disclosed, other aspects and embodiments are contemplated. The various disclosed aspects and embodiments are for purposes of illustration and are not intended to be limiting, with the true scope and spirit being indicated by the following claims.

Claims

1. a circuit board including a plurality of system-on-chip (SOC) devices and a plurality of microcontrollers corresponding to the SOC devices; a processor of the SOC device; Equipped with The processor: sending an initial timing message to a first microcontroller at an initial time according to a kernel clock of the processor; receiving a response message from the first microcontroller indicating a response time in a first controller clock of the first microcontroller; In response to receiving the response message from the first microcontroller, determining a completion time according to the kernel clock of the processor; calculating an error rate in the kernel clock representing a difference between the kernel clock of the processor and the controller clock of the first microcontroller based on the initial clock time, the completion time, and the response time; adjusting a frequency of the kernel clock based on the error rate to reduce the error rate between the kernel clock and the first controller clock; The ego is structured in this way.

2. The system of claim 1 , wherein the processor of the SOC device is configured to send the initial timing message in response to receiving a wake-up command from the first microcontroller.

3. The first microcontroller coupled to the processor includes: calculating a second error rate representing a difference between the first controller clock of the first microcontroller and a second controller clock of a second microcontroller based on the initial clock time between the first microcontroller and the second microcontroller, the completion time between the first microcontroller and the second microcontroller, and the response time between the first microcontroller and the second microcontroller; 2. The system of claim 1, further configured to adjust a second frequency of the second controller clock based on the second error rate to reduce the second error rate between the first controller clock and the second controller clock.

4. 4. The system of claim 3, wherein the first microcontroller is configured to determine whether a difference between the first controller clock and the second controller clock of the first microcontroller meets a threshold difference.

5. The system of claim 1 , wherein the first microcontroller is configured to transmit the initial timing message in response to executing a wake-up function of the first microcontroller.

6. 2. The system of claim 1, wherein a second microcontroller is configured to perform a restart function, and a second controller clock of the second microcontroller is updated to match the first controller clock of the first microcontroller indicated in a timing message received at the second microcontroller from the first microcontroller for the restart function.

7. sending, by a processor of a system on chip (SOC), an initial timing message to a first microcontroller at an initial time according to a kernel clock of said processor; receiving, by the processor, a response message from the first microcontroller indicating a response time in a first controller clock of the first microcontroller; determining, by the processor in response to receiving the response message, a completion time according to the kernel clock of the processor; calculating, by the processor, an error rate in the kernel clock representing a difference between the kernel clock of the processor and the first controller clock of the first microcontroller based on the initial clock time, the completion time, and the response time; adjusting, by the processor, a frequency of the kernel clock based on the error rate to reduce the error rate between the kernel clock and the first controller clock; A method comprising:

8. 8. The method of claim 7, further comprising receiving, by the processor, a wake-up command from the first microcontroller, the processor of the SOC device being configured to send the initial timing message in response to receiving the wake-up command.

9. 9. The method of claim 8, wherein the processor and the first microcontroller exchange one or more timing messages at startup according to a boot loader function of the first microcontroller.

10. calculating, by the first microcontroller, a second error rate representing a difference between the first controller clock of the first microcontroller and a second controller clock of a second microcontroller based on the initial clock time between the first microcontroller and the second microcontroller, the completion time between the first microcontroller and the second microcontroller, and the response time between the first microcontroller and the second microcontroller; adjusting, in the second microcontroller, a second frequency of the second controller clock based on the second error rate to reduce the second error rate between the first controller clock and the second controller clock; The method of claim 7 further comprising:

11. a circuit board including a plurality of system-on-chip (SOC) devices and a plurality of microcontrollers corresponding to the SOC devices; a first microcontroller; Equipped with The first microcontroller sending an initial timing message to a second microcontroller at an initial time according to a first controller clock of the first microcontroller; receiving a response message from the second microcontroller indicating a response time in a second controller clock of the second microcontroller; determining a completion time according to the first controller clock of the first microcontroller in response to receiving the response message from the second microcontroller; calculating an error rate representing a difference between the first controller clock of the first microcontroller and the second controller clock of the second microcontroller based on the initial clock time, the completion time, and the response time; adjusting a frequency of the second controller clock based on the error rate to reduce the error rate between the first controller clock and the second controller clock; The ego is structured in this way.

12. 12. The system of claim 11, wherein the first microcontroller is configured to determine whether a difference between the first controller clock and the second controller clock of the first microcontroller meets a threshold difference.

13. The system of claim 11 , wherein the first microcontroller is further configured to send a boot signal to a kernel of a processor of an SOC coupled to the first microcontroller.

14. The processor of the SOC calculating a second error rate representing a difference between a kernel clock of the kernel of the processor and the first controller clock of the first microcontroller based on the initial clock time between the SOC and the first microcontroller, the completion time between the SOC and the first microcontroller, and the response time between the SOC and the first microcontroller; adjusting a kernel frequency of the kernel clock based on the error rate to reduce the error rate between the kernel clock and the first controller clock; The system of claim 13 configured to:

15. 14. The system of claim 13, wherein the second microcontroller is configured to send a second wake-up signal to a second kernel of a second processor of a second SOC coupled to the second microcontroller.

16. 12. The system of claim 11, wherein the second microcontroller performs a restart function and the second controller clock is updated to match the first controller clock indicated in a timing message received at the second microcontroller from the first microcontroller.

17. sending, by a first microcontroller coupled to a first SOC, an initial timing message to a second microcontroller coupled to a second SOC at an initial time according to a first controller clock of the first microcontroller; receiving, by the first microcontroller, a response message from the second microcontroller indicating a response time in a second controller clock of the second microcontroller; determining, by the first microcontroller, in response to receiving the response message from the second microcontroller, a completion time according to the first controller clock of the first microcontroller; calculating, by the first microcontroller, an error rate representing the difference between the first controller clock of the first microcontroller and the second controller clock of the second microcontroller based on the initial clock time, the completion time, and the response time; adjusting, by the first microcontroller, a frequency of the second controller clock based on the error rate to reduce the error rate between the first controller clock and the second controller clock; A method comprising:

18. 18. The method of claim 17, wherein the first microcontroller is configured to determine whether a difference between the first controller clock and the second controller clock of the first microcontroller meets a threshold difference.

19. 20. The method of claim 17, further comprising sending, by the first microcontroller, a wake-up signal to a kernel of a processor of an SOC coupled to the first microcontroller.

20. calculating, by a processor of the first SOC, a second error rate representing a difference between a kernel clock of a kernel of the processor and the first controller clock of the first microcontroller based on the initial clock time between the SOC and the first microcontroller, the completion time between the SOC and the first microcontroller, and the response time between the SOC and the first microcontroller; adjusting, by the processor of the first SOC, a kernel frequency of the kernel clock based on the error rate to reduce the error rate between the kernel clock and the first controller clock; 20. The method of claim 17, further comprising:

Images