AI inference compiler and runtime tool chain

Abstract

Embodiments include systems and methods for processing sensor data and generating autologous (e.g., autonomous vehicle, robot) hardware operational instructions. The self-body includes any number of machine learning architectures, typically neural network architectures, for processing sensor data and identifying the environment around the self-body and making decisions on the behavior of the self-body. The autologous neural network architecture ingests the sensor data and uses the sensor data to perform any number of operations related to a particular domain or task, such as object recognition or path planning. A graph zoner is trained to assign functions and sensor data in neural network software to certain hardware processing units. An instruction is generated based on the assigned processing unit type using a number of compilers.

Description

[0001] Cross - Reference to Related Applications

[0002] This application claims the benefit of priority of U.S. Provisional Application No. 63 / 377,954, filed on September 30, 2022, the entire content of which is incorporated herein by reference for all purposes. Technical Field

[0003] This application generally relates to implementing neural network architectures for autonomous vehicles or other autonomous electronic devices, and more particularly, to systems and methods for remotely and efficiently compiling and deploying such neural network architectures. Background Art

[0004] Due to the rapid development of computer technology, autonomous navigation technologies for autonomous vehicles and robots (sometimes referred to as "autons") have become ubiquitous. These advancements allow for safer and more reliable autonomous navigation of autons. Autons typically need to navigate in complex and dynamic environments and terrains, which can include vehicles, traffic, pedestrians, cyclists, and various other static or dynamic obstacles. Understanding the surroundings of an auton is necessary for informed and capable decision-making to avoid collisions. This includes developing and deploying complex neural network architectures on autons.

[0005] The increase in data volume and feature fineness naturally gives rise to issues of resource requirements, thus demanding solutions for improving the computational efficiency of both the hardware and software components for autons. This requires sophisticated mechanisms for compiling and deploying neural network architectures on autons. In some cases, the challenges are exacerbated by, for example, remote deployment from software development sources to remote autons and / or deploying software updates of neural network architectures to predefined and fixed execution hardware of autons, etc. Summary of the Invention

[0006] Embodiments described herein include systems and methods that address various drawbacks in the art and may also provide various additional or alternative benefits. Embodiments include hardware and software configurations that improve the performance of the software components and computational hardware of an auton (e.g., autonomous vehicle, robot) in processing sensor data. An auton includes any number of machine learning architectures, typically neural network architectures, for processing sensor data and identifying the environment around the auton and making decisions regarding the behavior of the auton. The neural network architecture of an auton ingests sensor data and performs any number of operations related to a particular domain or task, such as object recognition or path planning, using the sensor data. Any number of compilers transform the software functionality of the neural network architecture and the sensor data into machine code execution instructions for execution by hardware components.

[0007] An embodiment may include a method that includes: obtaining, by a computer, software programming that includes a plurality of functions of a plurality of sub-neural networks of a neural network architecture; assigning, by the computer, one or more sub-neural networks to a plurality of processing units of the self, wherein for each sub-neural network, the computer assigns a processing unit to compute the plurality of functions of the sub-neural network; and generating, by the computer, a plurality of execution instructions for the plurality of processing units by executing a plurality of compilers on the plurality of functions of each sub-network. For each execution instruction, the computer uses a compiler among the plurality of compilers according to the processing unit of the self assigned to the plurality of functions of the sub-neural network.

[0008] The method may include generating, by the computer, a computer file that includes a plurality of execution instructions for the plurality of processing units to execute the plurality of sub-neural networks.

[0009] The method may include transmitting, by the computer, the computer file to the self.

[0010] At least one execution instruction may cause a circuit of the self that includes a plurality of chips to operate in an extended computing mode for parallel execution of the plurality of execution instructions.

[0011] At least one execution instruction may instruct a circuit that includes the self to operate in a redundant mode for primarily executing the plurality of execution instructions by a primary chip among the plurality of chips, the circuit of the self includes a plurality of chips, and the plurality of chips includes a plurality of processing units.

[0012] The method may include applying, by the computer, a scheduling optimizer engine to the execution instructions to generate an execution schedule of the execution instructions, the scheduling optimizer engine includes a neural network layer that is trained to generate an execution schedule for minimizing latency.

[0013] One or more processing units may include at least one of a GPU, a CPU, or an accelerator device.

[0014] One or more processing units may be heterogeneous and include at least two types of processing units.

[0015] The computer may assign a processing unit among the plurality of processing units of the self to apply a sub-neural network according to one or more graphs representing a circuit architecture of the self having a plurality of processing units.

[0016] The method may include training, by the computer, one or more sub-neural networks for quantization-aware training based on one or more processing units by applying each sub-neural network to a training data set that includes data having expected quantization characteristics.

[0017] An embodiment may include a system that includes: a computer including a processor, configured to: obtain software programming that includes multiple functions of multiple sub - neural networks of a neural network architecture; assign one or more sub - neural networks to multiple processing units of itself, wherein for each sub - neural network, the computer assigns processing units to compute the multiple functions of the sub - neural network; and generate multiple execution instructions for the multiple processing units of itself by executing multiple compilers on the multiple functions of each sub - neural network. For each execution instruction, the computer uses a compiler among the multiple compilers according to the processing units of itself assigned to the multiple functions of the sub - neural network.

[0018] The computer may also be configured to generate a computer file that includes multiple execution instructions for the multiple processing units to execute the multiple sub - neural networks.

[0019] The computer may also be configured to transfer the computer file to itself.

[0020] At least one execution instruction may cause multiple chips of itself to operate in an extended computing mode for parallel execution of multiple execution instructions.

[0021] At least one execution instruction may cause multiple chips of itself to operate in a redundant mode for the main chip among the multiple chips to mainly execute multiple execution instructions.

[0022] The computer may also be configured to apply a scheduling optimizer engine to the execution instructions to generate an execution schedule of the execution instructions. The scheduling optimizer engine includes a neural network layer that is trained to generate an execution schedule for minimizing latency.

[0023] The multiple processing units may include at least one of a GPU, a CPU, or an accelerator device.

[0024] The multiple processing units may be heterogeneous, including at least two types of processing units.

[0025] The computer may assign processing units among the multiple processing units of itself to sub - neural networks according to one or more graphs representing the circuit architecture of itself having multiple processing units.

[0026] The computer may also be configured to: train one or more sub - neural networks for quantization - aware training based on one or more processing units by applying each sub - neural network to a training data set stored in a database, the training data set including data having expected quantization characteristics.

[0027] An embodiment may include an apparatus that includes a circuit board, the circuit board including a plurality of system-on-chip (SOC) devices and a plurality of microcontrollers corresponding to the SOC devices; and a processor of the SOC device, the processor of the SOC device being configured to: transmit an initial timing message to a first microcontroller at an initial time according to a core clock of the processor; receive a response message from the first microcontroller, the response message indicating a response time at a first controller clock of the first microcontroller; in response to receiving the response message from the first microcontroller, determine a completion time according to the core clock of the processor; calculate an error rate for the core clock based on the initial clock time, the completion time, and the response time, the error rate representing a difference between the core clock of the processor and the controller clock of the first microcontroller; and adjust a frequency of the core clock based on the error rate to reduce the error rate between the core clock and the first controller clock.

[0028] The processor of the SOC device is configured to transmit the initial timing message in response to receiving a boot instruction from the first microcontroller.

[0029] The first microcontroller coupled to the processor may be configured to: calculate a second error rate representing a difference between a first controller clock of the first microcontroller and a second controller clock of a second microcontroller based on an initial clock time between the first microcontroller and the second microcontroller, a completion time between the first microcontroller and the second microcontroller, and a response time between the first microcontroller and the second microcontroller; and adjust a second frequency of the second controller clock based on the second error rate to reduce the second error rate between the first controller clock and the second controller clock.

[0030] The first microcontroller is configured to determine whether a difference between the first controller clock of the first microcontroller and the second controller clock satisfies a threshold difference.

[0031] The first microcontroller is configured to transmit the initial timing message in response to executing a boot function of the first microcontroller.

[0032] The second microcontroller is configured to execute a reboot function, and wherein a second controller clock of the second microcontroller is updated to match the first controller clock of the first microcontroller, the first controller clock being indicated in a timing message received at the second microcontroller from the first microcontroller for the reboot function.

[0033] An embodiment may include a method that includes: transmitting, by a processor of a system-on-chip (SOC), an initial timing message to a first microcontroller at an initial time according to a core clock of the processor; receiving, by the processor, a response message from the first microcontroller, the response message indicating a response time at a first controller clock of the first microcontroller; determining, by the processor in response to receiving the response message, a completion time according to the core clock of the processor; calculating, by the processor, an error rate for the core clock based on the initial clock time, the completion time, and the response time, the error rate representing a difference between the core clock of the processor and the first controller clock of the first microcontroller; and adjusting, by the processor, a frequency of the core clock based on the error rate to reduce the error rate between the core clock and the first controller clock.

[0034] The method may include: receiving, by the processor, a boot instruction from the first microcontroller, wherein the processor of the SOC device is configured to transmit the initial timing message in response to receiving the boot instruction.

[0035] The processor and the first microcontroller exchange one or more timing messages at boot time according to a bootloader function of the first microcontroller.

[0036] The method may include: calculating, by the first microcontroller, a second error rate representing a difference between a first controller clock of the first microcontroller and a second controller clock of a second microcontroller based on an initial clock time between the first microcontroller and the second microcontroller, a completion time between the first microcontroller and the second microcontroller, and a response time between the first microcontroller and the second microcontroller; and adjusting, at the second microcontroller, a second frequency of the second controller clock based on the second error rate to reduce the second error rate between the first controller clock and the second controller clock.

[0037] An embodiment may include an apparatus that includes: a circuit board that includes a plurality of system-on-chip (SOC) devices and a plurality of microcontrollers corresponding to the SOC devices; and a first microcontroller configured to: transmit, according to a first controller clock of the first microcontroller, an initial timing message to a second microcontroller at an initial time; receive a response message from the second microcontroller, the response message indicating a response time at a second controller clock of the second microcontroller; determine, in response to receiving the response message from the second microcontroller, a completion time according to the first controller clock of the first microcontroller; calculate an error rate representing a difference between the first controller clock of the first microcontroller and the second controller clock of the second microcontroller based on the initial clock time, the completion time, and the response time; and adjust a frequency of the second controller clock based on the error rate to reduce the error rate between the first controller clock and the second controller clock.

[0038] The first microcontroller can be configured to determine whether a difference between a first controller clock of the first microcontroller and a second controller clock satisfies a threshold difference.

[0039] The first microcontroller can also be configured to transmit a boot signal to a core of a processor of an SOC coupled to the first microcontroller.

[0040] The processor of the SOC can be configured to: calculate a second error rate representing a difference between a core clock of the core of the processor and the first controller clock of the first microcontroller based on an initial clock time between the SOC and the first microcontroller, a completion time between the SOC and the first microcontroller, and a response time between the SOC and the first microcontroller; and adjust a core frequency of the core clock based on the error rate to reduce an error rate between the core clock and the first controller clock.

[0041] The second microcontroller can be configured to transmit a second boot signal to a second core of a second processor of a second SOC coupled to the second microcontroller.

[0042] The second microcontroller can perform a reboot function. The second controller clock is updated to match the first controller clock indicated in a timing message received from the first microcontroller at the second microcontroller.

[0043] An embodiment can include a method that includes: transmitting, by a first microcontroller coupled to a first SOC, an initial timing message to a second microcontroller coupled to a second SOC at an initial time according to a first controller clock of the first microcontroller; receiving, by the first microcontroller, a response message from the second microcontroller, the response message indicating a response time at a second controller clock of the second microcontroller; determining, by the first microcontroller in response to receiving the response message from the second microcontroller, a completion time according to the first controller clock of the first microcontroller; calculating, by the first microcontroller, an error rate representing a difference between the first controller clock of the first microcontroller and the second controller clock of the second microcontroller based on the initial clock time, the completion time, and the response time; and adjusting, by the first microcontroller, a frequency of the second controller clock based on the error rate to reduce an error rate between the first controller clock and the second controller clock.

[0044] The first microcontroller can be configured to determine whether a difference between a first controller clock of the first microcontroller and a second controller clock satisfies a threshold difference.

[0045] The method can include transmitting, by the first microcontroller, a boot signal to a core of a processor of an SOC coupled to the first microcontroller.

[0046] The method may include calculating, by a processor of a first SOC, a second error rate representing a difference between a core clock of a core of the processor and a first controller clock of a first microcontroller based on an initial clock time between the SOC and the first microcontroller, a completion time between the SOC and the first microcontroller, and a response time between the SOC and the first microcontroller; and adjusting, by the processor of the first SOC, a core frequency of the core clock based on the error rate to reduce the error rate between the core clock and the first controller clock.

[0047] It should be understood that both the foregoing general description and the following detailed description are exemplary and explanatory and are intended to provide further explanation of the claimed invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Non-limiting embodiments of the present disclosure are described by way of examples in connection with the accompanying drawings, which are schematic and not intended to be drawn to scale. Unless indicated as representing the background art, the drawings represent aspects of the present disclosure.

[0049] Figure 1A Illustrates components of an AI-enabled visual data analysis system according to an embodiment.

[0050] Figure 1B Illustrates various sensors associated with a vehicle (or other type of self-vehicle) according to an embodiment.

[0051] Figure 1C Illustrates components of a self-vehicle according to an embodiment.

[0052] Figure 1D Illustrates certain hardware and software components of a self-vehicle for performing fully or partially autonomous driving (SD) operations according to an embodiment.

[0053] Figure 1E Illustrates certain hardware and software components of a self-vehicle for clock synchronization according to an embodiment.

[0054] Figures 2A - 2B Illustrates a data flow between hardware and software computing components of a computing system of a self-vehicle according to an embodiment.

[0055] Figure 2C Illustrates execution instructions arranged to perform scheduling for execution by IC hardware components of a system according to an embodiment. DETAILED DESCRIPTION

[0056] Reference will now be made to the illustrative embodiments depicted in the accompanying drawings, and specific language will be used here to describe them. However, it should be understood that no limitation of the scope of the claims or the present disclosure is thereby intended. Changes and further modifications of the features of the invention described herein, as well as additional applications of the principles of the subject matter described herein, which would be apparent to those in the relevant art and having the present disclosure, will be considered within the scope of the subject matter disclosed herein. Other embodiments may be used and / or other changes may be made without departing from the spirit or scope of the present disclosure. The illustrative embodiments described in the detailed description are not meant to limit the subject matter presented.

[0057] The embodiments described herein include systems and methods that address various drawbacks in the art and may also provide various additional or alternative benefits. The embodiments include hardware and software configurations that improve the performance of software components and computing hardware of an autonomous entity (e.g., autonomous vehicle, robot) in processing sensor data. The autonomous entity includes any number of machine learning architectures, typically neural network architectures, for processing sensor data, identifying the environment around the autonomous entity, and making decisions regarding the behavior of the autonomous entity.

[0058] At the software level, local or remote computer processing devices of the autonomous entity may execute various software routines of a neural network architecture (or other machine learning architecture). The software defines the layers and functions of the neural network architecture, or defines a hierarchical parent neural network architecture and one or more hierarchical child neural network architectures (sometimes referred to as child networks or subnetworks). The neural network architecture of the autonomous entity ingests sensor data and performs any number of operations related to a particular domain or task using the sensor data, such as object recognition or path planning.

[0059] At the hardware level, the autonomous entity includes various types of computing hardware resources, including various integrated circuit (IC) components and associated firmware or controllers, etc. The computing hardware components implement and execute the software functions of the neural network architecture using sensor data. Any number of compilers transform the software functions of the neural network architecture and the sensor data into machine code execution instructions for execution by the hardware components.

[0060] Embodiments include various software routines for improving the performance and efficiency of processing sensor data within computing hardware by partitioning sensor data into multiple data partitions or portions and partitioning a neural network architecture into sub - networks. The software routines can include a machine - learning architecture that includes a neural network layer defining a graph partitioner. The graph partitioner is configured and trained to assign portions of sensor data to certain functions of the sub - networks and then assign one or more hardware processing units (e.g., GPU, CPU, dedicated hardware AI accelerator device) to perform the functions using the portions of sensor data. The neural network of the graph partitioner is trained to identify and assign functions to be applied to sensor data based on, for example, the type of sensor data or the type of features of the sensor. In some embodiments, the graph partitioner can include a hard - coded or pre - configured mapping between the type of sensor data or features and the software functions of the neural network architecture.

[0061] The graph partitioner can be configured and trained to identify and assign functional portions of sensor data to specific hardware processing units. For example, the graph partitioner can be trained to select a hardware processing unit for performing a function based on desired performance behavior or results (such as optimizing the efficiency of the computing hardware, maximizing a performance metric of the computing hardware, or minimizing a performance metric of the computing hardware). For example, the graph partitioner can be trained to assign certain fine - grained functions to a specially designed AI accelerator device.

[0062] Embodiments can include a heterogeneous collection of hardware processing units. The graph partitioner can be trained to identify and assign a compiler capable of generating execution instructions for machine code compatible with the assigned processing units. The graph partitioner is trained to, for example, assign processing units to perform functions in order to optimize the functionality of the computing hardware, maximize certain performance behaviors of the computing hardware, or minimize certain performance behaviors. By training the graph partitioner to dynamically assign heterogeneous processing units to perform specified functions of the neural network architecture, the graph partitioner optimizes the execution of neural network functions within the hardware. This results in improved accuracy and performance of the neural network architecture when analyzing sensor data.

[0063] Embodiments include a scheduling optimizer that has been pre-configured or trained to identify relationships between different partitions of sensor data and neural network architectures. The scheduling optimizer is used to combine multiple compiled code segments (e.g., execution instructions) into one or more executable files. In some cases, linking generates an ordering or arrangement of instructions for execution by a hardware component. In these cases, by applying a trained neural network architecture of a linking engine to determine dependencies, for example, between data, functions to be performed by processing units, and the processing units assigned to perform the functions, the scheduling optimizer can generate an execution schedule that minimizes latency. The linking engine can then identify the execution schedule to optimize performance relative to the dependencies' constraints. In this way, the scheduling optimizer ensures that execution instructions are organized in a manner that reduces latency and delay, thereby allowing for improved real-time processing of sensor data.

[0064] Downstream hardware and software of the vehicle can ingest the outputs (such as trajectories, speeds, and other navigation determinations of a path planning network) generated by the SD circuitry that executes the neural network architecture to operate or maneuver the vehicle within an environment.

[0065] The vehicle's hardware includes SD circuitry (e.g., an integrated circuit (IC) board) having two (or more) system-on-a-chip (SOC) chips or similar types of IC devices. The SOC chips can perform certain neural network architecture functions by executing execution instructions that are compiled from the source code of a particular neural network architecture into an execution library. A server of a development system trains the neural network architecture on historical and / or current sensor data and the outputs of other neural network architectures. The server then applies the compiler toolchain described herein to the source code and data of the trained neural network architecture to generate the execution library. A neural network architecture of a graph partitioner is trained to assign portions of the execution library to corresponding processing units of the SOC chips. In some cases, a first neural network architecture is programmed to require or expect data inputs from a second neural network architecture. When a first SOC chip loads and executes the execution instructions for the first neural network architecture and a second SOC chip loads and executes the execution instructions for the second neural network architecture, the second SOC chip will then require or expect data inputs from the first SOC chip. The SD circuitry includes a bus that allows the SOC chips to transmit data signals.

[0066] Embodiments include hardware and / or software components within a circuit assembly of a vehicle, which allow the SOC chips of an SD circuit board to operate as if running on a single synchronous clock. In some embodiments, the SD circuit includes two (or more) SOC chips, each SOC chip being coupled to a SOC microcontroller, which can be any microcontroller device or similar processing circuit unit that maintains a chip clock for the corresponding SOC chip. The SOC chips are bootstrapped by a vehicle computer processor at the same time such that the SOC clocks of the SOCs are as close to each other as possible. Each SOC chip includes an operating system (OS) kernel that manages certain operations of the corresponding SOC chip. At preconfigured intervals, each SOC chip exchanges time messages with the microcontroller of the corresponding SOC to confirm that the kernel clock of the SOC is within a threshold distance from the controller clock of the SOC and correct or adjust the microcontroller of the SOC as needed. In this way, each SOC chip maintains clock synchronization between the OS kernel of the SOC and the microcontroller of the SOC. Moreover, at preconfigured intervals, the SOC microcontrollers exchange time messages to confirm that a first controller clock of a first SOC microcontroller is within a threshold distance from a second SOC controller clock of a second SOC microcontroller. If outside the threshold distance, the second microcontroller can be corrected or adjusted to reduce the distance between the controller clocks. In this way, the microcontrollers maintain clock synchronization between the SOC microcontrollers and, by extension, maintain clock synchronization between the SOC chips.

[0067] In some embodiments, the SD circuit includes a controller or other processing unit that maintains clock synchronization between the SOC chips based on interpreting timestamps of sensor inputs (e.g., timestamps of camera inputs) and translating the timestamps between the SOC chips based on differences between the current chip clocks of each SOC chip.

[0068] Figure 1A is a non-limiting example of a component of system 100 in which the methods and systems discussed herein can be implemented. For example, an analytics server can train an AI model and use the trained AI model to generate occupancy datasets and / or maps for one or more vehicles. Figure 1A Illustrates components of an AI-enabled visual data analytics system 100. System 100 can include an analytics server 110a, a system database 110b, an administrator computing device 120, vehicles 140a to 140b (collectively referred to as (multiple) vehicles 140), vehicle computing devices 141a to 141c (collectively referred to as vehicle computing devices 141), and a server 160. System 100 is not limited to the components described herein and can include additional or other components not shown for the sake of brevity, which will be considered within the scope of the embodiments described herein.

[0069] The above components can be connected via network 130. Examples of network 130 can include, but are not limited to, private or public LANs, WLANs, MANs, WANs, and the Internet. Network 130 can include wired and / or wireless communications according to one or more standards and / or via one or more transmission media.

[0070] Communications over network 130 can be performed according to various communication protocols such as Transmission Control Protocol and Internet Protocol (TCP / IP), User Datagram Protocol (UDP), and IEEE communication protocols. In one example, network 130 can include wireless communication according to a Bluetooth specification set or another standard or proprietary wireless communication protocol. In another example, network 130 can also include communication over a cellular network, including, for example, GSM (Global System for Mobile Communications), CDMA (Code Division Multiple Access), or EDGE (Enhanced Data for Global Evolution) networks.

[0071] System 100 illustrates an example of a system architecture and components that can be used to train and execute one or more AI models, such as (multiple) AI models 110c. Specifically, as Figure 1A depicted and described herein, analysis server 110a can use the methods discussed herein to train (multiple) AI models 110c using data retrieved from the self 140 (e.g., by using data streams 172 and 174). When (multiple) AI models 110c have been trained, each self in self 140 can access and execute (multiple) trained AI models 110c. For example, vehicle 140a having a self-computing device 141a can transmit its camera feed to (multiple) trained AI models 110c and can determine the occupancy status of its surrounding environment (e.g., data stream 174). In addition, data ingested and / or predicted by (multiple) AI models 110c with respect to self 140 (at inference time) can also be used to improve (multiple) AI models 110c. Thus, system 100 depicts a continuous loop that can periodically improve the accuracy of (multiple) AI models 110c. In addition, system 100 depicts a loop in which data received by self 140 can also be used in the training phase in addition to the inference phase.

[0072] The analysis server 110a can be configured to collect, process, and analyze navigation data (e.g., images captured during navigation) and various sensor data collected from the vehicle 140. The collected data can then be processed and prepared into a training dataset. The training dataset can then be used to train one or more AI models, such as the AI model 110c. The analysis server 110a can also be configured to collect visual data from the vehicle 140. Using the AI model 110c (trained using the methods and systems discussed herein), the analysis server 110a can generate a dataset and / or an occupancy map for the vehicle 140. The analysis server 110a can display the occupancy map on the vehicle 140 and / or transmit the occupancy map / dataset to the vehicle computing device 141, the administrator computing device 120, and / or the server 160.

[0073] In Figure 1A , the AI model 110c is illustrated as a component of the system database 110b, but the AI model 110c can be stored in a different or separate component, such as a cloud storage device or any other data repository accessible to the analysis server 110a.

[0074] The analysis server 110a can also be configured to display an electronic platform that illustrates various training attributes for training the AI model 110c. The electronic platform can be displayed on the administrator computing device 120 such that an analyst can monitor the training of the AI model 110c. An example of an electronic platform generated and hosted by the analysis server 110a can be a web-based application or website configured to display the training dataset collected from the vehicle 140 and / or the training status / metrics of the AI model 110c.

[0075] The analysis server 110a can be any computing device that includes a processor and a non-transitory machine-readable storage device capable of performing the various tasks and processes described herein. Non-limiting examples of such computing devices can include workstation computers, laptop computers, server computers, etc. Although the system 100 includes a single analysis server 110a, the system 100 can include any number of computing devices operating in a distributed computing environment, such as a cloud environment.

[0076] The vehicle 140 can represent various electronic data sources that transfer data associated with its previous or current navigation session to the analytics server 110a. The vehicle 140 can be any device configured for navigation, such as a car 140a and / or a truck 140c. The vehicle 140 is not limited to being a vehicle and can also include robotic devices. For example, the vehicle 140 can include a robot 140b, which can represent a general-purpose, bipedal, autonomous humanoid robot capable of navigating various terrains. The robot 140b can be equipped with software for achieving balance, navigation, perception, or interaction with the physical world. The robot 140b can also include various cameras configured to transfer visual data to the analytics server 110a.

[0077] Although referred to herein as a "vehicle", the vehicle 140 may or may not be an autonomous device configured for autonomous navigation. For example, in some embodiments, the vehicle 140 can be controlled by a human operator or a remote processor. The vehicle 140 can include various sensors, such as Figure 1B the sensors depicted in. The sensors can be configured to collect data as the vehicle 140 navigates various terrains (e.g., roads). The analytics server 110a can collect the data provided by the vehicle 140. For example, the analytics server 110a can obtain navigation session and / or road / terrain data (e.g., an image of the vehicle 140 navigating on a road) from various sensors, such that the collected data is ultimately used by the AI model 110c for training purposes.

[0078] As used herein, a navigation session corresponds to the journey of the vehicle 140's travel route, regardless of whether the journey is autonomous or human-controlled. In some embodiments, the navigation session can be used for data collection and model training purposes. However, in some other embodiments, the vehicle 140 can refer to a vehicle purchased by a consumer, and the purpose of the journey can be classified as daily use. A navigation session can begin when the vehicle 140 moves more than a threshold distance (e.g., 0.1 mile, 100 feet) or more than a threshold speed (e.g., more than 0 mph, more than 1 mph, more than 5 mph) from a non-moving position. A navigation session can end when the vehicle 140 returns to a non-moving position and / or shuts down (e.g., when the driver exits the vehicle).

[0079] The ego 140 can represent a collection of egos monitored by the analytics server 110a for training the AI model(s) 110c. For example, a driver of a vehicle 140a can authorize the analytics server 110a to monitor data associated with their respective vehicle. As a result, the analytics server 110a can utilize the various methods discussed herein to collect sensor / camera data and generate a training dataset to train the AI model(s) 110c accordingly. The analytics server 110a can then apply the trained AI model(s) 110c to analyze data associated with the ego 140 and predict an occupancy map for the ego 140. Additionally, additional / ongoing data associated with the ego 140 can also be processed and added to the training dataset so that the analytics server 110a can recalibrate the AI model(s) 110c accordingly. Thus, the system 100 depicts a loop in which navigation data received from the ego 140 can be used to train the AI model(s) 110c. The ego 140 can include a processor that executes the trained AI model(s) 110c for navigation purposes. During navigation, the ego 140 can collect additional data about its navigation session, and this additional data can be used to calibrate the AI model(s) 110c. That is, the ego 140 represents an ego that can be used for training, executing / using, and recalibrating the AI model(s) 110c. In a non-limiting example, the ego 140 represents vehicles purchased by customers that can use the AI model(s) 110c for autonomous navigation while improving the AI model(s) 110c.

[0080] The ego 140 can be equipped with various technologies that allow the ego to collect data from its surrounding environment and (possibly) navigate autonomously. For example, the ego 140 can be equipped with an inference chip to run autonomous driving software.

[0081] Various sensors for each ego 140 can monitor data collected associated with different navigation sessions and transmit this data to the analytics server 110a. Figures 1B - 1C A block diagram of sensors integrated within the ego 140 according to an embodiment is illustrated. The number and location of each sensor discussed with respect to Figures 1B - 1C can depend on the type of ego discussed in Figure 1A . For example, the robot 140b can include different sensors than the vehicle 140a or the truck 140c. For example, the robot 140b may not include a airbag activation sensor 170q. Additionally, the sensors of the vehicle 140a and the truck 140c can be positioned differently than shown in Figure 1C .

[0082] As discussed herein, the various sensors integrated within each vehicle body 140 can be configured to measure various data associated with each navigation session. The analysis server 110a can periodically collect the data monitored and collected by these sensors, where the data is processed according to the methods described herein and is used to train the AI model 110c and / or execute the AI model 110c to generate an occupancy map.

[0083] The vehicle body 140 can include a user interface 170a. The user interface 170a can refer to the user interface of the vehicle body computing device (e.g., Figure 1A the vehicle body computing device 141 in). The user interface 170a can be implemented as a display screen integrated with or coupled to the interior of the vehicle, a head-up display, a touch screen, etc. The user interface 170a can include input devices such as a touch screen, a knob, a button, a keyboard, a mouse, a gesture sensor, a steering wheel, etc. In various embodiments, the user interface 170a can be adapted to provide user input (e.g., as a kind of signal and / or sensor information) to other devices or sensors of the vehicle body 140 (e.g., Figure 1B the sensors shown) (such as the controller 170c).

[0084] The user interface 170a can also be implemented with one or more logic devices that can be adapted to execute instructions, such as software instructions, to implement any of the various processes and / or methods described herein. For example, the user interface 170a can be adapted to form a communication link, transmit and / or receive communications (e.g., sensor signals, control signals, sensor information, user input, and / or other information) or execute various other processes and / or methods. In another example, the driver can use the user interface 170a to control the temperature of the vehicle body 140 or activate its features (e.g., the autonomous driving or steering system 170o). Thus, the user interface 170a can combine with other sensors described herein to monitor and collect driving session data. The user interface 170a can also be configured to display various data generated / predicted by the analysis server 110a and / or the AI model 110c.

[0085] The orientation sensor 170b can be implemented as a compass, a float, an accelerometer, and / or one or more of other digital or analog devices capable of measuring the orientation of the vehicle body 140 (e.g., the magnitude and direction of roll, pitch, and / or yaw relative to one or more reference orientations such as gravity and / or magnetic north). The orientation sensor 170b can be adapted to provide a heading measurement to the vehicle body 140. In other embodiments, the orientation sensor 170b can be adapted to provide roll, pitch, and / or yaw rates to the vehicle body 140 using a time series of orientation measurements. The orientation sensor 170b can be positioned and / or adapted to perform orientation measurements relative to a specific coordinate system of the vehicle body 140.

[0086] The controller 170c can be implemented as any suitable logic device (e.g., a processing device, a microcontroller, a processor, an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a memory storage device, a memory reader, or other device or combination of devices) that can be adapted to execute, store, and / or receive appropriate instructions, such as software instructions implementing control loops for controlling various operations of the self 140. Such software instructions can also implement methods for processing sensor signals, determining sensor information, providing user feedback (e.g., via the user interface 170a), querying the device for operating parameters, selecting operating parameters for the device, or performing any of the various operations described herein.

[0087] The communication module 170e can be implemented as any wired and / or wireless interface configured to transmit sensor data, configuration data, parameters, and / or other data and / or signals to Figure 1A any of the features shown (e.g., the analytics server 110a). As described herein, in some embodiments, the communication module 170e can be implemented in a distributed manner such that portions of the communication module 170e are implemented within Figure 1B one or more of the elements and sensors shown. In some embodiments, the communication module 170e can delay the transmission of sensor data. For example, when the self 140 does not have a network connection, the communication module 170e can store the sensor data in a temporary data storage device and transmit the sensor data when the self 140 is identified as having an appropriate network connection.

[0088] The speed sensor 170d can be implemented as an electronic pitot tube, a metering gear or wheel, a water speed sensor, a wind speed sensor, a wind rate sensor (e.g., direction and magnitude), and / or other devices capable of measuring or determining the linear speed of the self 140 (e.g., in the surrounding medium and / or aligned with the longitudinal axis of the self 140) and providing such measurement as a sensor signal that can be transmitted to various devices.

[0089] The gyroscope / accelerometer 170f can be implemented as one or more electronic sextants, semiconductor devices, integrated chips, accelerometer sensors, or other systems or devices capable of measuring the angular velocity / acceleration and / or linear acceleration of the self 140 (e.g., direction and magnitude) and providing such measurement as a sensor signal that can be transmitted to various devices, such as the analytics server 110a. The gyroscope / accelerometer 170f can be positioned and / or adapted to make such measurements relative to a particular coordinate system of the self 140. In various embodiments, the gyroscope / accelerometer 170f can be associated with Figure 1BThe other elements depicted are implemented in a common housing and / or module to ensure a common frame of reference or known transformations between frames of reference.

[0090] The Global Navigation Satellite System (GNSS) 170h can be implemented as a Global Positioning Satellite receiver and / or another device capable of determining the absolute and / or relative position of the vehicle body 140 based on wireless signals received, for example, from space and / or ground sources and capable of providing measurements such as sensor signals (which can be transmitted to various devices). In some embodiments, the GNSS 170h can be adapted to determine the rate, velocity, and / or yaw rate of the vehicle body 140 (e.g., using a time series of position measurements), such as the yaw component of the absolute rate and / or angular rate of the vehicle body 140.

[0091] The temperature sensor 170i can be implemented as a thermistor, an electrical sensor, an electrical thermometer, and / or other devices capable of measuring the temperature associated with the vehicle body 140 and providing such a measurement as a sensor signal. The temperature sensor 170i can be configured to measure the ambient temperature associated with the vehicle body 140, such as the cockpit or dashboard temperature. For example, this ambient temperature can be used to estimate the temperature of one or more elements of the vehicle body 140.

[0092] The humidity sensor 170j can be implemented as a relative humidity sensor, an electrical sensor, an electrical relative humidity sensor, and / or another device capable of measuring the relative humidity associated with the vehicle body 140 and providing such a measurement as a sensor signal.

[0093] The steering sensor 170g can be adapted to physically adjust the heading of the vehicle body 140 according to one or more control signals provided by a logic device (such as the controller 170c) and / or user input. The steering sensor 170g can include one or more actuators and control surfaces of the vehicle body 140 (e.g., a rudder or other type of steering or trim mechanism) and can be adapted to physically adjust the control surface to various positive and / or negative steering angles / positions. The steering sensor 170g can also be adapted to sense the current steering angle / position of such a steering mechanism and provide such a measurement.

[0094] The propulsion system 170k can be implemented as a propeller, a turbine, or other thrust-based propulsion systems, mechanical wheeled and / or tracked propulsion systems, wind / sail-based propulsion systems, and / or other types of propulsion systems that can be used to provide power to the vehicle body 140. The propulsion system 170k can also monitor the direction of the motive power and / or thrust of the vehicle body 140 relative to the reference coordinate system of the vehicle body 140. In some embodiments, the propulsion system 170k can be coupled to the sensor 170g and / or integrated with the steering sensor 170g.

[0095] The passenger restraint sensor 170l can monitor seat belt detection and locking / unlocking components and other passenger restraint subsystems. The passenger restraint sensor 170l can include various environmental and / or status sensors, actuators, and / or other devices that facilitate the operation of safety mechanisms associated with the operation of the vehicle 140. For example, the passenger restraint sensor 170l can be configured to receive motion and / or status data from Figure 1B the other sensors depicted. The passenger restraint sensor 170l can determine whether a safety measure (e.g., seat belt) is being used.

[0096] As Figure 1C depicted, the camera 170m can refer to one or more cameras integrated within the vehicle 140 and can include multiple cameras integrated (or retrofitted) into the vehicle 140. The camera 170m can be an interior- or exterior-facing camera of the vehicle 140. For example, as Figure 1C depicted, the vehicle 140 can include one or more interior-facing cameras that can monitor and collect footage of the passengers of the vehicle 140. The vehicle 140 can include eight exterior-facing cameras. For example, the vehicle 140 can include a front camera 170m-1, front side cameras 170m-2, 170m-3, rear side cameras 170m-4 on each front fender, cameras 170m-5 on each side (e.g., integrated within the B-pillar), and a rear camera 170m-6.

[0097] Referring Figure 1B , the radar 170n and ultrasonic sensors 170p can be configured to monitor the distance of the vehicle 140 to other objects such as other vehicles or immovable objects (e.g., trees or garage doors). The vehicle 140 can also include an autonomous driving or steering system 170o configured to autonomously navigate the vehicle 140 using data collected via various sensors such as the radar 170n, speed sensor 170d, and / or ultrasonic sensors 170p.

[0098] Thus, the autonomous driving or steering system 170o can analyze various data collected by one or more of the sensors described herein to identify driving data. For example, the autonomous driving or steering system 170o can calculate the risk of a front collision based on the speed of the vehicle 140 and its distance to another vehicle on the road. The autonomous driving or steering system 170o can also determine whether the driver is touching the steering wheel. The autonomous driving or steering system 170o can transmit the analyzed data to various features discussed herein, such as an analysis server.

[0099] The airbag activation sensor 170q can predict or detect a collision and cause the activation or deployment of one or more airbags. The airbag activation sensor 170q can transmit data regarding airbag deployment, including data associated with the event that caused the deployment.

[0100] Referring again to Figure 1A , the administrator computing device 120 can represent a computing device operated by a system administrator. The administrator computing device 120 can be configured to display data retrieved or generated by the analytics server 110a (e.g., various analytics metrics and risk scores), where the system administrator can monitor the various models utilized by the analytics server 110a, review feedback, and / or facilitate the training of the (multiple) AI models 110c maintained by the analytics server 110a.

[0101] (Multiple) agents 140 can be any device configured to navigate various routes, such as a vehicle 140a or a robot 140b. As discussed with respect to Figures 1B - 1C , the agent 140 can include various telemetry sensors. The agent 140 can also include an agent computing device 141. Specifically, each agent can have its own agent computing device 141. For example, a truck 140c can have an agent computing device 141c. For the sake of brevity, the agent computing devices are collectively referred to as the (multiple) agent computing devices 141. The agent computing device 141 can control the content presentation on the infotainment system of the agent 140, process commands associated with the infotainment system, aggregate sensor data, manage the communication of data to electronic data sources, receive updates, and / or transmit messages. In one configuration, the agent computing device 141 communicates with an electronic control unit. In another configuration, the agent computing device 141 is an electronic control unit. The agent computing device 141 can include a processor and a non-transitory machine-readable storage medium capable of performing the various tasks and processes described herein. For example, the (multiple) AI models 110c described herein can be stored and executed (or directly accessed) by the agent computing device 141. Non-limiting examples of the agent computing device 141 can include vehicle multimedia and / or display systems.

[0102] In one example of how the (multiple) AI models 110c can be trained, the analytics server 110a can collect data from the agent 140 to train the (multiple) AI models 110c. Before executing the (multiple) AI models 110c to generate / predict an occupancy dataset, the analytics server 110a can use various methods to train the (multiple) AI models 110c. Training allows the (multiple) AI models 110c to ingest data from one or more cameras of one or more agents 140 (without receiving radar data) and predict occupancy data for the surrounding environment of the agent. The operations described in this example can be performed by the Figures 1A - 1Dexecuted by any number of computing devices (e.g., processors of the vehicle 140) operating in the distributed computing system described in

[0103] The analysis server 110a can use the sensors of the vehicle 140 to generate a first data set having a first set of data points, where each data point within the first set of data points corresponds to the position and sensor attributes of at least one voxel in the space around the vehicle 140, and the sensor attributes indicate whether the at least one voxel is occupied by an object having mass.

[0104] To train the AI model(s) 110c, the analysis server 110a can first employ one or more vehicles of the vehicle 140 to drive a specific route. During driving, the vehicle 140 can use one or more of its sensors (including one or more cameras) to generate navigation session data. For example, one or more vehicles 140 equipped with various sensors can navigate a designated route. When one or more of the vehicles 140 cross the terrain, their sensors can capture continuous (or periodic) data of their surrounding environment. The sensors can indicate the occupancy status of the surrounding environment of one or more vehicles 140. For example, the sensor data can indicate various objects having mass in the surrounding environment of one or more vehicles 140 as they navigate their route.

[0105] The analysis server 110a can use the sensor data received from one or more of the vehicles 140 to generate a first data set. The first data set can indicate the occupancy status of different voxels within the surrounding environment of one or more of the vehicles 140. As used herein in some embodiments, a voxel is a three-dimensional pixel that forms the building blocks of the surrounding environment of one or more of the vehicles 140. Within the first data set, each voxel can encapsulate sensor data indicating whether a mass is identified for that particular voxel. As used herein, mass can indicate or represent any object identified using the sensors. For example, in some embodiments, the vehicle 140 can be equipped with LiDAR, which identifies mass by emitting laser pulses and measuring the time it takes for these pulses to travel to an object (having mass) and return. The LiDAR sensor system can operate based on the principle of measuring the distance between the LiDAR sensor and an object within its field of view. This information can be analyzed in combination with other sensor data to identify and characterize different masses or objects within the surrounding environment of one or more of the vehicles 140.

[0106] Various additional data can be used to indicate whether the voxels in the surrounding environment of one or more of the vehicles 140 are occupied by an object having mass. For example, in some embodiments, a digital map of the surrounding environment of one or more of the vehicles 140 (e.g., a digital map of the route the vehicle is traversing) can be used to determine the occupancy status of each voxel.

[0107] In operation, when one or more avatars 140 are navigating, their sensors collect data and transmit the data to the analysis server 110a, as depicted by data stream 176. For example, the avatar 140 computing device 141 may use data stream 176 to transmit sensor data to the analysis server 110a.

[0108] The analysis server 110a may use the cameras of the avatar 140 to generate a second data set having a second set of data points, where each data point within the second set of data points corresponds to the position and image attributes of at least one voxel of the space surrounding the avatar 140.

[0109] The analysis server 110a may receive the camera feeds of one or more avatars 140 navigating the same route as in the first step. In some embodiments, the analysis server 110a may perform the first step and the second step simultaneously (or at the same time). Alternatively, two (or more) different avatars 140 may navigate the same route, where one avatar transmits its sensor data and the second avatar 140 transmits its camera feed.

[0110] One or more avatars 140 may include one or more high-resolution cameras that capture a continuous visual data stream from the surroundings of the one or more avatars 140 as the one or more avatars 140 navigate in the route. The analysis server 110a may then use the camera feeds to generate a second data set, where the visual elements / depictions of different voxels of the surroundings of the one or more avatars 140 are included within the second data set.

[0111] In operation, when one or more avatars 140 are navigating, their cameras collect data and transmit the data to the analysis server 110a, as depicted by data stream 172. For example, the avatar computing device 141 may use data stream 172 to transmit image data to the analysis server 110a.

[0112] The analysis server 110a may use the first data set and the second data set to train an AI model, whereby the AI model 110c trains itself using the corresponding positions of each data point to associate each data point within the first set of data points with the corresponding data point within the second set of data points, where after being trained, the AI model 110c is configured to receive a camera feed from a new avatar 140 and predict the occupancy status of at least one voxel of the camera feed.

[0113] Using the first data set and the second data set, the analysis server 110a can train the AI model(s) 110c such that the AI model(s) 110c can associate different visual attributes of a voxel (within a camera feed in the second data set) with the occupancy state of the voxel (within the first data set). In this way, after being trained, the AI model(s) 110c can receive a camera feed (e.g., from a new ego body 140) without receiving sensor data and then determine the occupancy state of each voxel for the new ego body 140.

[0114] The analysis server 110a can generate a training data set including the first data set and the second data set. The analysis server 110a can use the first data set as ground truth. For example, the first data set can indicate different positions of voxels and their occupancy states. The second data set can include visual (e.g., camera feed) depictions of the same voxels. Using the first data set, the analysis server 110a can label the data such that the data record(s) associated with each voxel corresponding to an object are indicated as having a positive occupancy state.

[0115] The labeling of the occupancy states of different voxels can be performed automatically and / or manually. For example, in some embodiments, the analysis server 110a can use a human reviewer to label the data. For example, as discussed herein, a camera feed from one or more cameras of a vehicle can be shown to a human reviewer on an electronic platform for labeling. Additionally or alternatively, the AI model(s) 110c can ingest the entire data, where the AI model(s) 110c identify the corresponding voxels, analyze the first digital map, and associate the image(s) of each voxel with its corresponding occupancy state.

[0116] Using the ground truth, the AI model(s) 110c can be trained to analyze the visual elements of each voxel and associate them with whether the voxel is occupied by a mass. Thus, the AI model 110c can retrieve the occupancy state of each voxel (using the first data set) and use this information as the ground truth. The AI model(s) 110c can also retrieve the visual attributes of the same voxels using the second data set.

[0117] In some embodiments, the analysis server 110a can use a supervised training method. For example, using the ground truth and the received visual data, the AI model(s) 110c can train itself such that it can predict the occupancy state for a voxel using only the image of the voxel. As a result, when trained, the AI model(s) 110c can receive a camera feed, analyze the camera feed, and determine the occupancy state for each voxel within the camera feed (without using radar).

[0118] The analysis server 110a can feed a series of training data sets to the AI model(s) 110c and obtain a set of predicted outputs (e.g., predicted occupancy status). The analysis server 110a can then compare the predicted data with the ground truth data to determine the differences and train the AI model(s) 110c by adjusting the internal weights and parameters of the AI model 110c proportional to the determined differences according to a loss function. The analysis server 110a can train the AI model(s) 110c in a similar manner until the predictions of the trained AI model 110c are accurate to a certain threshold (e.g., recall or precision).

[0119] Additionally or alternatively, the analysis server 110a can use an unsupervised method in which the training data sets are not labeled. Since labeling the data within the training data sets can be time-consuming and may require excessive computing power, the analysis server 110a can utilize unsupervised training techniques to train the AI model 110c.

[0120] After the AI model 110c is trained, the ego 140 can use it to predict the occupancy data of the surrounding environment of one or more egos 140. For example, the AI model(s) 110c can divide the surrounding environment of the ego into different voxels and predict the occupancy status for each voxel. In some embodiments, the AI model(s) 110c (or the analysis server 110a using the data predicted by the AI model 110c) can generate an occupancy map or occupancy network representing the surrounding environment of one or more egos 140 at any given time.

[0121] In another example of how to use the AI model(s) 110c, after training the AI model(s) 110c, the analysis server 110a (or the local chip of the ego 140) can collect data from the ego (e.g., one or more egos among the egos 140) to predict the occupancy data set for one or more egos 140. This example describes how the AI model(s) 110c can be used to predict the occupancy data for one or more egos 140 in real time or near real time. This configuration can have a processor that executes the AI model, such as the analysis server 110a. However, one or more actions can be locally executed via, for example, a chip located within one or more egos 140. In operation, the AI model(s) 110c can be locally executed via the ego 140 such that the results can be used for autonomous navigation itself.

[0122] The processor can input image data of the space around the ego object 140 into the AI model 110c using the camera of the ego object 140. The processor can collect and / or analyze data received from various cameras (e.g., outward-facing cameras) of one or more ego objects 140. In another example, the processor can collect and aggregate the footage recorded by one or more cameras of the ego object 140. The processor can then transmit the footage to the (multiple) AI models 110c trained using the methods discussed herein.

[0123] The processor can predict the occupancy attributes of multiple voxels by executing the AI model 110c. The (multiple) AI models 110c can use the methods discussed herein and use the received image data to predict the occupancy status for different voxels surrounding one or more ego objects 140.

[0124] The processor can generate a data set based on the multiple voxels and their corresponding occupancy attributes. The analysis server 110a can generate a data set including its occupancy status according to the corresponding coordinate values of different voxels. This data set can be a queryable data set available for transmitting the predicted occupancy status to different software modules.

[0125] In operation, one or more ego objects 140 can collect image data from their cameras and transmit the image data to the processor (locally placed on one or more ego objects 140) and / or the analysis server 110a, as depicted by the data stream 172. The processor can then execute the (multiple) AI models 110c to predict the occupancy data for one or more ego objects 140. If the prediction is performed by the analysis server 110a, the occupancy data can be transmitted to one or more ego objects 140 using the data stream 174. If the processor is locally placed within one or more ego objects 140, the occupancy data is transmitted to the ego computing device 141 ( Figure 1A not shown in the figure).

[0126] Using the methods discussed herein, the training of the (multiple) AI models 110c can be performed such that the execution of the (multiple) AI models 110c can be locally executed (at inference time) on any ego within the ego object 140. The data collected (e.g., navigation data collected during the navigation of the ego object 140, such as image data of the journey) can then be fed back into the (multiple) AI models 110c so that additional data can improve the (multiple) AI models 110c.

[0127] Figure 1DIllustrates certain hardware and software components of vehicle 140 for performing all or part of the self-driving (SD) operation according to an embodiment. Vehicle 140 includes an SD circuit 150 and a vehicle computing device 141, and the vehicle computing device 141 may include components that are the same as or different from those of the SD circuit 150. The SD circuit 150 includes SD chips 152a to 152b (generally referred to as SD chips 152), such as system-on-chip (SoC) integrated circuit chips. Each SD chip 152 includes a non-transitory machine-readable memory, such as DRAMs 190a to 190b (generally referred to as DRAMs 190) and SRAM. The SD chip 152 also includes various types of processing units, including GPUs 191, CPUs 193a to 193c (generally referred to as CPUs 193), and specially designed AI accelerator devices 192a to 192b (generally referred to as AI accelerator devices 192). The SD chip 152 includes inter-chip interfaces 194a to 194b (generally referred to as chip interfaces 194), such as peripheral component interconnect (PCI) or high-speed PCI (PCIe). The SD chip 152 transmits signals via an inter-chip bus 199 according to the protocol and programming of the chip interface 194.

[0128] As regarding Figure 1AAs mentioned, the analysis server 110 (or other computing device) can compile the compiled execution binary for the neural network architecture software and download the compiled execution binary to the self 140 and / or the self-computing device 141. The self-computing device 141 can generate and / or execute various software programming operations and execution binaries for managing the operation of the SD circuit 150 (or other hardware), which can include execution instructions for applying the neural network architecture to sensor data types from the self 140. The executable instructions received, generated, and / or executed by the self-computing device 141 can include executable instructions for managing the operation of the components of the SD circuit 150. For example, the compiled executable binary can include instructions that, for example, indicate the destination SD chip 152 for transmitting data signals between the SD chips 152 via the bus 199, or indicate the execution SD chip 152 for performing certain functions of the neural network architecture. For example, the instructions generated and compiled for a neural network architecture for identifying traffic sign objects using data from the camera 170m can be loaded into and executed by the components of the first SD chip 152a, and the instructions compiled for a neural network architecture for path planning can be loaded and executed by the components of the second SD chip 152b. If the path planning neural network architecture is trained to use the data output generated by the traffic sign neural network architecture, then the instructions for the first SD chip 152a direct the first SD chip 152a to transmit the output data signal to the second SD chip 152b via the bus 199; and the instructions for the second SD chip 152b direct the second SD chip 152b to use such data signals received from the first SD chip 152a.

[0129] In an example embodiment, the SD circuit 150 includes two SD chips 152a to 152b. In many cases, the SD chips 152 operate in a redundant mode or a failover mode of operation, where the first SD chip 152a acts as the primary chip and the second SD chip 152b acts as the secondary chip. For example, the first SD chip 152a is prioritized to execute most of the executable instructions, and in the event of a problem with the first SD chip 152a, the second SD chip 152b is called to operate as a failover or redundancy.

[0130] The SD circuit 150 can operate in an extended computing mode that balances the execution instruction pipeline between the SD chips 152. As an example, the self-computing device 141 executes software routines for compiling the execution instructions to be executed by the processing units 191 to 193 of the SD chips 152 and distributing the execution instructions to the optimal hardware components of the SD circuit 150.

[0131] The SD chip 152 includes inter-chip memory sequencers 195a to 195b (commonly referred to as the inter-chip memory sequencer 195). The inter-chip memory sequencer 195 includes a hardware IC device that is used to coordinate the communication of signals between system-on-chips (SoCs), such as the SD chip 152. In some implementations, the inter-chip memory sequencer 195 may include a non-transitory storage location that provides a shared storage space accessible to the SD chip 152. In some implementations, the inter-chip memory sequencer 195 performs operations to coordinate the transfer of data signals between the SD chips 152, for example, by generating various control signals. The inter-chip memory sequencer 195 may implement one or more inter-chip communication protocols, such as PCIe, SPI, or I2C, etc.

[0132] The hardware and software components of the runtime system of the self 140 (e.g., the self-computing device 141, the SD circuit board 150, the controller 180) receive a compilation program schedule (e.g., Figures 2A - 2C the execution instructions 218a to 218h of the execution schedule 217 of

[0133] which may be in the form of an execution binary file 216), and run the execution instructions on various types of heterogeneous cores (e.g., the processing units 190 to 193 of each chip 152) and across the various chips 152 of the circuit 150. The runtime system (represented as being executed by the controller 180) contains software components for an inter-chip computing scheduler, a heterogeneous hardware scheduler (e.g., CPU accelerator, GPU accelerator, AI accelerator), an inter-chip memory sequencer 195 for scheduling and managing inter-chip signals via the chip interface 194 and the bus 199, and a clock synchronizer (e.g., the OS kernel). In some implementations, the runtime system programming supports model parallelism across multiple SD chips 152. In this way, the self 140 can review and delve into the clock synchronizer.

[0134] The SD chip 152 includes one or more DRAMs 190 or other types of non-transitory memories for storing data inputs to the SD chip 152. The data inputs can be stored in the DRAM 190 for various computational references by the processing unit. In some configurations, the AI accelerator device 192 includes SRAM such that the SD chip 152 moves data from the DRAM 190 for storage into the SRAM of the AI accelerator device 192. The AI accelerator device 192 performs computations according to execution instructions and moves the data back to the DRAM 190 or other destinations of the SD circuitry 150.

[0135] The SD chip 152 includes various types of processing units, which can include any hardware integrated circuit (IC) processor device capable of performing the various processes and tasks described herein. Non-limiting examples of processing unit types include GPU 191, CPU 193, AI accelerator device 192, microcontroller, ALU, ASIC, and FPGA, among others. The processing unit can execute the computational functions of the programming layers that define the neural network architecture or sub-architecture. The compiler outputs execution instructions representing the operations of the neural network architecture, which are executed by the autologous computing device 141 (or other components of the autologous 140).

[0136] The AI accelerator device 192 is a hardware accelerator designed specifically for neural network operations, which beneficially focuses on improvements such as optimizing power and performance (e.g., low latency). The AI accelerator device 192 includes hardware IC devices (e.g., microcontroller, ALU, ASIC, FPGA, processor device) designed to operate quickly when processing neural network architectures. For example, as transformer neural network architectures (e.g., GPT) and other types of neural network modeling techniques become more popular, other types of processing units (e.g., CPU 193, GPU 191) may become slower due to design philosophies aimed at more general implementation use cases. For example, neural network architectures, sub-neural networks (e.g., Figure 2BThe moving object network 206b) or the child neural network in [object] performs computer vision or object recognition by implementing various GPTs (or other types of transformers) on the image sensor data, thus beneficially replacing the prior art for post-processing of visual neural networks. The AI accelerator device 192 is specifically designed for neural network operations that allow the GPT transformer to run locally in the computing components of the vehicle 140, such that the AI accelerator device 192 provides faster and more efficient processing than a conventional GPU 191 or CPU 193 that performs similar GPT transformations. In this way, the AI accelerator device 192 reduces or eliminates latency and improves overall efficiency, contributing to the vehicle 140's ability to make real-time decisions. Moreover, when performing the more refined and complex functions of a neural network architecture such as a transformer network (e.g., a transformer), the structural design and design theory of the AI accelerator device 192 draw relatively less power than a conventional GPU 191 or CPU 193.

[0137] In some embodiments, a transformer (e.g., GPT) may be adapted to execute in the vehicle 140, thereby improving the overall performance of the computing components for an autonomous or semi-autonomous vehicle 140. For example, typical transformers are generally resource-intensive, consume a large amount of power, and / or cause significant latency when processing outputs, and thus impede the overall performance of the vehicle 140. As such, transformers are powerful neural network architectures that are generally not deployed in the vehicle 140. To address this issue, the transformers of the vehicle 140 described herein can be deployed without the attention module softmax, which typically appears in conventional transformers. The embodiments described herein may include transformers with softmax-free attention and may implement ReLU activation. By replacing softmax with ReLU in such transformers, the transformer architecture can now be deployed in an autonomous or semi-autonomous vehicle 140.

[0138] In some embodiments, the SD circuit includes a controller or other processing unit that maintains clock synchronization between SOC chips based on interpreting timestamps of sensor inputs (e.g., timestamps of camera inputs) and translating the timestamps between SOC chips according to the differences between the current chip clocks of each SOC chip.

[0139] Figure 1EShows certain hardware and software components of the self - body 140 for maintaining clock synchronization between SD chips 152 according to an embodiment. In such an embodiment, the SD circuit 150 includes two (or more) SD chips 152 and two (or more) microcontrollers 155a - 155b (commonly referred to as microcontrollers 155), and the SD chips 152 are coupled to the corresponding microcontrollers 155. For example, the first SD chip 152a is coupled to the first microcontroller 155a, and the second SD chip 152b is coupled to the second microcontroller 155b. Each SD chip 152 includes an OS kernel 153 and executes the OS kernel 153, which manages the operations of the SD chip 152, including executing various execution instructions of the neural network architecture and managing the operations of the SD chip 152.

[0140] The OS kernel 153 can include any type of OS capable of executing the various processes and tasks described herein, including managing the execution of execution instructions and maintaining a software - based OS clock. The OS of the OS kernel 153 includes, for example, Linux, Unix, etc.

[0141] The microcontroller 155 includes any type of processing circuit unit capable of executing the various processes and tasks described herein. Non - limiting examples include microcontrollers, controllers, ALUs, FPGAs, ASICs, etc. The SOC chip 152 includes one or more processing units that execute the OS kernel 153 (e.g., Linux) and one or more smaller microcontrollers 155. The microcontroller 155 includes low - level programming for performing various low - level functions. For example, the functions of the microcontroller 155 include a bootloader function, where the microcontroller 155 boots various components of the SD chip 152, including the processing unit that executes the OS kernel 153. In some embodiments, the microcontroller 155 or other devices of the SD circuit 150 can include or be coupled to a clock oscillator or counter that oscillates at a given frequency or increments a monotonic clock.

[0142] The microcontroller 155 can communicate with the processing unit that executes the OS kernel 153 via a given interface (e.g., mailbox, Ethernet, UART, PCIe) to exchange data signals such as time messages or correction instructions. Similarly, the first microcontroller 155a can communicate with the second microcontroller 155b via another interface (e.g., mailbox, Ethernet, UART, PCIe) to exchange time messages or correction instructions.

[0143] Typically, the microcontroller 155 executes a bootloader function to boot the microcontroller 155 and the OS kernel 153 at the same time and relatively early after the startup self 140. At boot time, the microcontroller 155 and the OS kernel 153 pass various timing messages to synchronize with each other. In this way, before the processor unit of the OS kernel 153 has a chance to start execution, the boot and synchronization functions of the microcontroller 155 logically form a common monotonic time clock for the SD circuit 150.

[0144] When the programming of the OS kernel 153 (e.g., Linux) starts to boot on the processor, early in the boot of the OS kernel 153, the OS kernel 153 resets the monotonic kernel clock of the OS kernel 153. The OS kernel 153 can reset the kernel clock to match the controller clock of the corresponding microcontroller 155. This reset can occur before anything else or before a large number of operations have a chance to start on the SD chip 152, or before not too many operations have started in the OS kernel 153. When the OS kernel 153 boots, the OS kernel 153 synchronizes the kernel clock to the corresponding microcontroller 155.

[0145] In some implementations, at a preconfigured resynchronization phase or threshold time, the SD chip 152 and the microcontroller 155 actively maintain synchronized kernel and controller clocks through a control loop operation, in which the SD chip 152 and the microcontroller 155 exchange timing messages. For example, at a synchronization interval (e.g., once per second), the OS kernel 153 will measure the time error relative to the corresponding microcontroller 155, and in some cases, the OS kernel 153 will instruct the connected microcontroller 155 to initiate a small adjustment to correct the controller clock.

[0146] In some embodiments, the clock synchronization operation includes a redundant fallback function. In some cases, the SD chip 152 (e.g., the second SD chip 152b) may suffer a fatal error and must be rebooted, while the other SD chip 152 (e.g., the first SD chip 152a) can continue to operate until the rebooted SD chip 152 (e.g., the second SD chip 152b) recovers. In such a case, the operational SD chip 152 (e.g., the first SD chip 152a) continues to maintain the kernel clock of the OS kernel 153 (e.g., the first OS kernel 153a) and the controller clock of the corresponding microcontroller 155 (e.g., the first microcontroller 155a), and thus, by extension, maintains the overall logic synchronization clock for the SD circuit 150. As such, when the rebooted SD chip 152b recovers, the overall synchronization can continue via, for example, synchronization messages between the first microcontroller 155a and the second microcontroller 155b. The rebooted SD chip 152b does not need to restart a new kernel clock and a new controller clock at zero or some other initialization time. The microcontroller 155b and the OS kernel 153b can start the kernel clock and the controller clock at the current monotonic time of the overall synchronization clock for the SD circuit 150. The microcontroller 155b can perform a recovery, reboot, or boot process that includes a synchronization process with the operating microcontroller 155a, where the microcontrollers 155 exchange time messages to indicate the current time of the controller clock of the operating microcontroller 155a, which reflects the overall synchronization clock for the SD circuit 150. Then, the recovered microcontroller 155b and the recovered OS kernel 153b can exchange time messages indicating the current time of the controller clock of the microcontroller 155, which reflects the overall synchronization clock of the SD circuit 150. In this way, the components of the SD circuit 150 do not need to calculate, distribute, or translate the time difference between the discontinuous clocks of the SD chip 152. After rebooting, the application software executed in the recovered SD chip 152b and the recovered OS kernel 153b can almost immediately start executing and participating to reconstruct the pre-failure state with limited latency and / or without a continuous time delay due to the continuous calculation of what would be discontinuous clocks. This beneficially improves fault tolerance and supports failover redundancy for the self 140.

[0147] The OS kernel 153 and / or the microcontroller 155 can perform an error correction function that adjusts the frequency of the controller clock by a relatively small amount such that the microcontroller 155 or the OS kernel 153 increases or decreases the frequency and the clock time (e.g., the controller clock, the kernel clock) by some amount.

[0148] As mentioned, each SD chip 152 has multiple types of synchronization operations, including: kernel-to-controller synchronization operations between the OS kernel 153 and the corresponding microcontroller 155; and SOC synchronization or controller-to-controller synchronization operations between the microcontrollers 155 of the SD chip 152.

[0149] In the first type of synchronization operation (e.g., synchronizing the OS kernel 153a with the corresponding microcontroller 155a), the OS kernel 153 sends a timing message to the microcontroller 155. The OS kernel 153 and the microcontroller 155 of the SD chip 152 each include a communication interface (e.g., mailbox, Ethernet, UART, PCI) for exchanging timing messages (or other types of messages) via a signal connection, line, or bus according to the protocol of a specific interface.

[0150] At boot time and / or at a preconfigured interval, the OS kernel 153 transmits a timing message to the microcontroller 155 and receives a return timing message from the microcontroller 155, and refers to the relevant clock time to determine whether one or more clocks have drifted beyond a threshold distance. The OS kernel 153 sends an initial timing message to the microcontroller 155 at a first time (T1). The OS kernel 153 retrieves the current time with reference to the kernel clock and assigns the current time as the initial message time (T1) for the initial timing message. The microcontroller 155 receives the initial timing message and refers to the current time of the controller clock of the microcontroller 155. The microcontroller 155 sends a response timing message to the OS kernel 153 at a response time (T2). The OS kernel 153 assigns the response time to the response timing message according to the current time of the controller clock. The OS kernel 153 receives the response timing message indicating the response time (T2) and retrieves the current time of the kernel clock with reference to the kernel clock. When the OS kernel 153 receives the response message, the OS kernel 153 assigns the current time of the kernel clock as the completion time (T3) to the response timing message. The OS kernel 153 calculates an average time based on the average of the kernel times, which includes the initial message time (T1) and the completion time (T3). Then, the OS kernel 153 compares the average time with the response time received from the microcontroller 155 to calculate and output the difference between the average time and the response time. In an example configuration, the first OS kernel 153a can calculate an offset that represents the estimated amount of time error between the first monotonic kernel clock of the first OS kernel 153a and the first monotonic controller clock of the first microcontroller 155a. In this example configuration, the offset is calculated for the difference between the average time and the response time.

[0151] In a second type of synchronization operation (e.g., synchronizing the first microcontroller 155a with the second microcontroller 155b), the OS kernel 153 sends a timing message to the microcontrollers 155. Each microcontroller 155 includes a communication interface (e.g., mailbox, Ethernet, UART, PCI) for exchanging timing messages (or other types of messages) with other microcontrollers 155 via a signal connection, line, or bus according to the protocol of the specific interface.

[0152] At boot time and / or at a preconfigured interval, the microcontrollers 155 automatically start exchanging timing messages with each other. The microcontrollers 155 do not need to establish a handshake or exchange any prior or predicate communication; the microcontrollers 155 may be sending timing messages. Each microcontroller 155 uses the timing messages to capture and determine the message time (T1) and the completion time (T3), and receives the response time (T2) returned from another microcontroller 155. Then, each microcontroller 155 can calculate the offset with respect to the OS kernel 153 as described above. Thus, each microcontroller 155 can estimate the offset with respect to the peer microcontroller 155. And they just do it automatically.

[0153] In some cases, a particular microcontroller 155 is rebooted and restored. In such a case, the reboot or restore function of the restored microcontroller 155b can act as a follower of the operating microcontroller 155a, while the operating microcontroller 155a acts as the host. The first microcontroller 155a and the second microcontroller 155b consider the controller clock of the first microcontroller 155a as the master controller clock. At reboot and restore, the first microcontroller 155a and the second microcontroller 155b exchange a timing message that directly assigns the controller clock of the first microcontroller 155a to the controller clock of the second microcontroller 155b.

[0154] In some implementations, one or more offsets representing the difference between two clocks are transmitted as error signals to, for example, a proportional integral (PI) controller or a phase locked loop (PLL) controller. Using known algorithmic techniques, the PLL or PI controller can calculate an error rate based on the offset, where the input is the offset as a phase and the output is a rate. The error rate is the rate that the OS kernel 153 (or microcontroller 155) must correct. At each instance or interval (e.g., per second), a new measurement of the rate, the rate difference (offset) between the two clocks, is calculated, and the OS kernel 153 (or microcontroller 155) can adjust the kernel clock (or controller clock) by applying the rate to the kernel clock (or controller clock). As an example, if the calculated rate indicates that the kernel clock of the first OS kernel 153a is two parts per million faster than the first microcontroller 155a, the first OS kernel 153a adjusts to a new frequency that matches a frequency two parts per million slower in the first OS kernel 153a in order to match the frequency oscillation of the first microcontroller 155a.

[0155] The first microcontroller 155a transmits a timing message to the second microcontroller 155b and receives a return timing message from the second microcontroller 155b, and determines whether one or more clocks have drifted beyond a threshold distance with reference to the relevant clock times. The first microcontroller 155a sends an initial timing message to the second microcontroller 155b at a first time (T1). The OS kernel 153 retrieves the current time with reference to the kernel clock and assigns the current time as the initial message time (T1) for the initial timing message. The microcontroller 155 receives the initial timing message and references the current time of the controller clock of the microcontroller 155. The microcontroller 155 sends a response timing message to the OS kernel 153 at a response time (T2). The OS kernel 153 assigns the response time to the response timing message based on the current time of the controller clock. The OS kernel 153 receives the response timing message indicating the response time (T2) and retrieves the current time of the kernel clock with reference to the kernel clock. When the OS kernel 153 receives the response message, the OS kernel 153 assigns the current time of the kernel clock as the completion time (T3) to the response timing message. The OS kernel 153 calculates an average time based on the average of the kernel times, which includes the initial message time (T1) and the completion time (T3). Then, the OS kernel 153 compares the average time with the response time (T2) received from the microcontroller 155 to calculate and output the difference between the average time and the response time. In an example configuration, the first OS kernel 153a can calculate an offset that represents the estimated amount of time error between the first monotonic kernel clock of the first OS kernel 153a and the first monotonic controller clock of the first microcontroller 155a. In this example configuration, the offset is calculated for the difference between the average time and the response time.

[0156] Figures 2A - 2B Depicts the data flow between the hardware and software computing components of a system 200 according to an embodiment, the system 200 being used to develop and compile executable instructions 218a through 218h (commonly referred to as execution instructions 218) at a development system 201 for loading as an execution binary 216 into a self 202. Figure 2C Illustrates execution instructions 218 generated and organized into an execution schedule 217 for execution by the circuit hardware components of system 200.

[0157] One or more computing devices of the development system 201 (e.g., the analytics server 110) may execute software programming that defines one or more neural network architectures 204, a hardware model training engine 207, a compiler 210, and an execution scheduler 212, as well as other types of software programming routines. Additionally, the (multiple) computing devices of the development system 201 may execute software programming for training, retraining, and tuning the neural network architecture 204 or portions thereof (e.g., parameters, hyperparameters, weights, layers, functions) on various forms of historical and / or current sensor data from any number of selves 202 for prediction accuracy and consistency. Additionally or alternatively, the (multiple) computing devices of the development system 201 may execute software programming for training, retraining, and tuning the neural network architecture 204 or portions thereof (e.g., parameters, hyperparameters, weights, layers, functions) on various types of input data, output data, or prediction data of the neural network architecture 204, where the input data, output data, or prediction data is relative to or scaled to, or optimized for, the data size and format implemented by the hardware components of the self 202, for example.

[0158] During training or inference time, the (multiple) computing devices of the development system 201 extract features or tensors from input data, such as historical or current sensor data collected by sensors of the self 202, or historical or current sensor data retrieved from a database of the development system 201 containing historical data captured by the self 202. The (multiple) computing devices of the development system 201 feed the input data into the neural network architecture 204 or sub-architecture for various operations (e.g., computer vision, object recognition), apply the neural network architecture 204 to the input data to generate a predicted output, and adjust or retrain portions of the neural network architecture 204 (e.g., parameters, hyperparameters, weights, layers) during training.

[0159] The (multiple) computing devices of the development system 201 apply a graph partitioner to the sensor data to generate data partitions or portions. The autologous computing device 141 applies a compiler set (not shown), which may logically form a compiler toolchain for the neural network architecture of the autologous 202, for compiling and debugging code for executing the layers of the neural network architecture for sensor data interpretation. Each compiler is used to transform a high-level programming language into machine code including execution instructions to be executed by the hardware of the SD circuit 150. The compiler may be configured or optimized to compile programming code according to the specific architecture or type of the processing unit of the SD chip (e.g., CPU 193, GPU 191, or dedicated AI accelerator device 192 hardware). The scheduling optimizer of the execution scheduler may combine multiple compiled code fragments (e.g., executable instructions) into one or more executable files or data streams for execution scheduling (not shown).

[0160] The scheduling optimizer and the execution scheduler obtain an instruction set for execution and map the execution instructions to the hardware components of the SD circuit (e.g., GPU 191, AI accelerator device 192, CPU 193) to execute specific execution instructions. In some implementations, the scheduling optimizer of the execution scheduler is trained to optimize the operations to be executed in the hardware components of the SD circuit. The scheduling optimizer is trained to determine or is preconfigured with the time or latency requirements for the hardware components to execute the operations of the execution instructions. This is generally possible because such performance timing or latency metrics are known, substantially static, quickly computable, or pre-stored. In this way, the scheduling optimizer maps the execution instructions to the components of the SD circuit based on minimizing or optimizing the latency. Additionally or alternatively, the scheduling optimizer determines which hardware components of the SD circuit should execute which execution instructions based on the characteristics of the execution instructions (e.g., which compiler generated the machine code of the execution instructions). In this way, the scheduling optimizer maps the execution instructions to the processing units based on the compiler that generated the specific execution instructions.

[0161] As Figure 2AAs shown in FIG. 0, system 200 includes software programming for executing neural network architecture 204 at the (multiple) computing devices of development system 201, including various domain - specific or task - specific sub - networks 206a through 206e (commonly referred to as sub - networks 206), but may include other types of machine - learning architectures. The source code of the software programming defines aspects of neural network architecture 204 (e.g., parameters, hyper - parameters, weights, layers, functions), which includes the source code defining any number of sub - networks 206, these sub - networks 206 including a traffic signal network 206a, a moving object network 206b, a lane network 206c, an occupancy network 206d, and a path planning network 206e for performing operations for a specific domain or task, but embodiments may include additional or alternative types of sub - networks 206. The software components of development system 201 may also include a compiler 210, a hardware model training engine 207, and an execution scheduler 212, and the execution scheduler 212 includes the functionality defining a scheduling optimizer 214.

[0162] During the training of neural network architecture 204 and sub - networks 206, development system 201 executes hardware model training engine 207 to train sub - networks 206 on models or representative data for the hardware components of autobot 202. In this way, hardware model training engine 207 provides quantization - aware training (QAT) to neural network architecture 204 such that sub - networks 206 can be optimized for the hardware components of autobot 202 and quantization resilience. Advantageously, the QAT functionality of hardware model training engine 207 trains neural network architecture 204 and sub - networks 206 to be more efficient and smaller in size. The QAT functionality of hardware model training engine 207 trains sub - networks 206 by applying sub - networks 206 to various or desired quantization weights (e.g., 16 - bit floating - point values; 8 - bit floating - point values; 8 - bit integers) and activations (e.g., 16 - bit floating - point values; 8 - bit floating - point values; 8 - bit integers). The QAT function of hardware model training engine 207 forces neural network architecture 204 and / or each sub - network 206 to learn to operate with lower - precision numbers. For example, hardware model training engine 207 may train sub - networks 206 to ingest or produce 8 - bit floating - point or integer values instead of, for example, ingesting or producing 16 - bit floating - point or integer values.

[0163] Advantageously, by processing smaller quantized data sizes, the hardware model training engine 207 can improve efficiency and reduce the demand for computing resources on the self 202. Additionally, this can reduce the power consumption required by the hardware of the self 202. For example, by training the neural network architecture 204 to be quantization-aware or flexible, the neural network architecture 204 and the hardware executing the compiled neural network architecture 204 operate sufficiently on lower-precision data values or primitives. In this way, compared to the power consumption that would otherwise be used if the hardware of the self 202 ran a neural network architecture 204 trained and compiled at a higher precision, the hardware of the self 202 can run a neural network architecture 204 trained and compiled at a lower precision with a relatively lower power consumption.

[0164] The neural network architecture 204 can include or be connected to the neural network of the graph partitioner 208. The graph partitioner 208 can partition the sensor data received via the ingestion layer 202 into subnets 206 for processing the sensor data. The neural network architecture 204 is logically partitioned into subnets 206. The neural network layer of the graph partitioner 208 is trained to parse the sensor data into data parts and then map the data parts to the subnets 206 to perform the functions of the specific subnets 206. The graph partitioner 208 maps the sensor data parts to the subnets 206 according to, for example, the data type or features in the sensor data used by the subnets 206.

[0165] After assigning the sensor data and functions to the subnets 206, the graph partitioner 208 can then assign which hardware components of the SD circuit 201 should implement and execute the functions of the subnets 206. The neural network layer of the graph partitioner 208 is trained to assign the sensor data parts and the functions of the specific subnets 206 to specific processing units (such as CPUs, GPUs, dedicated hardware AI accelerator devices) of the chips 203a to 203b (commonly referred to as chips 203) (such as SoCs, SD chips) of the SD circuit 201. For example, the graph partitioner 208 is configured and trained to assign relatively simple functions of the subnets 206 using the sensor data parts to the CPU of the chip 203 and assign relatively complex functions of another subnet 206 using another sensor data part to the AI accelerator device of the chip 203.

[0166] Reference Figure 2B, the domain - specific sub - network 206 includes a traffic signal network 206a, a moving object network 206b, a lane network 206c, an occupancy network 206d, and a path planning network 206e. The sub - network 206 performs various types of domain - specific or task - related functions for a given purpose (e.g., object recognition, path planning) according to the software programming of the neural network layers of the specific sub - network 206. As an example, the functions of the traffic sign network 206a include recognizing certain object image data (or other types of sensor data), such as traffic control, stop signs, yield signs, speed limit signs, and topological signs. As another example, the functions of the occupancy network 206d include determining per - voxel occupancy, per - voxel rate, and 3D surface geometry semantics, as well as other image - related metrics in the image data (or other types of sensor data). As another example, the functions of the path planning network 206e include generating a trajectory or path for self - navigation using image data or other types of sensor data, and adjusting the path to avoid collisions, etc.

[0167] The sub - network 206 performs various operations or functions to compute sensor data and produce the output of the specific sub - network 206. In some cases, these functions include process computations or operations. In some cases, these functions include the child neural networks of the specific sub - network 206. Non - limiting examples of the child network types of the sub - network 206 include ingestion or intake layers (sometimes referred to as "head layers"), correction layers, regularized neural network (RegNet) layers, transformer layers, and multi - layer perceptron (MLP) layers, etc.

[0168] In some cases, the graph partitioner 208 is also configured and trained to assign input sensor data portions to the specific sub - network 206 based on the type of child neural networks in the sub - network 206. Additionally or alternatively, the graph partitioner 208 is trained to assign functions and sensor data to the hardware of the SD circuit 201 based on the capabilities of the hardware. For example, the graph partitioner 208 is trained to optimize the efficiency of the hardware and / or reduce the latency of the hardware, or to achieve any additional or alternative performance behaviors of the hardware. As an example, the graph partitioner 208 assigns a series of inter - related or dependent functions of the child networks within the sub - network 206 to one or more CPUs of the same first chip 203a, which can improve efficiency. As another example, the graph partitioner 208 assigns complex functions of the sub - network 206 to the AI accelerator device of the chip 203, which can improve the computing speed. The graph partitioner 208 can be trained to maximize or minimize performance metrics, or the graph partitioner can be trained to balance and optimize according to multiple performance metrics.

[0169] Back to Figure 2A, the system 200 includes a compiler toolchain that includes a collection of compilers 210a to 210c (commonly referred to as compiler 210). The compiler 210 includes software programming that is configured to transform the high-level programming language of the layers and functions of the neural network architecture 204, as well as sensor data, into machine code of execution instructions 218 that can be executed by the hardware of the SD circuit 201. The system 200 includes a heterogeneous collection of processing units and compiler 210, where the compiler 210 is configured to transform from a given high-level programming language (e.g., the functions and sensor data portion of the sub-network 206) into a given machine code (e.g., execution instructions 218) compatible with the assigned processing unit. In some embodiments, the software routines of the compiler 210 can selectively compile machine code for more than one type of processing unit, such that the compiler 210 can generate execution instructions 218 for more than one type of processing unit (e.g., CPU, GPU, AI accelerator device).

[0170] The graph partitioner 208 (or other components of the system 200) is configured and trained to identify which compiler 210 should be assigned to compile which functions of the sub-network 206. For example, continuing with the previous example, the graph partitioner 208 is configured and trained to assign the relatively simple functions of the sub-network 206 to the first compiler 210a, which is programmed to generate execution instructions 218 for the CPU, and to assign the relatively complex functions of another sub-network 206 to the second compiler 210b, which is programmed to generate execution instructions 218 for the AI accelerator device. The output of the compiler 210 is the execution instructions 218 compiled from the sensor data portion and the software programming for the functions of the sub-network 206.

[0171] The system 200 includes an execution scheduler 212 for the neural network, which includes layers that define a scheduling optimizer 214 (sometimes referred to as a "linker"). The scheduling optimizer 214 of the execution scheduler 212 can combine multiple compiled code fragments (e.g., executable instructions 218) generated by the compiler 210 of the compiler toolchain into one or more execution binaries 216, which include one or more executable files or a data stream of execution instructions 218. The execution scheduler 212 can arrange or order the execution instructions 218 for sequential execution by the hardware of the SD circuit 201.

[0172] The execution binary file 216 is downloaded from the software of the system 200 to the non-transitory memory of the hardware of the SD circuit 201. In the SD circuit 201, the software- or firmware-based controller component of the SD circuit 201 parses the execution instructions 218 of the execution binary file 216 and loads the execution instructions 218 into one or more non-transitory memories (not shown) accessible by the assigned processing unit (or other hardware components) of the chip 203. Then, the processing unit executes the execution instructions 218 to execute the functions of the neural network architecture 204.

[0173] Reference Figure 2C , in some cases, the execution instructions 218 generated by the compiler 210 can be arranged and logically represented as an execution schedule 217. The output of the compiler toolchain includes the execution instructions 218 for the functions of the sub-network 206. For ease of understanding, Figure 2C the execution instructions 218 in show the chip 203 assigned to perform the operation, the processing unit (e.g., GPU, CPU, AI accelerator device) assigned to perform the operation, and which neural network architecture's functions will be executed. However, the execution instructions 218 of potential embodiments can include additional or alternative types of information, such as input data sources or interfaces, output data destinations or interfaces, and computational instructions.

[0174] As an example, the first execution instruction 218a indicates that the first GPU (gpu0) of the second chip 203b (SoC1) is assigned to perform these functions. The first execution instruction 218a indicates that the first GPU of the second chip 203b (i.e., running gpu0, soc1) executes the functions of the corrective neural network architecture within the occupancy network 206d.

[0175] The downstream hardware and software of the vehicle can ingest the output generated by the SD circuit 201 that executes the neural network architecture 204, such as the trajectory, speed, and other navigation determinations of the path planning network 206e, to operate or maneuver the vehicle within the environment.

[0176] The various illustrative logical blocks, modules, circuits, and algorithm steps described in connection with the embodiments disclosed herein can be implemented as electronic hardware, computer software, or a combination of both. To clearly illustrate this interchangeability of hardware and software, the various illustrative components, blocks, modules, circuits, and steps have been generally described above in terms of their functionality. Whether this functionality is implemented as hardware or software depends on the particular application and the design constraints imposed on the overall system. Those skilled in the art can implement the described functionality in different ways for each particular application, but such implementation decisions should not be construed as causing a departure from the scope of the present invention.

[0177] Embodiments implemented in computer software can be implemented in software, firmware, middleware, microcode, hardware description language, or any combination thereof. Code segments or machine-executable instructions can represent a process, function, subroutine, program, routine, subroutine, module, software package, class, or any combination of instructions, data structures, or program statements. By passing and / or receiving information, data, arguments, properties, or memory contents, a code segment can be coupled to another code segment or hardware circuit. Information, arguments, properties, data, etc. can be passed, forwarded, or transmitted via any suitable means, including memory sharing, message passing, token passing, network transmission, etc.

[0178] The actual software code or dedicated control hardware used to implement these systems and methods does not limit the present invention. Thus, without referring to specific software code, the operation and behavior of the systems and methods can be described, and it can be understood that the software and control hardware can be designed to implement the systems and methods based on the description herein.

[0179] When implemented in software, the functions can be stored as one or more instructions or code on a non-transitory computer-readable or processor-readable storage medium. The steps of the methods or algorithms disclosed herein can be embodied in processor-executable software modules, which can reside on a computer-readable or processor-readable storage medium. The non-transitory computer-readable or processor-readable medium includes both computer storage media and tangible storage media, and the tangible storage media facilitates the transfer of a computer program from one place to another. The non-transitory processor-readable storage medium can be any available medium accessible by a computer. By way of example and not limitation, such non-transitory processor-readable media can include RAM, ROM, EEPROM, CD-ROM, or other optical disk storage, magnetic disk storage, or other magnetic storage devices, or any other tangible storage medium that can be used to store the desired program code in the form of instructions or data structures and can be accessed by a computer or processor. The disks and optical disks used herein include optical discs (CDs), laser discs, optical discs, digital versatile discs (DVDs), floppy disks, and Blu-ray discs, where "disks" generally reproduce data magnetically, while "optical discs" reproduce data optically with a laser. Combinations of the above should also be included within the scope of computer-readable media. Additionally, the operations of the methods or algorithms can reside as one or any combination or set of code and / or instructions on a non-transitory processor-readable medium and / or computer-readable medium, which can be incorporated into a computer program product.

[0180] The foregoing description of the disclosed embodiments is provided to enable a person skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments without departing from the spirit or scope of the invention. Thus, the invention is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the following claims and the principles and novel features disclosed herein.

[0181] Although various aspects and embodiments have been disclosed, other aspects and embodiments are also contemplated. The disclosed aspects and embodiments are for illustrative purposes only and are not intended to be limiting, and the true scope and spirit are indicated by the following claims.

Claims

1. An entity, comprising: A circuit board, the circuit board including a plurality of system-on-chip (SOC) devices and a plurality of microcontrollers corresponding to the SOC devices; And A processor of the SOC device, the processor of the SOC device being configured to: Transmit an initial timing message to a first microcontroller at an initial time according to a core clock of the processor; Receive a response message from the first microcontroller, the response message indicating a response time at a first controller clock of the first microcontroller; In response to receiving the response message from the first microcontroller, determine a completion time according to the core clock of the processor; Calculate an error rate for the core clock based on the initial clock time, the completion time, and the response time, the error rate representing a difference between the core clock of the processor and the controller clock of the first microcontroller; And Adjust a frequency of the core clock based on the error rate to reduce the error rate between the core clock and the first controller clock.

2. The system according to claim 1, wherein the processor of the SOC device is configured to transmit the initial timing message in response to receiving a boot instruction from the first microcontroller.

3. The system according to claim 1, wherein the first microcontroller coupled to the processor is configured to: calculate a second error rate based on the initial clock time between the first microcontroller and the second microcontroller, the completion time between the first microcontroller and the second microcontroller, and the response time between the first microcontroller and the second microcontroller, the second error rate representing the difference between the first controller clock of the first microcontroller and the second controller clock of the second microcontroller; and adjust a second frequency of the second controller clock based on the second error rate to reduce the second error rate between the first controller clock and the second controller clock.

4. The system according to claim 3, wherein the first microcontroller is configured to determine whether the difference between the first controller clock of the first microcontroller and the second controller clock satisfies a threshold difference.

5. The system according to claim 1, wherein the first microcontroller is configured to transmit the initial timing message in response to executing a boot function of the first microcontroller.

6. The system according to claim 1, wherein the second microcontroller is configured to execute a reboot function, and wherein the second controller clock of the second microcontroller is updated to match the first controller clock of the first microcontroller indicated in the timing message received from the first microcontroller at the second microcontroller for the reboot function.

7. A method, comprising: An initial timing message is transmitted by a processor of a system-on-chip (SOC) to a first microcontroller at an initial time according to a core clock of the processor; A response message is received by the processor from the first microcontroller, the response message indicating a response time at a first controller clock of the first microcontroller; In response to receiving the response message, a completion time is determined by the processor according to the core clock of the processor; An error rate for the core clock is calculated by the processor based on the initial clock time, the completion time, and the response time, the error rate representing a difference between the core clock of the processor and the first controller clock of the first microcontroller;And The frequency of the core clock is adjusted by the processor based on the error rate to reduce the error rate between the core clock and the first controller clock.

8. The method according to claim 7, further comprising receiving, by the processor, a boot instruction from the first microcontroller, wherein the processor of the SOC device is configured to transmit the initial timing message in response to receiving the boot instruction.

9. The method according to claim 8, wherein the processor and the first microcontroller exchange one or more timing messages at boot time according to the boot loader function of the first microcontroller.

10. The method according to claim 7, further comprising: A second error rate is calculated by the first microcontroller based on the initial clock time between the first microcontroller and a second microcontroller, the completion time between the first microcontroller and the second microcontroller, and the response time between the first microcontroller and the second microcontroller, the second error rate representing a difference between the first controller clock of the first microcontroller and a second controller clock of the second microcontroller; And At the second microcontroller, a second frequency of the second controller clock is adjusted based on the second error rate to reduce the second error rate between the first controller clock and the second controller clock.

11. A system, comprising: A circuit board, the circuit board including a plurality of system-on-chip (SOC) devices and a plurality of microcontrollers corresponding to the SOC devices; A first microcontroller, the first microcontroller being configured to: Transmit an initial timing message to a second microcontroller at an initial time according to a first controller clock of the first microcontroller; Receive a response message from the second microcontroller, the response message indicating a response time at a second controller clock of the second microcontroller; In response to receiving the response message from the second microcontroller, determine a completion time according to the first controller clock of the first microcontroller; Calculate an error rate based on the initial clock time, the completion time, and the response time, where the error rate represents the difference between the first controller clock of the first microcontroller and the second controller clock of the second microcontroller; And Based on the error rate, adjust the frequency of the second controller clock to reduce the error rate between the first controller clock and the second controller clock.

12. The system according to claim 11, wherein the first microcontroller is configured to determine whether the difference between the first controller clock of the first microcontroller and the second controller clock satisfies a threshold difference.

13. The system according to claim 11, wherein the first microcontroller is further configured to transmit a boot signal to a core of a processor of an SOC coupled to the first microcontroller.

14. The system according to claim 13, wherein the processor of the SOC is configured to: calculate a second error rate based on the initial clock time between the SOC and the first microcontroller, the completion time between the SOC and the first microcontroller, and the response time between the SOC and the first microcontroller, the second error rate representing a difference between a core clock of the core of the processor and the first controller clock of the first microcontroller; and adjust a core frequency of the core clock based on the error rate to reduce the error rate between the core clock and the first controller clock.

15. The system according to claim 13, wherein the second microcontroller is configured to transmit a second boot signal to a second core of a second processor of a second SOC coupled to the second microcontroller.

16. The system according to claim 11, wherein the second microcontroller performs a reboot function, and wherein the second controller clock is updated to match the first controller clock indicated in a timing message received at the second microcontroller from the first microcontroller.

17. A method, comprising: The first microcontroller coupled to the first SOC transmits an initial timing message to the second microcontroller coupled to the second SOC at an initial time according to the first controller clock of the first microcontroller; The first microcontroller receives a response message from the second microcontroller, where the response message indicates a response time at the second controller clock of the second microcontroller; The first microcontroller, in response to receiving the response message from the second microcontroller, determines a completion time according to the first controller clock of the first microcontroller; The first microcontroller calculates an error rate based on the initial clock time, the completion time, and the response time, where the error rate represents the difference between the first controller clock of the first microcontroller and the second controller clock of the second microcontroller; And The first microcontroller adjusts the frequency of the second controller clock based on the error rate to reduce the error rate between the first controller clock and the second controller clock.

18. The method according to claim 17, wherein the first microcontroller is configured to determine whether the difference between the first controller clock of the first microcontroller and the second controller clock meets a threshold difference.

19. The method according to claim 17, further comprising transmitting, by the first microcontroller, a boot signal to a core of a processor of an SOC coupled to the first microcontroller.

20. The method according to claim 17, further comprising: The processor of the first SOC calculates a second error rate based on the initial clock time between the SOC and the first microcontroller, the completion time between the SOC and the first microcontroller, and the response time between the SOC and the first microcontroller, where the second error rate represents the difference between the core clock of the core of the processor and the first controller clock of the first microcontroller; And The processor of the first SOC adjusts the core frequency of the core clock based on the error rate to reduce the error rate between the core clock and the first controller clock.

Images