Method, processing system and non-transitory processor-readable medium for allocating processing resources to concurrently-executing neural networks

Abstract

Various embodiments include a method executed by a vehicle's processor for allocating computational resources to concurrently executing neural networks. These methods may include: determining a priority order for each neural network based on the contribution of each of the plurality of neural networks executing on the vehicle processing system to the overall vehicle safety performance; and allocating computational resources to the plurality of neural networks based on the determined priority order. In some embodiments, these methods may dynamically adjust the hyperparameters of one or more neural networks.

Description

Technical Field

[0001] This case concerns the allocation of processing resources to neural networks that are running concurrently. Prior Technology

[0002] A growing number of autonomous vehicles (such as autonomous and semi-autonomous cars, drones, mobile robots, and other suitable machines) include multiple sensors for collecting information about their environment, and processing systems for processing that information for route planning, navigation, collision avoidance, and so on. One example is the Advanced Driver Assistance Systems (ADAS) used in autonomous and semi-autonomous vehicles. To enable rapid analysis of sensor data and quick decisions based on it, data from each sensor is processed by neural networks, and thus the vehicle's computing system (such as a System-on-a-Chip (SOC)) executes many neural networks concurrently. Summary of the Invention

[0003] Each state includes a method for allocating computing resources to concurrently executing neural networks, which can be implemented in a processing device within the vehicle. Each state executed by the vehicle processor may include: determining the priority of each neural network based on the contribution of each of the plurality of neural networks executing on the vehicle processing system to the overall vehicle safety performance; and allocating computing resources to the plurality of neural networks based on the determined priority of each neural network.

[0004] In some embodiments, determining the priority of each neural network based on its contribution to the overall vehicle safety performance of a plurality of neural networks executing on the vehicle processing system may include: determining the priority of each neural network based on its contribution to the overall vehicle safety performance in the context of vehicle operation. In some embodiments, determining the priority of each neural network based on its contribution to the overall vehicle safety performance of a plurality of neural networks executing on the vehicle processing system may include: determining the priority of the neural networks based on an indication of the contribution of each neural network to the overall vehicle safety performance, wherein the indication is provided by each neural network. In some embodiments, the overall vehicle safety performance may be calculated based on a model using the inference accuracy and speed of the plurality of neural networks as input values. In some embodiments, the overall vehicle safety performance may indicate ride quality based on factors perceptible to human passengers, such as improving passenger experience.

[0005] In some embodiments, determining the priority of each of the plurality of neural networks executing on the vehicle processing system may include: determining the relative performance of one or more of the plurality of neural networks based on the output inference per second and output accuracy of each neural network. In some embodiments, allocating computational resources to the plurality of neural networks based on the determined priority of each neural network may include: adjusting one or more hyperparameters of the one or more neural networks based on the determined priority of the one or more neural networks. In some embodiments, adjusting one or more hyperparameters of the one or more neural networks based on the determined priority of the one or more neural networks may include: adjusting one or more hyperparameters based on the performance-efficiency curves of the one or more neural networks.

[0006] Some states may also include: determining the performance of one or more neural networks among the plurality of neural networks that will use the allocated computing resources; and reallocating computing resources to one or more neural networks among the plurality of neural networks based on the determined performance of the one or more neural networks. Some states may also include: monitoring the dynamic availability of the computing resources and the actual use of the computing resources. In such a state, reallocating computing resources to one or more neural networks among the plurality of neural networks based on the determined performance of the one or more neural networks may include: reallocating computing resources to one or more neural networks among the plurality of neural networks based on the dynamic availability of the computing resources and the actual use of the computing resources.

[0007] In some cases, reallocating computational resources to one or more neural networks based on the determined performance of one or more of the plurality of neural networks may include: readjusting one or more hyperparameters of one or more neural networks based on the determined performance of each of the plurality of neural networks. Some cases may include: determining whether available computational resources have increased, decreased, or remained the same; and, in response to the determination that available computational resources have increased, adjusting the hyperparameters of one or more neural networks that have a relatively large impact on overall vehicle safety performance. Such cases may also include: in response to the determination that available computational resources have decreased, adjusting the hyperparameters of one or more neural networks that have a relatively small impact on overall vehicle safety performance.

[0008] Another embodiment includes a vehicle comprising a processor configured with processor-executable instructions to perform the operations of any of the methods outlined above. Another embodiment includes a non-transitory processor-readable storage medium having processor-executable software instructions stored thereon, the processor-executable software instructions being configured to cause the processor to perform the operations of any of the methods outlined above. Another embodiment includes a processing apparatus for use in a vehicle and configured to perform the operations of any of the methods outlined above. Simple Explanation of the Diagram

[0009] The accompanying drawings, which are incorporated herein and form part of this specification, illustrate exemplary embodiments and, together with the general description provided above and the detailed description provided below, serve to explain the features of the various embodiments.

[0010] Figures 1A and 1B are block diagrams illustrating components suitable for implementing various embodiments of the vehicle.

[0011] Figure 1C is a block diagram of components of a vehicle suitable for implementing various embodiments.

[0012] Figure 2A is a component block diagram illustrating the components of an example vehicle management system according to various embodiments.

[0013] Figure 2B is a component block diagram illustrating components of another example vehicle management system according to various embodiments.

[0014] Figure 3 is a block diagram illustrating components of an example on-chip system for use in a vehicle according to various embodiments.

[0015] Figure 4 is a component block diagram of an example system configured to allocate processing resources to concurrently executing neural networks according to various embodiments.

[0016] Figure 5A is a conceptual diagram illustrating components of a vehicle computing system adapted to allocate processing resources to concurrently executing neural networks according to various embodiments.

[0017] Figure 5B is a table illustrating the functions that can be performed by various neural networks according to various embodiments.

[0018] Figure 5C is a table showing the configuration of hyperparameters for various neural networks according to various embodiments.

[0019] Figure 5D is a graph and illustration of examples of performance-effectiveness curves for neural networks according to various embodiments.

[0020] Figures 5E, 5F, and 5G are graphical and illustration examples of the effects of adjusting hyperparameters according to various embodiments.

[0021] Figure 5H is a graph and illustration comparing thermal throttling and the allocation of processing resources to concurrently executing neural networks according to various embodiments.

[0022] Figure 6A is a flowchart illustrating the operation of a method for allocating processing resources to concurrently executing neural networks, which can be executed by a vehicle's processor according to various embodiments.

[0023] Figures 6B and 6C are program flowcharts illustrating operations that can be executed by a vehicle's processor as part of a method for allocating processing resources to concurrently executing neural networks, according to various embodiments. Implementation

[0024] The various embodiments will be described in detail with reference to the accompanying drawings. Where possible, the same component symbols will be used throughout the drawings to refer to the same or similar parts. References to specific examples and embodiments are for illustrative purposes and are not intended to limit the scope of the various embodiments or claims.

[0025] Vehicle computing systems executing numerous concurrent neural networks have limited computational resources and must operate within a thermal envelope. Traditional approaches to addressing processing limitations in vehicle computing systems involve reducing the operating frequency of the processor executing all neural networks. While these methods are effective in ensuring that thermal limits are not exceeded, they result in an overall reduction in processing speed and the output of each neural network, which can compromise safety functions in some cases. However, not all concurrently executing neural networks are equally critical for safe vehicle operation. For example, neural networks executing maneuvering and collision avoidance functions may be more critical to ensuring safe vehicle operation at all times than neural networks used for dynamic route changes based on traffic density. Therefore, traditional resource and thermal management methods may unnecessarily degrade the performance of safety-critical neural networks.

[0026] Various embodiments include methods and vehicle computing systems implementing these methods that selectively allocate processing resources to concurrently executing neural networks, as necessary, based at least in part on their relative impact on overall vehicle safety performance (especially taking into account current operating conditions, internal and external conditions), to avoid exceeding processing limits. Therefore, instead of limiting resources for applications or reducing the operating frequency of processors executing neural networks based on arbitrary ranking or priority, the embodiments consider how to reduce resources for various applications affecting overall vehicle safety performance when allocating processing resources to various neural networks within the vehicle computing system.

[0027] The ground transportation industry is increasingly looking to leverage the growing capabilities of cellular and wireless communication technologies by adopting Intelligent Transportation Systems (ITS) to improve interoperability and safety for both driver-operated and autonomous vehicles. The Cellular Vehicle-to-Everything (C-V2X) protocol, defined by the 3GPP (3rd Generation Partnership Project), supports ITS technology and serves as the foundation for direct communication between vehicles and surrounding communication devices.

[0028] C-V2X defines two transmission modes that together provide 360º non-line-of-sight awareness and a higher level of predictability to enhance road safety and autonomous driving. The first transmission mode includes direct C-V2X, encompassing vehicle-to-vehicle (V2V), vehicle-to-infrastructure (V2I), and vehicle-to-pedestrian (V2P) communication, and provides enhanced range and reliability in a dedicated ITS 5.9 GHz spectrum independent of cellular networks. The second transmission mode includes vehicle-to-network (V2N) communications in mobile broadband systems and technologies such as: third-generation (3G) wireless mobile communication technologies (e.g., GSM Evolution (EDGE) systems, CDMA2000 systems, etc.), fourth-generation (4G) wireless mobile communication technologies (e.g., LTE systems, improved LTE systems, Mobile World Congress Microwave Access (Mobile WiMAX) systems, etc.), and fifth-generation (5G) wireless mobile communication technologies (e.g., 5G New Radio (5G NR) systems, etc.).

[0029] The term system-on-chip (SOC) is used herein to refer to a single integrated circuit (IC) chip that contains multiple resources or processors integrated on a single substrate. A single SOC can contain circuits for digital, analog, mixed-signal and radio frequency functions. A single SOC can also include any number of general-purpose or specialized processors (digital signal processors, modem processors, video processors, etc.), memory blocks (such as ROM, RAM, flash memory, etc.) and resources (such as timers, voltage regulators, oscillators, etc.). SOCs can also include software used to control integrated resources and processors and to control peripherals.

[0030] The term “system-level package” (SIP) may be used herein to refer to a single module or package containing multiple resources, computing units, cores, or processors on two or more IC chips, substrates, or SOCs. For example, a SIP may include a single substrate on which multiple IC chips or semiconductor chips are stacked in a vertical configuration. Similarly, a SIP may include one or more multi-chip modules (MCMs) on which multiple IC or semiconductor grains are encapsulated into a unified substrate. SIPs may also include multiple separate SOCs coupled together via high-speed communication circuits and tightly packaged in, for example, a single motherboard or a single wireless device. The proximity of the SOC facilitates high-speed communication and the sharing of memory and resources.

[0031] As used herein, the terms "part," "system," "unit," "module" are intended to include computer-related entities such as, but not limited to, hardware, firmware, combinations of hardware and software, software, or software in execution configured to perform a particular operation or function. For example, a part may be, but is not limited to being: a program running on a processor, a processor, an object, an executable, a running thread, a program, and / or a computer. By way of exposition, both the application and the communication device running on the processor of the communication device may be referred to as components. One or more parts may be located in a program and / or execution thread, and the parts may be located on one processor or core and / or distributed between two or more processors or cores. In addition, these parts can be executed from various non-temporary computer-readable media having various instructions and / or data structures stored thereon. Parts may communicate by means of local and / or remote procedures, function or program dialing, electronic signals, data packets, memory read / write, and other known communication methods related to networks, computers, processors, and / or programs.

[0032] A growing number of autonomous vehicles (such as autonomous and semi-autonomous cars, drones, mobile robots, and other suitable machines) include multiple sensors for collecting information about their environment, and processing systems that process the collected information to perform vehicle control, route planning, navigation, collision avoidance, and similar vehicle safety and control functions. One example is the Advanced Driver Assistance Systems (ADAS) deployed in autonomous and semi-autonomous vehicles. To enable rapid analysis and decision-making from sensor data, data from one or more sensors can be processed by neural networks dedicated to sensor data and / or specific analysis functions. Decomposing complex data analysis functions into operations performed by specially trained neural networks enables rapid data analysis and parallel processing of data from different sensors. Therefore, the vehicle's computing system can execute many neural networks concurrently during normal operation.

[0033] Various embodiments include methods for allocating processing resources to various neural networks based on their contribution or relative importance to overall vehicle safety and reliable operation, and processing systems configured to implement such methods. Some embodiments may include allocating vehicle processing resources to each neural network based on the priority or importance of concurrently executing neural networks. In some embodiments, (e.g., the processor of the vehicle processing system) may control its processing requirements by adjusting certain hyperparameters of the neural networks. Various embodiments enable a significant reduction in resource allocation to neural networks that are less important to safe vehicle operation (e.g., lower priority), while executing neural networks that are more important to safe vehicle operation (e.g., higher priority) at full capacity or high efficiency.

[0034] Various embodiments may include: determining the priority of each neural network based on the contribution of each of the plurality of neural networks executed on the vehicle processing system to the overall vehicle safety performance; and allocating computing resources to the plurality of neural networks based on the determined priority of each neural network.

[0035] In some embodiments, the vehicle processing system may prioritize each neural network based on the contribution of each of a plurality of neural networks executing on the vehicle processing system in the context of vehicle operation to the overall vehicle safety performance. In some embodiments, the vehicle processing system may prioritize each neural network based on indications provided by each of the plurality of neural networks executing on the vehicle processing system regarding its contribution to the overall vehicle safety performance. In some embodiments, the vehicle processing system may determine the relative performance of one or more of the plurality of neural networks based on the output inference per second and the output accuracy of each neural network. In some embodiments, each neural network may provide an indication of its dynamic relative importance (according to a "voting" scheme). In some embodiments, the dynamic relative importance of each neural network may not be limited to the output of the neural network. For example, the processing system may determine the dynamic relative importance of a neural network based on a specific fidelity or accuracy in the output provided by the neural network.

[0036] In some embodiments, the vehicle processing system may assign priorities to neural networks based on their relative importance to safe vehicle operation in the context of the vehicle's current operating state and / or environment. Different relative priorities may be assigned to different neural networks depending on whether the vehicle is on a highway, in busy urban traffic, or in a parking lot. For example, when the vehicle is traveling on a highway, neural networks used for collision avoidance functions may be considered more important than neural networks used for dynamic traffic route change functions.

[0037] In some embodiments, the processing system may assign priority to neural networks performing non-mission-critical tasks or functions, such as neural networks performing operations that are relatively less important to safe vehicle operation. For example, neural networks performing relatively unimportant tasks or functions may include neural networks supporting functions such as lane change frequency, dynamic route change frequency, weather detection and / or ambient temperature control, and vehicle night vision. In some embodiments, the processing system may consider user preferences when assigning priorities to some neural networks. For example, user preferences may indicate preferences for avoiding traffic jams or avoiding inclement weather.

[0038] In some embodiments, the processing system may semi-dynamically assign priorities to neural networks based on vehicle operating scenarios or contexts. In some embodiments, the processing system may be configured with a lookup table storing relative importance values ​​or indications for individual neural networks associated with certain driving scenarios or contexts. For example, the lookup table may include individual values ​​or indications for each neural network in driving conditions such as highway driving, night driving, residential driving, parking, and other appropriate driving scenarios or contexts. In some embodiments, the processing system may dynamically change the priority order of neural networks based on a determination of the driving scenario or context. In some embodiments, the processing system may associate the dynamically changed priority order of neural networks with the relative importance of the neural networks relative to the determined driving scenario or context.

[0039] In some embodiments, the processing system can dynamically assign priorities to neural networks. In some embodiments, the processing system can assign priorities to neural networks based on the outputs or inferences provided by the neural networks. In some embodiments, the priorities assigned to neural networks can be determined as a function of detected objects, distance to objects, object category or type, and / or other inputs or outputs of the neural network. By assigning priorities to neural networks in this way, neural networks that provide more meaningful or accurate inferences can be assigned relatively higher priorities.

[0040] In some embodiments, the processing system may determine the overall resource budget that can be allocated to a plurality of neural networks. In some embodiments, one or more resources may include power, thermal power envelope, or another suitable resource.

[0041] In some embodiments, the processing system may adjust one or more hyperparameters of one or more neural networks among a plurality of neural networks based on a determined priority order of one or more neural networks. In some embodiments, the processing system may dynamically adjust the hyperparameters of each neural network based on a dynamic priority order of each neural network within the overall resource budget. Non-limiting examples of hyperparameters that the processing system may adjust include pruning level, parameter quantization, inference per second, neural network configuration, input resolution, batch size, field of view, and context-mapped partial masking of the input image. Other examples of hyperparameters that the processing system may adjust are also possible. When allocated resources decrease, hyperparameters with a smaller impact on security performance are selected for adjustment. When allocated resources increase, hyperparameters with a larger impact on security performance are selected for adjustment.

[0042] In some embodiments, the processing system may determine the relative performance of the neural network as a function of accuracy and output, such as the number of inferences per second or the number of output frames per second. In some embodiments, the relative performance of the neural network may be expressed as: Where Accuracy represents the number of accurate successes of inferences, frames, or other outputs; Output FPS represents the number of inferences per second of the neural network's outputs; Accuracy (best possible) represents the target or theoretical maximum accuracy of the neural network; and Target FPS represents the target or theoretical maximum inference per second of the neural network's outputs.

[0043] In some embodiments, the processing system may determine the effectiveness of the neural network based on whether the neural network performs functions that are relatively important (e.g., relatively high priority) or relatively unimportant (e.g., relatively low priority) to safe vehicle operation. In some embodiments, the effectiveness of relatively high priority functions (HPF) can be expressed as: Here, "# functions" represents the output quantity, such as the number of inferences per second or the number of output frames per second; and "rank" represents the priority assigned to the neural network.

[0044] In some embodiments, the effectiveness of a relatively low priority function (LPF) can be expressed as:

[0045] In some embodiments, the processing system may adjust one or more hyperparameters of each neural network based on the performance-effectiveness curves of one or more neural networks among a plurality of neural networks. In some embodiments, the performance-effectiveness curve may include a measure of the impact of adjusting the hyperparameters on the performance of each neural network. In some embodiments, each performance-effectiveness curve may be based on an assessment of the effectiveness and relative performance of each neural network. In some embodiments, the processing system may adjust one or more hyperparameters based on the performance-effectiveness curves of the neural networks.

[0046] In some embodiments, the processing system may determine an overall estimate or evaluation of the performance of each neural network. In some embodiments, this overall evaluation may be referred to as a "Quality of Driving" (QoD) metric, which may be expressed as:

[0047] In some embodiments, based on the estimated QoD metric of the vehicle and the estimated consumption of budget resources, the processing system can iteratively adjust the hyperparameters of a plurality of neural networks. In some embodiments, QoD can be computed from a model using inference accuracy and neural network speed as input values.

[0048] In some embodiments, QoD can provide an indication of ride quality based on factors that are perceptible to human passengers and may influence their ride experience in the vehicle. Factors perceptible to human passengers (such as vertical and lateral accelerations felt by the passenger and external conditions visible to the passenger (e.g., distance to other vehicles)) can be determined based on sensor outputs (e.g., accelerometers, lidar, cameras, speedometers, etc.) and analyzed by a processing system against various thresholds and / or user settings. However, some vehicle movements or behaviors within the safe operating capabilities of an autonomous vehicle may cause discomfort or distress to passengers. Given the capabilities of an autonomous vehicle, while high-speed cornering, sudden maneuvers, rapid braking, rapid acceleration, and following closely behind another vehicle (e.g., a "trailer") may be perfectly safe, such movements and external conditions may be perceived as dangerous or uncomfortable by human passengers and / or may induce motion sickness, potentially reducing the ride quality experienced by the passenger. For example, high-speed cornering and sudden maneuvers may frighten human passengers and / or induce motion sickness. As another example, following closely behind another vehicle or sudden braking may make passengers feel unsafe. Therefore, in some embodiments, the processing system may be configured with one or more thresholds relating to one or more vehicle movements or behaviors or factors perceptible to human passengers, wherein such thresholds are set to improve passenger experience. In these embodiments, in response to determining that one or more vehicle movements, behaviors, or other factors perceptible to passengers exceed thresholds established or set for improving passenger experience, the processing system may determine that QoD is inappropriate, undesirable, or unacceptable to human passengers. Therefore, in some embodiments, the processing system may determine overall vehicle safety performance at least in part based on (or considering) whether one or more passenger-perceptible factors exceed thresholds established or set for the purpose of improving passenger experience.

[0049] In some embodiments, the processing system may determine the performance of one or more neural networks among a plurality of neural networks using allocated computing resources, and may reallocate computing resources to one or more neural networks among the plurality of neural networks based on the determined performance of one or more neural networks among the plurality of neural networks. In some embodiments, reallocating computing resources to one or more neural networks among the plurality of neural networks based on the determined performance of one or more neural networks among the plurality of neural networks may include: readjusting one or more hyperparameters of one or more neural networks among the plurality of neural networks based on the determined performance of one or more (e.g., each) neural networks among the plurality of neural networks.

[0050] In some embodiments, the processing system may be configured to attempt to maximize a quality of performance (QoD) metric. In some embodiments, the processing system may determine the QoD metric relative to the thermal power envelope of the processing system. In some embodiments, the processing system may determine the current QoD metric based on the thermal power envelope of the processing system and the relative importance of each neural network (e.g., the rank or priority assigned to each neural network). Subsequently, the processing system may determine an estimated power consumption of the processing system.

[0051] In response to the determination that the currently estimated processing system power consumption is greater than the thermal power envelope, the processing system may adjust one or more hyperparameters of one or more neural networks to reduce the processing system power consumption. In response to the determination that the currently estimated processing system power consumption is less than the thermal power envelope, the processing system may adjust one or more hyperparameters of one or more neural networks to allow for increased power consumption to improve performance. In some embodiments, adjusting one or more hyperparameters of one or more neural networks to allow for increased power consumption may increase the QoD metric. In some embodiments, when operating conditions permit, the processing system may increase the processing resources allocated to one or more neural networks by readjusting the hyperparameters of the neural networks (such as setting the hyperparameters back to preset or normal settings).

[0052] In some embodiments, the processor can determine whether available computing resources have increased, decreased, or remained unchanged. In response to a determination that available computing resources have increased, the processor can adjust one or more neural network hyperparameters that have a relatively large impact on overall vehicle safety performance. In response to a determination that available computing resources have decreased, the processor can adjust one or more neural network hyperparameters that have a relatively small impact on overall vehicle safety performance.

[0053] Various embodiments improve the functionality of a vehicle system by providing at least some computational resources to neural networks critical to the safe operation of the vehicle, even when overall computational resources are reduced. Various embodiments also improve vehicle safety by dynamically allocating limited computational resources to a plurality of vehicle neural networks based on the importance of each vehicle neural network to safe vehicle operation in a particular vehicle operating situation or context.

[0054] Various embodiments can be implemented in various vehicles, with an example vehicle 100 illustrated in Figures 1A and 1B. Referring to Figures 1A and 1B, vehicle 100 may include a control unit 140 and a plurality of sensors 102-138, including a satellite geolocation system receiver 108, occupant sensors 112, 116, 118, 126, 128, tire pressure sensors 114, 120, cameras 122, 136, microphones 124, 134, a collision sensor 130, radar 132, and laser radar 138. The plurality of sensors 102-138 disposed in or on the vehicle can be used for various purposes (such as autonomous and semi-autonomous navigation and control, collision avoidance, position determination, etc.) and to provide sensor data about objects and people in or on the vehicle 100. Sensors 102-138 may include one or more of a variety of sensors capable of detecting various information useful for navigation and collision avoidance. Each of sensors 102-138 can communicate with control unit 140 and with each other via wired or wireless means. Specifically, sensors may include one or more cameras 122, 136 or other optical sensors or photographic optical sensors. Sensors may also include other types of object detection and ranging sensors, such as radar 132, laser radar 138, IR sensors and ultrasonic sensors. Sensors may also include tire pressure sensors 114, 120, humidity sensors, temperature sensors, satellite geolocation sensors 108, accelerometers, vibration sensors, gyroscopes, gravimeters, impact sensors 130, force gauges, stress gauges, strain sensors, fluid sensors, chemical sensors, gas content analyzers, pH sensors, radiation sensors, Geiger counters, neutron detectors, biomaterial sensors, microphones 124, 134, occupant sensors 112, 116, 118, 126, 128, proximity sensors, and other sensors.

[0055] The vehicle control unit 140 may be configured to have processor-executable instructions to perform various embodiments using information received from various sensors (specifically, cameras 122, 136). In some embodiments, the control unit 140 may supplement the processing of camera images with distance and relative position (e.g., relative azimuth) obtainable from the radar 132 and / or lidar 138 sensors. The control unit 140 may also be configured to control the steering, braking, and speed of the vehicle 100 when operating in autonomous or semi-autonomous mode, using information about other vehicles determined using the various embodiments.

[0056] Figure 1C is a block diagram illustrating components and support systems suitable for implementing various embodiments of the system 150. Referring to Figures 1A, 1B, and 1C, the vehicle 100 may include a control unit 140, which may include various circuitry and devices for controlling the operation of the vehicle 100. In the example shown in Figure 1C, the control unit 140 includes a processor 164, a memory 166, an input module 168, an output module 170, and a radio module 172. The control unit 140 may be coupled to and configured to control a drive control unit 154, a navigation unit 156, and one or more sensors 158 of the vehicle 100.

[0057] Control unit 140 may include processor 164, which may be configured with processor-executable instructions to control the handling, navigation, and / or other operations of vehicle 100, including operations according to various embodiments. Processor 164 may be coupled to memory 166. Control unit 162 may include input module 168, output module 170, and radio module 172.

[0058] Radio module 172 can be configured for wireless communication. Radio module 172 can exchange signals 182 with network transceiver 180 (e.g., command signals for control operation, signals from navigation facilities, etc.), and can provide signals 182 to processor 164 and / or navigation unit 156. In some embodiments, radio module 172 can enable vehicle 100 to communicate with wireless communication device 190 via wireless communication link 192. Wireless communication link 192 can be a bidirectional or unidirectional communication link, and can use one or more communication protocols.

[0059] The input module 168 can receive sensor data from one or more vehicle sensors 158 and electronic signals from other components, including the drive control unit 154 and the navigation unit 156. The output module 170 can be used to communicate with or activate various components of the vehicle 100, including the drive control unit 154, the navigation unit 156, and the sensors 158.

[0060] Control unit 140 may be coupled to drive control unit 154 to control physical components of vehicle 100 related to vehicle handling and navigation, such as engine, electric motor, throttle, steering components, flight control components, braking or deceleration components, etc. Drive control unit 154 may also include components that control other equipment of the vehicle, including environmental controls (e.g., air conditioning and heating), exterior and / or interior lighting, interior and / or exterior information displays (which may include display screens or other devices for displaying information), safety devices (e.g., haptic devices, audible alarms, etc.) and other similar devices.

[0061] Control unit 140 may be coupled to navigation component 156 and may receive data from navigation component 156 and be configured to use that data to determine the current position and orientation of vehicle 100, as well as a suitable route to its destination. In various embodiments, navigation component 156 may include or be coupled to a Global Navigation Satellite System (GNSS) receiver system (e.g., one or more Global Positioning System (GPS) receivers), enabling vehicle 100 to use GNSS signals to determine its current position. Alternatively or additionally, navigation component 156 may include a radio navigation receiver for receiving navigation beacons or other signals from radio nodes (such as Wi-Fi access points, cellular network sites, radio stations, remote computing devices, other vehicles, etc.). Processor 164 may control the navigation and maneuvering of vehicle 100 via drive control component 154. The processor 164 and / or navigation component 156 can be configured to communicate with the server 184 on a network 186 (e.g., the Internet) using a wireless connection 182 to the cellular data network 180 to receive commands for controlling operations, receiving data useful for navigation, providing real-time location reports, and evaluating other data.

[0062] The control unit 162 may be coupled to one or more sensors 158. The sensors 158 may include sensors 102-138 as described, and may be configured to provide various data to the processor 164.

[0063] Although the control unit 140 is described as comprising separate components, in some embodiments, some or all of the components (e.g., processor 164, memory 166, input module 168, output module 170, and radio module 172) may be integrated into a single device or module (such as a system-on-a-chip (SOC) processing device). Such an SOC processing device may be configured for use in a vehicle and is configured, for example, to have processor-executable instructions that execute in processor 164 to perform the operations of the various embodiments when installed in a vehicle.

[0064] Figure 2A illustrates examples of vehicle applications, subsystems, computing components, or units within a vehicle management system 200 that may be used within vehicle 100. Referring to Figures 1A-2A, in some embodiments, various vehicle applications, computing components, or units within the vehicle management system 200 may be implemented within a system of interconnected computing devices (i.e., subsystems) that transmit data and commands to each other (e.g., indicated by arrows in Figure 2A). In other embodiments, the vehicle management system 200 may be implemented as a plurality of vehicle applications executing within a single computing device, such as individual threads, programs, algorithms, or computing components. However, the use of the term "vehicle application" in describing various embodiments is not intended to imply or require that the corresponding functionality be implemented within a single autonomous (or semi-autonomous) vehicle management system computing device, although this is a potential implementation embodiment. More precisely, the use of the term "vehicle application" is intended to encompass subsystems with independent processors, computing components (e.g., threads, algorithms, subroutines, etc.) executing in one or more computing devices, and combinations of subsystems and computing components.

[0065] In various embodiments, vehicle applications executing in the vehicle management system stack 200 may include (but are not limited to) radar perception application 202, camera perception application 204, localization engine application 206, map fusion and arbitration application 208, route planning application 210, sensor fusion and road world model (RWM) management application 212, motion planning and control application 214, and behavior planning and prediction application 216. Vehicle applications 202-216 are merely examples of some vehicle applications in one instance configuration of the vehicle management system stack 200. In other configurations consistent with the various embodiments, other vehicle applications may be included, such as additional vehicle applications for other perception sensors (e.g., LiDAR perception layers, etc.), additional vehicle applications for planning and / or control, additional vehicle applications for modeling, etc., and / or some of the vehicle applications 202-216 may be removed from the vehicle management system stack 200. As shown by the arrows in Figure 2A, each of the vehicle applications 202-216 can exchange data, calculation results, and commands.

[0066] The vehicle management system stack 200 can receive and process data from sensors (e.g., radar, lidar, cameras, inertial measurement units (IMUs), etc.), navigation systems (e.g., GPS receivers, IMUs, etc.), vehicle networks (e.g., Controller Area Network (CAN) buses), and databases in memory (e.g., digital map data). The vehicle management system stack 200 can output vehicle control commands or signals to a drive-by-wire (DBW) system / control unit 220, which is a system, subsystem, or computing device that directly interfaces with vehicle steering, throttle, and braking controls. The configuration of the vehicle management system stack 200 and DBW system / control unit 220 shown in Figure 2A is merely an example configuration, and other configurations of the vehicle management system and other vehicle components can be used in various embodiments. As an example, the configuration of the vehicle management system stack 200 and DBW system / control unit 220 shown in Figure 2A can be used in vehicles configured for autonomous or semi-autonomous operation, while different configurations can be used in non-autonomous vehicles.

[0067] Radar-aware vehicle application 202 may receive data from one or more detection and ranging sensors (such as radar (e.g., 132) and / or lidar (e.g., 138) and process the data to identify and determine the positions of other vehicles and objects in the vicinity of vehicle 100. Radar-aware vehicle application 202 may include using neural network processing and artificial intelligence methods to identify objects and vehicles and pass such information to sensor fusion and RWM management vehicle application 212.

[0068] Camera-aware vehicle application 204 may receive data from one or more cameras (such as cameras (e.g., 122, 136)) and process the data to identify and determine the positions of other vehicles and objects in the vicinity of vehicle 100. Camera-aware vehicle application 204 may include using neural network processing and artificial intelligence methods to identify objects and vehicles and pass such information to sensor fusion and RWM management vehicle application 212.

[0069] The positioning engine vehicle application 206 can receive and process data from various sensors to determine the position of vehicle 100. These various sensors may include, but are not limited to, GPS sensors, IMUs, and / or other sensors connected via a CAN bus. The positioning engine vehicle application 206 may also utilize input from one or more cameras (such as cameras (e.g., 122, 136)) and / or any other available sensors (such as radar, LiDAR, etc.).

[0070] The map fusion and arbitration vehicle application 208 can access data in a high-definition (HD) map database and receive output from the positioning engine vehicle application 206, and process the data to further determine the position of vehicle 100 within the map, such as its position within a traffic lane, its position within a street map, etc. The HD map database can be stored in memory (e.g., memory 166). For example, the map fusion and arbitration vehicle application 208 can convert latitude and longitude information from GPS into a position within a road surface map contained in the HD map database. GPS position locking includes errors, so the map fusion and arbitration vehicle application 208 can be used to arbitrate between GPS coordinates and HD map data to determine the best guessed position of vehicle 100 within the road. For example, although GPS coordinates might place vehicle 100 near the middle of a two-lane road in the HD map, the map fusion and arbitration vehicle application 208 can determine from the direction of travel that vehicle 100 is most likely aligned with the lane traveling in the same direction. The map fusion and arbitration vehicle application 208 can then pass the map-based position information to the sensor fusion and RWM management vehicle application 212.

[0071] Route planning vehicle application 210 can use an HD map and input from an operator or scheduler to plan a route for vehicle 100 to a specific destination. Route planning vehicle application 210 can pass map-based location information to sensor fusion and RWM management vehicle application 212. However, it is not necessary for other vehicle applications (such as sensor fusion and RWM management vehicle application 212) to use the previous map. For example, other stacks can operate and / or control the vehicle solely based on perception data without providing a map, constructing lanes, or defining boundaries, and receive a local map as the perception data concept.

[0072] The sensor fusion and RWM management vehicle application 212 can receive data and outputs generated by one or more of the radar sensing vehicle application 202, camera sensing vehicle application 204, map fusion and arbitration vehicle application 208, and route planning vehicle application 210, and use some or all of these inputs to estimate or refine the position and state of vehicle 100 relative to the road, other vehicles on the road, and other objects near vehicle 100. For example, the sensor fusion and RWM management vehicle application 212 can combine image data from camera sensing vehicle application 204 with arbitration map position information from map fusion and arbitration vehicle application 208 to refine the determined position of the vehicle within a traffic lane. As another example, the sensor fusion and RWM management vehicle application 212 can combine object identification and image data from camera sensing vehicle application 204 with object detection and ranging data from radar sensing vehicle application 202 to determine and refine the relative positions of other vehicles and objects near the vehicle. As another example, the sensor fusion and RWM management vehicle application 212 can receive information about the position and direction of travel of other vehicles from vehicle-to-vehicle (V2V) communication (such as via a CAN bus) and combine this information with information from radar-sensing vehicle application 202 and camera-sensing vehicle application 204 to refine the position and motion of other vehicles. The sensor fusion and RWM management vehicle application 212 can output the refined position and status information of vehicle 100, as well as the refined position and status information of other vehicles and objects in the vicinity of that vehicle, to motion planning and control vehicle application 214 and / or behavior planning and prediction vehicle application 216.

[0073] As another example, sensor fusion and RWM management vehicle application 212 can use dynamic traffic control commands that instruct vehicle 100 to change speed, lane, direction of travel, or other navigation elements, and combine this information with other received information to determine refined location and status information. Sensor fusion and RWM management vehicle application 212 can output the refined location and status information of vehicle 100, as well as the refined location and status information of other vehicles and objects near vehicle 100, to motion planning and control vehicle application 214, behavior planning and prediction vehicle application 216, and / or devices remote from vehicle 100 (such as data servers, other vehicles, etc.) via wireless communication (such as via C-V2X connection, other wireless connection, etc.).

[0074] As another example, the sensor fusion and RWM management vehicle application 212 can monitor perception data from various sensors (such as perception data from radar-sensing vehicle application 202, camera-sensing vehicle application 204, other sensing vehicle applications, etc., and / or data from one or more sensors themselves) to analyze the status in the vehicle sensor data. The sensor fusion and RWM management vehicle application 212 can be configured to detect status in the sensor data (such as sensor measurements being at, above, or below thresholds, the occurrence of certain types of sensor measurements, etc.) and can output the sensor data as part of refined location and status information of vehicle 100 via wireless communication (such as via C-V2X connection, other wireless connections, etc.). This refined location and status information of vehicle 100 is provided to behavior planning and prediction vehicle application 216 and / or devices remote to vehicle 100 (such as data servers, other vehicles, etc.).

[0075] The refined location and status information may include vehicle descriptors associated with vehicle 100 and the vehicle owner and / or operator, such as: vehicle specifications (e.g., size, weight, color, type of onboard sensors, etc.); vehicle location, speed, acceleration, direction of travel, attitude, orientation, destination, fuel / power level, and other status information; vehicle emergency status (e.g., whether the vehicle is an emergency vehicle or a private individual in an emergency); vehicle restrictions (e.g., heavy / wide load, turning restrictions, high-occupancy vehicle (HOV) authorization, etc.); vehicle capabilities (e.g., all-wheel drive, four-wheel drive, snow tires, chains, supported connection types, onboard sensor execution status, onboard sensor resolution level, etc.); equipment problems (e.g., low tire pressure, weak braking, sensor power failure, etc.); owner / operator driving preferences (e.g., preferred lanes, roads, routes and / or destinations, preferences to avoid toll booths or highways, preferences for the fastest route, etc.); permission to provide sensor data to the data agent server (e.g., 184); and / or owner / operator identity information.

[0076] The autonomous vehicle system stack vehicle application 200's behavior planning and prediction vehicle application 216 can use refined position and state information of vehicle 100, output from sensor fusion and RWM management vehicle application 212, as well as position and state information of other vehicles and objects, to predict the future behavior of other vehicles and / or objects. For example, behavior planning and prediction vehicle application 216 can use such information to predict the future relative positions of other vehicles based on its own vehicle position and speed, as well as the positions and speeds of other vehicles nearby. This prediction can take into account information from HD maps and route planning to anticipate changes in relative vehicle positions as the host and other vehicles follow the road. Behavior planning and prediction vehicle application 216 can output predictions of other vehicle and object behavior and positions to motion planning and control vehicle application 214.

[0077] Furthermore, the behavior planning and predictive vehicle application 216 can combine object behavior with position prediction to plan and generate control signals for controlling the motion of vehicle 100. For example, based on route planning information, refined position in road information, and the relative positions and motions of other vehicles, the behavior planning and predictive vehicle application 216 can determine whether vehicle 100 needs to change lanes and accelerate, for example, to maintain or achieve a minimum distance from other vehicles, and / or prepare to turn or leave. As a result, the behavior planning and predictive vehicle application 216 can calculate or otherwise determine changes in wheel steering angles and throttle settings that will be commanded to the motion planning and control vehicle application 214 and the DBW system / control vehicle application 220, as well as various parameters necessary to achieve such lane changes and accelerations. One such parameter could be the calculated steering wheel command angle.

[0078] The motion planning and control vehicle application 214 can receive data and information output from the sensor fusion and RWM management vehicle application 212, as well as other vehicle and object behaviors and position predictions from the behavior planning and prediction vehicle application 216, and use this information to plan and generate control signals for controlling the motion of vehicle 100 and verify that such control signals meet the safety requirements for vehicle 100. For example, based on route planning information, refined positions in road information, and the relative positions and movements of other vehicles, the motion planning and control vehicle application 214 can verify various control commands or instructions and pass them to the DBW system / control vehicle application 220.

[0079] The DBW system / control unit 220 can receive commands or instructions from the motion planning and vehicle control application 214 and convert such information into mechanical control signals for controlling the wheel angles, braking, and throttle of the vehicle 100. For example, the DBW system / vehicle control application 220 can respond to the calculated steering wheel command angle by sending a corresponding control signal to the steering wheel controller.

[0080] In various embodiments, the vehicle management system stack 200 may include functions that perform safety checks or supervise various commands, plans, or other decisions by individual vehicle applications that may affect vehicle and occupant safety. Such safety check or supervision functions may be implemented within a dedicated vehicle application or distributed across various vehicle applications and included as part of the functionality. In some embodiments, various safety parameters may be stored in memory, and the safety check or supervision functions may compare determined values ​​(e.g., relative distance to nearby vehicles, distance from the road centerline, etc.) with the corresponding safety parameters and issue warnings or commands when safety parameters are violated or are about to be violated. For example, a safety or supervisory function in behavior planning and prediction vehicle application 216 (or a separate vehicle application) can determine the current or future individual distance between another vehicle (refined by sensor fusion and RWM management vehicle application 212) and vehicle 100 (e.g., based on a world model refined by sensor fusion and RWM management vehicle application 212), compare this distance with a safe distance parameter stored in memory, and if the current or predicted distance violates the safe distance parameter, issue a command to motion planning and control vehicle application 214 to accelerate, decelerate, or steer. As another example, a safety or supervisory function in motion planning and control vehicle application 214 (or a separate vehicle application) can compare a determined or commanded steering wheel command angle with a safe wheel angle limit or parameter, and issue an overwrite command and / or alarm in response to the commanded angle exceeding the safe wheel angle limit.

[0081] Some safety parameters stored in memory can be static (i.e., do not change over time), such as maximum speed. Other safety parameters stored in memory can be dynamic, as these parameters are continuously or periodically determined or updated based on vehicle status information and / or environmental conditions. Non-limiting examples of safety parameters include maximum safe speed, maximum braking pressure, maximum acceleration, and safe wheel angle limits, all of which can be functions of road and weather conditions.

[0082] Figure 2B illustrates examples of vehicle applications, subsystems, computing components, or units within a vehicle management system 250 that can be used within vehicle 100. Referring to Figures 1A-2B, in some embodiments, vehicle applications 202, 204, 206, 208, 210, 212, and 216 of the vehicle management system stack 200 can be similar to those layers described with reference to Figure 2A, and the vehicle management system stack 250 can operate similarly to the vehicle management system stack 200, except that the vehicle management system stack 250 can pass various data or instructions to the vehicle safety and collision avoidance system 252 instead of the DBW system / control vehicle application 220. For example, the configuration of the vehicle management system stack 250 and the vehicle safety and collision avoidance system vehicle application 252 shown in Figure 2B can be used in non-autonomous vehicles.

[0083] In various embodiments, the behavior planning and prediction vehicle application 216 and / or the sensor fusion and RWM management vehicle application 212 can output data to the vehicle safety and collision avoidance system vehicle application 252. For example, the sensor fusion and RWM management vehicle application 212 can output sensor data as part of the refined position and state information of vehicle 100 provided to the vehicle safety and collision avoidance system vehicle application 252. The vehicle safety and collision avoidance system vehicle application 252 can use the refined position and state information of vehicle 100 to make safety decisions regarding vehicle 100 and / or vehicle 100's occupants. As another example, the behavior planning and prediction vehicle application 216 can output behavior models and / or predictions related to the motion of other vehicles to the vehicle safety and collision avoidance system 252. The vehicle safety and collision avoidance system vehicle application 252 can use behavior models and / or predictions related to the motion of other vehicles to make safety decisions related to vehicle 100 and / or vehicle 100's occupants.

[0084] In various embodiments, the vehicle safety and collision avoidance system vehicle application 252 may include the following functions: performing safety checks or monitoring various commands, plans, or other decisions by various vehicle applications that may affect vehicle and occupant safety, as well as human driver actions. In some embodiments, various safety parameters may be stored in memory, and the vehicle safety and collision avoidance system vehicle application 252 may compare determined values ​​(e.g., relative distance to nearby vehicles, distance to the road centerline, etc.) with corresponding safety parameters and issue warnings or commands when safety parameters are violated or will be violated. For example, the vehicle safety and collision avoidance system vehicle application 252 may perform the following operations: determine the current or future distance between this vehicle and another vehicle (refined by sensor fusion and RWM management vehicle application 212) (e.g., based on a world model refined by sensor fusion and RWM management vehicle application 212); compare this distance with a safe distance parameter stored in memory; and if the current or predicted distance violates the safe distance parameter, issue a command to the driver to accelerate, decelerate, or turn. As another example, a vehicle safety and collision avoidance system vehicle application 252 can compare changes in steering wheel angle by a human driver with safe wheel angle limits or parameters, and issue an overwrite command and / or alarm in response to the steering wheel angle exceeding the safe wheel angle limit.

[0085] Figure 3 illustrates an example SOC architecture suitable for implementing various embodiments of the processing device system-on-a-chip (SOC) 300 in a vehicle. Referring to Figures 1A-3, the processing device SOC 300 may include various processing resources, including multiple heterogeneous processors such as a digital signal processor (DSP) 303, a modem processor 304, an image and object recognition processor 306, a mobile display processor 307, an application processor 308, and a resource and power management (RPM) processor 317. The processing device SOC 300 may also include one or more auxiliary processors 310 (e.g., vector auxiliary processors) connected to one or more of the heterogeneous processors 303, 304, 306, 307, 308, and 317. Each of these processors may include one or more cores and an independent / internal clock. Each processor / core may operate independently of the other processors / cores. For example, the processing device SOC 300 may include a processor executing a first type of operating system (e.g., FreeBSD, LINUX, OS X, etc.) and a processor executing a second type of operating system (e.g., Microsoft Windows). In some embodiments, the application processor 308 may be the main processor, central processing unit (CPU), microprocessor unit (MPU), arithmetic logic unit (ALU), etc. of the SOC 300. The graphics processor 306 may be a graphics processing unit (GPU).

[0086] The processing device SOC 300 may include analog circuitry and custom circuitry 314 for managing sensor data, analog-to-digital conversion, wireless data transmission, and performing other specialized operations, such as processing encoded audio and video signals for presentation in a web browser. The processing device SOC 300 may also include system components and other subsystems 316, such as voltage regulators, oscillators, phase-locked loops, peripheral bridges, data controllers, memory controllers, system controllers, access ports, timers, and other similar components for supporting processors and software user clients (e.g., web browsers) running on a computing device.

[0087] The processing device SOC 300 also includes dedicated circuitry for camera actuation and management (CAM) 305, which includes, provides, controls, and / or manages the operation of one or more cameras 122, 136 (e.g., main camera, network camera, 3D camera, etc.), video display data from camera firmware, image processing, video preprocessing, video front-end (VFE), embedded JPEG, high-definition video transcoder, etc. CAM 305 may be a separate processing unit and / or include a separate or internal clock.

[0088] In some embodiments, the image and object recognition processor 306 may be configured with processor-executable instructions and / or dedicated hardware to perform image processing and object recognition analysis as described in various embodiments. For example, the image and object recognition processor 306 may be configured to perform operations that process images received from cameras (e.g., 122, 136) via CAM 305 to identify and / or identify other vehicles, and otherwise perform the functions of the camera-aware vehicle application 204 as described. In some embodiments, the processor 306 may be configured to process radar or lidar data and perform the functions of the radar-aware vehicle application 202 as described.

[0089] System components and other subsystems 316, analog and custom circuitry 314, and / or the CAM 305 may include circuitry for interfacing with peripheral devices such as cameras 122, 136, radar 132, laser radar 138, electronic displays, wireless communication devices, external memory chips, etc. Processors 303, 304, 306, 307, and 308 may be interconnected via interconnect / bus modules 324 to one or more memory components 312, system components and other subsystems 316, analog and custom circuitry 314, the CAM 305, and the RPM processor 317. The interconnect / bus modules 324 may include reconfigurable gate arrays and / or implement bus architectures (e.g., CoreConnect, AMBA, etc.). Communication may be provided by advanced interconnects, such as high-performance on-chip networks (NoC).

[0090] The processing device SOC 300 may also include input / output modules (not shown) for communicating with external resources (such as clock 318 and voltage regulator 320). External resources (e.g., clock 318, voltage regulator 320) may be shared by two or more of the internal SOC processors / cores (e.g., DSP 303, modem processor 304, graphics processor 306, application processor 308, etc.).

[0091] In some embodiments, the processing device SOC 300 may be included in a control unit (e.g., 140) for use in a vehicle (e.g., 100). The control unit may include a communication link for communicating with a telephone network (e.g., 180), the Internet, and / or a network server (e.g., 184) as described.

[0092] The processing device SOC 300 may also include additional hardware and / or software components suitable for collecting sensor data from sensors, including motion sensors (e.g., accelerometers and gyroscopes of an IMU), user interface components (e.g., input buttons, touch screen displays, etc.), microphone arrays, sensors for monitoring physical conditions (e.g., position, orientation, motion, direction, vibration, pressure, etc.), cameras, compasses, GPS receivers, communication circuitry (e.g., Bluetooth®, WLAN, WiFi, etc.), and other known components of modern electronic devices.

[0093] Some or all of the components within the SOC 300 described with reference to FIG3 may be computing resources allocated to vehicle applications according to various embodiments. However, allocable computing resources also include (such as voltage (or current) supplied to the SOC 300 and / or specific components therein by voltage regulator 320) and / or (such as operating frequency allocated to the SOC 300 and / or specific components therein by clock 318). For example, the purpose of allocating computing resources may involve reducing (or increasing): the rate of processing operations (e.g., the number of operations performed per second); memory usage; processor operating frequency; processor current; access to auxiliary processors or other components, etc., which are generally and collectively referred to herein as computing resources. Furthermore, as described herein with reference to FIG5A, the reduction of allocated computing resources (e.g., processing operations, memory usage, processor frequency, processor current, etc.) can be achieved by various vehicle applications by adjusting one or more parameters or alternatives that affect application performance or operation. For example, the computational resources for each vehicle application can be configured by adjusting one or more inferences per second, the input image resolution, processing accuracy, output quality, the algorithms used by the application, or the functions of the application.

[0094] Figure 4 is a block diagram illustrating a system 400 configured to allocate computing resources to applications executed by a vehicle's processor according to various embodiments. In some embodiments, system 400 may include one or more vehicle computing devices 402 and / or one or more remote platforms 404. Referring to Figures 1A-4, vehicle computing device 402 may include one or more processors 430, such as processor 164, processing device 300, and / or control unit 140 of the vehicle (e.g., 100) (variously referred to as "processor" 430). Remote platform 404 may include the vehicle's (e.g., 100) processor (e.g., 164), processing device (e.g., 300), and / or control unit (e.g., 140) (variously referred to as "processor").

[0095] The vehicle computing device 402 may include one or more processors 430 configured by machine-executable instructions 406. The machine-executable instructions 406 may include one or more instruction modules. The instruction modules may include computer program modules. The instruction modules may include one or more of a priority determination module 408, a computing resource allocation module 410, and / or other instruction modules.

[0096] The priority determination module 408 can be configured to determine the priority of each neural network based on the contribution of each of the plurality of neural networks executed on the vehicle processing system to the overall vehicle safety performance.

[0097] The computing resource allocation module 410 can be configured to allocate computing resources to each neural network based on the determined priority order of each of the plurality of neural networks.

[0098] Vehicle computing device 402 may include electronic storage device 428, one or more processors 430 and / or other components. Vehicle computing device 402 may include communication lines or ports to enable information exchange with networks and / or other vehicle computing devices. The illustration of vehicle computing device 402 in Figure 4 is not intended to be limiting. Vehicle computing device 402 may include a plurality of hardware, software and / or firmware components that operate together to provide the functions attributed to vehicle computing device 402 herein. For example, vehicle computing device 402 may be implemented by a cloud of vehicle computing devices operating together as vehicle computing device 402.

[0099] Electronic storage device 428 may include non-transitory storage media that stores information electronically. The electronic storage media of electronic storage device 428 may include one or both of a system storage device integrated with (i.e., substantially non-removable) and / or a removable storage device, which is removably connected to vehicle computing device 402 via, for example, a port (e.g., a Universal Serial Bus (USB) port, a FireWire port, etc.) or a drive (e.g., a disk drive, etc.). Electronic storage device 428 may include one or more of the following: optically readable storage media (e.g., optical discs, etc.), magnetically readable storage media (e.g., magnetic tape, hard disks, floppy disk drives, etc.), charge-based storage media (e.g., EEPROM, RAM, etc.), solid-state storage media (e.g., flash memory drives, etc.), and / or other electronically readable storage media. Electronic storage device 428 may include one or more virtual storage resources (e.g., cloud storage devices, virtual private networks, and / or other virtual storage resources). Electronic storage device 428 may store software algorithms, information determined by processor 430, information received from vehicle computing device 402, and / or other information that enables vehicle computing device 402 to perform as described herein.

[0100] Processor 430 may be configured to provide information processing capabilities in vehicle computing device 402. Therefore, processor 430 may include one or more of the following: a digital processor, an analog processor, digital circuitry designed to process information, analog circuitry designed to process information, a state machine, and / or other mechanisms for electronically processing information. Although processor 430 is shown as a single entity in FIG. 4, this is for illustrative purposes only. In some implementations, processor 430 may include a plurality of processing units and / or processor cores. Processing units may be physically located within the same device, or processor 430 may represent the processing functions of multiple devices operating in a coordinated manner. Processor 430 may be configured to execute modules 408, 410, and / or other modules via: software; hardware / firmware; a combination of software, hardware, and / or firmware; and / or other mechanisms for configuring processing capabilities on processor 430. As used herein, the term "module" may refer to any component or group of components that perform the functions belonging to that module. This can include one or more physical processors, processor-readable instructions, circuitry, hardware, storage media, or any other component during the execution of processor-readable instructions.

[0101] It should be understood that although modules 408 and 410 are shown in Figure 4 as implemented within a single processing unit, in an implementation of processor 430 comprising multiple processing units and / or processor cores, one or more of modules 408 and 410 may be implemented remotely with other modules. The description of the functionality provided by the different modules 408 and 410 described below is for illustrative purposes and is not intended to be limiting, as any of modules 408 and 410 may provide more or fewer functionality than described. For example, one or more of modules 408 and 410 may be eliminated, and some or all of their functionality may be provided by other modules 408 and 410. As another example, processor 430 may be configured to execute one or more additional modules that may perform some or all of the functionality attributed to one of modules 408 and 410.

[0102] Figure 5A is a conceptual diagram of components of a vehicle computing system 500 according to various embodiments of a method for allocating processing resources to concurrently executing neural networks. Referring to Figures 1–5A, in some embodiments, the components may be implemented using hardware, software, or a combination of hardware and software, for example, via a processor of a vehicle (e.g., vehicle 100) (e.g., processors 164, 303, 304, 306, 307, 308, 310, 317, 430).

[0103] In some embodiments, the vehicle computing system 500 may execute a number of concurrent neural networks and may dynamically evaluate and change the configuration (such as hyperparameters) of each of the concurrently executed neural networks to increase QoD metrics within the budget of the computing system 500’s limited resources (such as power budget, thermal budget, load budget, etc.).

[0104] In some embodiments, the System-on-Chip (SOC) resource budget controller 502 can receive, detect, or determine the ambient temperature of the vehicle's operating environment. The SOC resource budget controller 502 can determine a resource budget for the SOC (e.g., power budget, thermal budget, load budget, etc.) and can provide the resource budget to the neural network resource budget controller 504. The neural network resource budget controller 504 can determine the hyperparameters of one or more concurrently executing neural networks and provide the determined hyperparameters to the SOC 506.

[0105] The SOC 506 can execute a plurality of neural networks, which may include a plurality of preprocessing operations 508, a plurality of backbones 510, a plurality of heads 512, and a plurality of postprocessing operations 514. Examples of preprocessing operations 508 include image spatial calibration, format conversion, resolution scaling, noise reduction, and other image enhancement algorithms. Examples of postprocessing operations 514 include object motion trajectory tracking, object filtering, priority partitioning, and other algorithms. The backbones 510 may use a spiral neural network to extract features from the input image. The detection heads 512 may use neural networks to generate inferences such as object type (e.g., classification) and object location (e.g., horizontal and vertical coordinates). Based on the operations of the SOC 506, the QoD estimator 516 may determine the estimated QoD metric of the vehicle. Furthermore, based at least in part on the operations of the SOC 506, the SOC resource consumption estimator 518 may determine the resource consumption (e.g., power consumption, heat generation, computational resource consumption, etc.) of the vehicle computing system 500. In some embodiments, the operation of SOC 506 can be based on the operations of a plurality of concurrently executed neural networks. Furthermore, the operations of the plurality of concurrently executed neural networks can be based on the hyperparameters of each neural network. QoD estimator 516 can provide the estimated QoD metric of the vehicle to neural network resource budget controller 504. SOC resource consumption estimator 518 can provide the estimated resource consumption of the vehicle computing system 500 to neural network resource budget controller 504.

[0106] Based on the estimated QoD metric of the vehicle and / or the estimated resource consumption of the vehicle computing system 500, the neural network resource budget controller 504 can determine one or more tuned hyperparameters for one or more neural networks within the SOC resource budget. The neural network resource budget controller 504 can provide one or more tuned hyperparameters to the SOC 506, and the SOC 506 can implement one or more tuned hyperparameters in the applicable neural networks. Examples of tuneable hyperparameters include pruning level, parameter quantization, inference per second, neural network configuration, input resolution, batch size, field of view, context-mapped input image partial masking, and / or other appropriate hyperparameters.

[0107] Figure 5B is a table illustrating the functions 530 that can be performed by the various neural networks. Figure 5C is a table illustrating the configuration 540 of the hyperparameters of the various neural networks. Referring to Figures 1–5C, in some embodiments, the neural networks can be implemented in hardware, software, or a combination of hardware and software, for example, via a processor of a vehicle (e.g., processors 164, 303, 304, 306, 307, 308, 310, 317, 430). Referring to Figure 5B, for example, vehicle ADAS can provide various functions such as dynamic route changes for the vehicle's path or route, weather detection, (e.g., obstacles, people, etc.) rear collision warning, forward collision warning, and blind spot detection, and other functions, or among others. Each of the functions of vehicle ADA can be performed by neural networks that execute concurrently, examples of which include DenseNet, VGG16, RESNET50, MobileNet, Inception, and / or other neural networks. Referring to Figure 5C, each of these neural networks can be configured with various hyperparameters, such as quantization level, pruning level parameters, input resolution, inferences per second, and / or other appropriate parameters or configurations.

[0108] Figure 5D is an example graph and illustration of the performance-effectiveness curves 550 for eight neural networks 552-566. Referring to Figures 1–5D, in some embodiments, the neural networks can be implemented using hardware, software, or a combination of hardware and software, for example, via a processor (e.g., processors 164, 303, 304, 306, 307, 308, 310, 317, 430) of a vehicle (e.g., vehicle 100). The four curves 552, 554, 556, and 558 represent higher-priority neural networks, i.e., neural networks that perform higher-priority functions. For example, neural network 552 can perform forward collision avoidance, neural network 554 can perform lane keeping assist, neural network 556 can perform blind spot detection, and neural network 558 can perform rear collision avoidance. The four curves 560, 562, 564, and 566 represent neural networks with lower priority, i.e., neural networks that perform functions with lower priority. For example, neural network 560 can perform lane change frequency function, neural network 562 can perform dynamic route change frequency function, neural network 564 can perform weather detection function, and neural network 566 can perform vehicle night vision function.

[0109] In various embodiments, horizontal lines 570, 572, and 574 represent an overall assessment of the performance or QoD metric of each of neural networks 552-566 based on the effectiveness and relative performance of each of these neural networks. In some embodiments, QoD metric 570 may represent high performance, optimal performance, or maximum performance of the neural network. In some embodiments, QoD metric 572 may represent intermediate performance of the neural network. In some embodiments, QoD metric 574 may represent the minimum permissible or minimum tolerated performance from each neural network. Arrow 576 illustrates that reducing processor resources (e.g., the operating frequency of CPU, GPU, etc.) reduces the relative performance of neural networks 552-566, and the reduction in relative performance for neural networks 552-558 is significantly different compared to neural networks 560-566.

[0110] As mentioned above, in some embodiments, the QoD metric can be expressed as: Furthermore, it can be based on the estimated QoD metric of the vehicle and the estimated consumption of budgeted resources. In some embodiments, the determination of the effectiveness of the neural network can depend in part on whether the neural network is classified as performing relatively high-priority functions or relatively low-priority functions in relation to the vehicle's safe operating state. In some embodiments, the effectiveness of relatively high-priority functions (HPF) (such as neural networks 552, 554, 556, and 558) can be expressed as: Where "# functions" represents the output quantity, such as the number of inferences per second or the number of output frames per second; "rank" represents the priority assigned to the neural network. In some embodiments, the effectiveness of relatively low priority functions (LPF) (such as neural networks 560, 562, 564, and 566) can be represented as:

[0111] In some embodiments, the processing system may adjust one or more hyperparameters of each neural network based on the performance-effectiveness curves of one or more neural networks among a plurality of neural networks. In some embodiments, the performance-effectiveness curve may include a measure of the impact of adjusting the hyperparameters on the performance of each neural network. In some embodiments, each performance-effectiveness curve may be based on an assessment of the effectiveness and relative performance of each neural network. In some embodiments, the processing system may adjust one or more hyperparameters based on the performance-effectiveness curves of one or more neural networks among a plurality of neural networks.

[0112] Figures 5E, 5F, and 5G are graphical and illustrations of examples of the various effects of adjusting hyperparameters according to various embodiments. Adjusting the hyperparameters of a neural network allows control over the network's power consumption and / or processing time. Changing the hyperparameters can also affect the accuracy and inferences per second of the neural network. For example, Figure 5E illustrates an example of how changes in input resolution (in units of primitives) and neural network pruning levels can affect the number of inferences that can be generated per second. As another example, Figure 5F illustrates an example of how changes in input resolution (in units of primitives) and neural network pruning levels can affect the dynamic power consumption of the processor executing the neural network. As another example, Figure 5G illustrates an example of how changes in input resolution (in units of primitives) and neural network pruning levels can affect the inference accuracy of the neural network. Such changes in power consumption, inference accuracy, and inferences per second in response to changes in pruning levels and input resolution hyperparameters can be modeled based on testing or offline simulations (e.g., during the design phase).

[0113] Figure 5H is a graphical and illustration comparing thermal throttling and the allocation of processing resources to concurrently executing neural networks according to various embodiments. Curve 560 illustrates the effect of thermal throttling on QoD, and curve 562 illustrates the effect of allocating processing resources to concurrently executing neural networks on QoD. Curve 560 illustrates a significant decrease in QoD when the thermal power envelope, measured in watts (W), decreases (such as a decrease in the thermal power budget). Region 560a of curve 560 illustrates that QoD rapidly approaches zero as the thermal power envelope decreases, indicating a significant degradation in QoD. In contrast, curve 562 illustrates that even with a significant decrease in the thermal power envelope (e.g., the thermal power budget), dynamically allocating processing resources to concurrently executing neural networks maintains a relatively high QoD. Specifically, region 562a of curve 562 illustrates a relatively small decrease in QoD as the thermal power envelope decreases.

[0114] In some embodiments, the reduction in QoD shown in region 562a may be imperceptible or minimally perceptible to human passengers. In some embodiments, the thermal power budget envelope (which may be correlated with the power consumption of the SoC (e.g., SoC 506)) may be reduced by 30-50%, with a relatively minimal reduction in QoD. Curve 562 illustrates a scenario where the thermal power envelope is reduced (e.g., a thermally limited scenario), but the QoD is still 3 to 4 times better than the QoD caused by thermal throttling shown in curve 560. In some embodiments, these results may be achieved automatically via operations that allocate processing resources to concurrently executing neural networks, rather than via manual optimization performed during software development.

[0115] Figure 6A is a flowchart illustrating the operation of a method 600a, which can be executed by a vehicle's processor according to various embodiments, for allocating processing resources to concurrently executing neural networks. Referring to Figures 1-6A, the operation of method 600a can be executed by a processor (e.g., processors 164, 303, 304, 306, 307, 308, 310, 317, 430) of a vehicle (e.g., vehicle 100).

[0116] In block 602, the processor may determine the priority of one or more neural networks (e.g., each) based on their contribution to the overall vehicle safety performance. In some embodiments, the processor may determine the priority of one or more neural networks (e.g., each) based on their contribution to the overall vehicle safety performance in the context of vehicle operation. In some embodiments, the processor may determine the priority of one or more neural networks (e.g., each) based on indications provided by one or more neural networks (e.g., each) regarding their contribution to the overall vehicle safety performance. In some embodiments, the processor may determine the relative performance of one or more neural networks based on their output inference per second and output accuracy. The unit used to perform the operation of block 602 may include processors 164, 303, 304, 306, 307, 308, 310, 317, and 430.

[0117] In block 604, the processor may allocate computational resources to one or more neural networks (e.g., each) based on a determined priority order of one or more neural networks. In some embodiments, allocating computational resources to the plurality of neural networks based on the determined priority order of one or more neural networks (e.g., each) may include adjusting one or more hyperparameters of one or more neural networks based on the determined priority order of one or more neural networks. In some embodiments, the processor may adjust one or more hyperparameters based on the performance-efficiency curves of one or more neural networks.

[0118] The unit used to perform the operation of block 604 may include processors 164, 303, 304, 306, 307, 308, 310, 317, and 430.

[0119] In some embodiments, the processor may repeatedly perform the operations of block 602 within block 604. In this way, the processor can dynamically allocate processing resources to a plurality of concurrently executing neural networks.

[0120] Figures 6B and 6C are flowcharts illustrating operations 600b and 600c, which can be executed by a vehicle's processor as part of a method 600a for allocating processing resources to concurrently executing neural networks, according to various embodiments. Referring to Figures 1-6CB, operations 600b and 600c can be executed by a processor of the vehicle (e.g., processors 164, 303, 304, 306, 307, 308, 310, 317, 430).

[0121] Referring to FIG6B, in some embodiments, after performing the operation of block 604 of method 600a (FIG. 6A), in block 610, the processor may determine the performance of one or more neural networks among a plurality of neural networks using allocated computing resources. In some embodiments, the processor may determine the effectiveness of one or more of these neural networks. In some embodiments, the processor may determine the actual use of computing resources based on the computing resources used by each neural network (e.g., as a percentage, proportion, or another metric). The unit for performing the operation of block 610 may include processors 164, 303, 304, 306, 307, 308, 310, 317, and 430.

[0122] In block 612, the processor can monitor the dynamic availability and actual usage of computing resources. In some embodiments, the processor can determine the dynamic availability of computing resources based on changes in the demand for computing resources. Such changes in the demand for computing resources can be based on the increase or decrease in the importance of one or more neural networks among a plurality of neural networks. Changes in the importance of one or more neural networks can be based on changes in the vehicle operating context, such as changes in operating state and / or environment. In some embodiments, the processor can determine that the availability of computing resources has increased or decreased. The unit for performing the operations of block 612 may include processors 164, 303, 304, 306, 307, 308, 310, 317, and 430.

[0123] In block 614, the processor can reallocate computing resources to one or more neural networks among a plurality of neural networks based on the performance of one or more of the determined neural networks. In some embodiments, the processor can readjust one or more hyperparameters of one or more neural networks among a plurality of neural networks based on the performance of one or more (e.g., each) of the determined neural networks. The unit for performing the operation of block 614 may include processors 164, 303, 304, 306, 307, 308, 310, 317, and 430.

[0124] Subsequently, the processor can execute the operation of block 602 of method 600a (Figure 6A).

[0125] Referring to Figure 6C, after performing the operation in block 610 (Figure 6B), in decision block 620, the processor can determine whether the available computing resources have been increased, decreased, or remained the same. The unit used to perform the operation in block 620 may include processors 164, 303, 304, 306, 307, 308, 310, 317, and 430.

[0126] In response to the decision to keep the available computing resources the same (i.e., decision block 620 = "same"), in block 622, the processor can maintain the current hyperparameter settings.

[0127] In response to the decision that available computing resources have increased (i.e., decision block 620 = "increase"), in block 624, the processor can adjust hyperparameters that have a significant impact on overall vehicle safety performance.

[0128] In response to the decision that available computing resources have been reduced (i.e., decision block 620 = "reduced"), in block 626, the processor can adjust hyperparameters that have a smaller impact on overall vehicle safety performance.

[0129] In various embodiments, the processor can determine the impact of hyperparameter adjustments on overall vehicle safety performance based on the relative importance of one or more neural networks affected by the hyperparameters to be adjusted.

[0130] After performing the operations of block 622, block 624 or block 626, the processor may perform the operation of block 602 as described in method 600a (Figure 6A).

[0131] Implementation examples are described in the following paragraphs. Although some of the implementation examples below are described in accordance with the example method, other example implementations may include: the example method implemented by a vehicle computing system, as discussed in the following paragraphs, the vehicle computing system including a processor configured with processor-executable instructions to perform the operations of the methods of the following implementation examples; the example method implemented by a vehicle computing system, as discussed in the following paragraphs, the vehicle computing system including units for performing the functions of the methods of the following implementation examples; and the example method discussed in the following paragraphs may be implemented as a non-transitory processor-readable storage medium having processor-executable instructions stored thereon, the processor-executable instructions being configured to cause the processor of the vehicle computing system to perform the operations of the methods of the following implementation examples.

[0132] Example 1: A method executed by a vehicle processor for allocating computing resources to concurrently executing neural networks, comprising: determining a priority order of each neural network based on the contribution of each of the plurality of neural networks executing on the vehicle processing system to the overall vehicle safety performance; and allocating computing resources to the plurality of neural networks based on the determined priority order of each neural network.

[0133] Example 2, according to the method of Example 1, wherein determining the priority order of each neural network based on the contribution of each of the plurality of neural networks executed on the vehicle processing system to the overall vehicle safety performance includes: determining the priority order of each neural network based on the contribution of each of the plurality of neural networks executed on the vehicle processing system to the overall vehicle safety performance in the context of vehicle operation.

[0134] Example 3, according to the method of Example 1 or 2, wherein determining the priority order of each neural network based on the contribution of each of the plurality of neural networks executed on the vehicle processing system to the overall vehicle safety performance includes: determining the priority order of each neural network based on the indication provided by each of the plurality of neural networks executed on the vehicle processing system regarding the contribution of the neural network to the overall vehicle safety performance.

[0135] Example 4: The method according to any one of Examples 1-3, wherein the overall vehicle safety performance is calculated based on a model using the inference accuracy and speed of the plurality of neural networks as input values.

[0136] Example 5: The method according to any one of Examples 1-4, wherein the overall vehicle safety performance indicator is based at least in part on the ride quality of factors perceptible to human passengers.

[0137] Example 6, the method according to any one of Examples 1-5, wherein determining the priority order of each of the plurality of neural networks executed on the vehicle processing system comprises: determining the relative effectiveness of one or more of the plurality of neural networks based on the output inference per second and output accuracy of each neural network.

[0138] Example 7. The method according to any one of Examples 1-6, wherein allocating computational resources to the plurality of neural networks based on the determined priority of each neural network comprises: adjusting one or more hyperparameters of the one or more neural networks among the plurality of neural networks based on the determined priority of one or more neural networks among the plurality of neural networks.

[0139] Example 8, according to the method of Example 7, wherein adjusting one or more hyperparameters of one or more neural networks among the plurality of neural networks based on the determined priority of one or more neural networks includes: adjusting one or more hyperparameters based on the performance-efficiency curves of one or more neural networks among the plurality of neural networks.

[0140] Example 9, the method according to any one of Examples 1-8, also includes: determining the performance of one or more neural networks among the plurality of neural networks using the allocated computing resources; and reallocating computing resources to one or more neural networks among the plurality of neural networks based on the determined performance of one or more neural networks.

[0141] Example 10, according to the method of Example 9, also includes: monitoring the dynamic availability of the computing resource and the actual use of the computing resource, wherein reallocating the computing resource to one or more neural networks among the plurality of neural networks based on the performance of one or more neural networks determined includes: reallocating the computing resource to one or more neural networks among the plurality of neural networks based on the dynamic availability of the computing resource and the actual use of the computing resource.

[0142] Example 11. According to the method of Example 9, wherein reallocating computational resources to one or more neural networks among the plurality of neural networks based on the performance of one or more neural networks determined includes: readjusting one or more hyperparameters of one or more neurons among the plurality of neural networks based on the performance of each of the plurality of neural networks determined.

[0143] Example 12, the method according to any one of Examples 1-11, also includes: determining whether the available computing resources have increased, decreased, or remained the same; and in response to determining that the available computing resources have increased, adjusting one or more neural network hyperparameters that have a relatively large impact on the overall vehicle safety performance.

[0144] Example 13, according to the method of Example 12, also includes: in response to the determination that the available computing resources have been reduced, adjusting one or more neural network hyperparameters that have a relatively small impact on the overall vehicle safety performance.

[0145] The various embodiments shown and described are provided merely as examples to illustrate the various features of the requested item. However, the features shown and described with respect to any given embodiment are not necessarily limited to the associated embodiment and can be used or combined with other embodiments shown and described. Furthermore, the requested item is not intended to be limited to any one of the example embodiments.

[0146] The foregoing method descriptions and flowcharts are provided as illustrative examples only and are not intended to require or imply that the blocks in the various embodiments must be executed in the provided order. As will be understood by those skilled in the art, the order of the blocks in the foregoing embodiments can be executed in any order. Words such as "then," "following," and "next" are not intended to limit the order of the blocks; these words are only used to guide the reader through the description of the method. Furthermore, any reference to a claim element using the articles "a," "an," or "the" in the singular should not be construed as limiting that element to the singular.

[0147] The various illustrative logic blocks, modules, circuits, and algorithm blocks described in conjunction with the embodiments disclosed herein can be implemented as electronic hardware, computer software, or a combination of both. To clearly illustrate this interchangeability between hardware and software, they have been generally described above regarding the functionality of the various illustrative components, blocks, modules, circuits, and units. Whether this functionality is implemented as hardware or software depends on the specific application and the design constraints imposed on the overall system. Those skilled in the art to which this invention pertains can implement the described functionality in varying ways for each specific application; however, such implementation decisions should not be construed as causing a departure from the scope of the various embodiments.

[0148] The hardware used to implement or execute the various illustrative logic units, logic blocks, modules, and circuits described herein can be implemented using a general-purpose processor, digital signal processor (DSP), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA) or other programmable logic device, individual gate or transistor logic, individual hardware component, or any combination thereof designed to perform the functions described herein. The general-purpose processor may be a microprocessor, but alternatively, the processor may be any known processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors combined with a DSP core, or any other such configuration. Alternatively, some blocks or methods may be executed by a circuit system specific to a given function.

[0149] In various embodiments, the described functions can be implemented using hardware, software, firmware, or any combination thereof. If implemented in software, such functions can be stored as one or more instructions or code on a non-transitory computer-readable medium or a non-transitory processor-readable medium. The operation of the methods or algorithms disclosed herein can be embodied in a processor-executable software module, which can reside on a non-transitory computer-readable or processor-readable storage medium. A non-transitory computer-readable or processor-readable storage medium can be any storage medium accessible by a computer or processor. By way of example, and not limitation, such a non-transitory computer-readable or processor-readable medium can include RAM, ROM, EEPROM, flash memory, CD-ROM or other optical disc storage, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store desired program code in the form of instructions or data structures and that can be accessed by a computer. As used herein, magnetic disks and optical disks include CDs, laser discs, optical discs, DVDs, floppy disks, and Blu-ray discs, where magnetic disks typically copy data magnetically, while optical discs use lasers to copy data optically. The above combinations also fall within the scope of non-transitory computer-readable and processor-readable media. Furthermore, the operation of methods or algorithms may reside as code and / or instructions, or any combination or set thereof, on non-transitory processor-readable and / or computer-readable media, which may be incorporated into computer program products.

[0150] The prior description of the disclosed embodiments is provided to enable those skilled in the art to implement or use the embodiments herein. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be applied to other embodiments without departing from the scope of the embodiments. Therefore, the embodiments are not intended to be limited to those shown herein, but are given the broadest scope consistent with the following claims and the principles and novel features disclosed herein.

[0151] 100: Vehicles 102: Sensor 108: Satellite Geographic Positioning System Receiver 112: Occupant Sensor 114: Tire pressure sensor 116: Occupant Sensor 118: Occupant Sensor 120: Tire pressure sensor 122: Camera 124: Microphone 126: Occupant Sensor 128: Occupant Sensor 130: Collision Sensor 132: Radar 134: Microphone 136: Camera 138: Laser Radar 140: Control Unit 150: System 154: Drive control components 156: Navigation Components 158: Sensor 162: Control Unit 164: Processor 166: Memory 168: Input Module 170: Output Module 172: Radio Module 180: Hive Data Network 182: Wireless Connection 184: Server 186: Network 190: Wireless communication equipment 192: Wireless communication link 200: Vehicle Management System 202: Radar Sensing Applications 204: Camera Sensing Applications 206: Location Engine Applications 208: Map Fusion and Arbitration Applications 210: Route Planning Applications 212: Sensor Fusion and Road World Model (RWM) Management Applications 214: Motion Planning and Control Applications 216: Applications of Behavioral Planning and Prediction 220: Drive-by-wire (DBW) system / control unit 250: Vehicle Management System 252: System Vehicle Applications 300: Processing Device System-on-Chip (SOC) 303: Heterogeneous Processor 304: Heterogeneous Processor 305: Camera Actuation and Management (CAM) 306: Heterogeneous Processor 307: Heterogeneous Processor 308: Heterogeneous Processor 310: Processor 312: Memory Components 314: Analog and Custom Circuits 316: System components and other subsystems 317: Heterogeneous Processor 318: Clock 320: Voltage Regulator 324: Interconnect / Bus Module 400: System 402: Vehicle computing equipment 404: Remote Platform 406: Machine-executable instructions 408: Priority determines the module 410: Computing Resource Allocation Module 428: Electronic storage device 430: Processor 500: Vehicle Calculation System 502: SOC Resource Budget Controller 504: Neural Network Resource Budget Controller 506:SOC 508: Preprocessing Operation 510: Main trunk 512: Head 514: Post-processing operations 516: QoD Estimator 518: SOC Resource Consumption Estimator 530: Function 540: Configuration 550: Performance-Effectiveness Curve 552: Curve 554: Curve 556: Curve 558: Curve 560: Curve 560a: Area 562: Curve 562a: Area 564: Curve 566: Curve 570: Horizontal line 572: Horizontal line 574: Horizontal line 576: Arrow 600a:Method 600b: Operation 600c: Operation 602: Square 604: Square 610: Square 612: Square 614: Square 620: Square 622: Square 624: Square 626: Square

[0152] Domestic storage information (please note in order of storage institution, date, and number) none Overseas storage information (please note in the order of storage country, institution, date, and number) none

Claims

1. A method for allocating computing resources to concurrently executing neural networks, executed by a processor of a vehicle, comprising the steps of: determining a priority order for each neural network based on an indication of a contribution of each of a plurality of neural networks executing on a vehicle processing system to overall vehicle safety performance, wherein the indication is provided by each neural network; and allocating computing resources to the plurality of neural networks based on the determined priority order of each neural network.

2. The method according to claim 1, wherein determining a priority order of each neural network based on a contribution of each of the plurality of neural networks executed on a vehicle processing system to the overall vehicle safety performance comprises the following steps: determining the priority order of each neural network based on a contribution of each of the plurality of neural networks to the overall vehicle safety performance in a vehicle operating context.

3. The method of claim 1, wherein the overall vehicle safety performance is calculated based on a model using the inference accuracy and speed of the plurality of neural networks as input values.

4. The method according to claim 1, wherein the overall vehicle safety performance indicator is a driving quality based on a factor perceptible to a human passenger.

5. The method according to claim 1, wherein determining a priority order for each of the plurality of neural networks executed on the vehicle processing system includes: The relative effectiveness of one or more neural networks among a plurality of neural networks is determined based on the output inference per second and the output accuracy of each neural network.

6. The method according to claim 1, wherein allocating computational resources to the plurality of neural networks based on the determined priority of each neural network includes: Adjust one or more hyperparameters of one or more neural networks among the plurality of neural networks based on the determined priority order of one or more neural networks among the plurality of neural networks.

7. The method according to claim 6, wherein adjusting one or more hyperparameters of one or more neural networks among the plurality of neural networks based on the determined priority of one or more neural networks among the plurality of neural networks includes the step of: adjusting one or more hyperparameters based on a performance-efficiency curve of one or more neural networks among the plurality of neural networks.

8. The method according to claim 1 also includes the following steps: determining the performance of one or more neural networks among the plurality of neural networks using the allocated computing resources; and reallocating computing resources to one or more neural networks among the plurality of neural networks based on the determined performance of one or more neural networks.

9. The method according to claim 8 also includes the following steps: monitoring a dynamic availability of the computing resource and an actual usage of the computing resource, wherein reallocating the computing resource to one or more of the plurality of neural networks based on the determined performance of one or more of the neural networks includes: Based on the dynamic availability and actual use of the computing resources, computing resources are reallocated to one or more of the plurality of neural networks.

10. The method of claim 8, wherein reallocating computational resources to one or more neural networks among the plurality of neural networks based on the determined performance of one or more neural networks includes: Based on the performance of each of the plurality of neural networks, one or more hyperparameters of one or more of the plurality of neural networks are readjusted.

11. The method according to claim 1 also includes the following steps: determining whether available computing resources have increased, decreased, or remained the same; in response to the determination that the available computing resources have increased, adjusting one or more neural network hyperparameters that have a relatively large impact on overall vehicle safety performance; and in response to the determination that the available computing resources have decreased, adjusting one or more neural network hyperparameters that have a relatively small impact on overall vehicle safety performance.

12. A processing system, comprising: A processor configured with processor-executable instructions to perform the following operations: determining a priority order for each neural network based on an indication of the contribution of each of a plurality of neural networks executing on a vehicle processing system to the overall vehicle safety performance, wherein the indication is provided by each neural network; and allocating computing resources to the plurality of neural networks based on the determined priority order of each neural network.

13. The processing system according to claim 12, wherein the processor is also configured with processor-executable instructions to perform the following operation: determining the priority order of each neural network based on the contribution of each of the plurality of neural networks executing on the vehicle processing system in a vehicle operating context to the overall vehicle safety performance.

14. The processing system according to request item 12, wherein the processor is also configured with processor-executable instructions such that the overall vehicle safety performance is calculated based on a model using the inference accuracy and speed of the plurality of neural networks as input values.

15. The processing system according to claim 12, wherein the processor is also configured with processor-executable instructions such that the overall vehicle safety performance indicator is a driving quality based on a factor perceptible to a human passenger.

16. The processing system according to claim 12, wherein the processor is also configured with processor-executable instructions to perform the following operation: determining a relative performance of one or more of the plurality of neural networks based on an output inference per second and an output accuracy of each neural network.

17. The processing system according to request item 12, wherein the processor is also configured with processor-executable instructions to perform the following operation: adjusting one or more hyperparameters of one or more neural networks among the plurality of neural networks based on the determined priority order of one or more neural networks.

18. The processing system according to request item 17, wherein the processor is also configured with processor-executable instructions to perform the following operation: adjusting one or more hyperparameters based on a performance-effectiveness curve of one or more of the plurality of neural networks.

19. The processing system according to claim 12, wherein the processor is also configured with processor-executable instructions to perform the following operations: determining the performance of one or more of the plurality of neural networks using allocated computing resources; and reallocating computing resources to one or more of the plurality of neural networks based on the determined performance of one or more of the plurality of neural networks.

20. The processing system according to claim 19, wherein the processor is also configured with processor-executable instructions to perform the following operations: monitoring a dynamic availability of the computing resource and an actual use of the computing resource; and reallocating the computing resource to one or more of the plurality of neural networks based on the dynamic availability of the computing resource and the actual use of the computing resource.

21. The processing system according to request item 12, wherein the processor is also configured with processor-executable instructions to perform the following operation: readjusting one or more hyperparameters of one or more of the plurality of neural networks based on the performance of each of the plurality of neural networks.

22. The processing system according to claim 12, wherein the processor is also configured with processor-executable instructions to perform the following operations: determining whether available computing resources have increased, decreased, or remained the same; in response to determining that the available computing resources have increased, adjusting one or more neural network hyperparameters that have a relatively large impact on overall vehicle safety performance; and in response to determining that the available computing resources have decreased, adjusting one or more neural network hyperparameters that have a relatively small impact on overall vehicle safety performance.

23. A non-transitory processor-readable medium having processor-executable instructions stored thereon, the processor-executable instructions being configured to cause a processing device of a vehicle to perform operations including: determining a priority order for each neural network based on an indication of a contribution of each of a plurality of neural networks executing on a vehicle processing system to the overall vehicle safety performance, wherein the indication is provided by each neural network; and allocating computing resources to the plurality of neural networks based on the determined priority order of each neural network.

24. A processing system, comprising: Unit for determining a priority order of each neural network based on an indication of the contribution of each neural network executing on a vehicle processing system to the overall vehicle safety performance, wherein the indication is provided by each neural network; and unit for allocating computing resources to the plurality of neural networks based on the determined priority order of each neural network.

25. The processing system according to claim 24, wherein the unit for determining a priority order of each neural network based on a contribution of each of the plurality of neural networks executed on a vehicle processing system to the overall vehicle safety performance comprises: A unit for determining the priority order of each neural network based on the contribution of each of the plurality of neural networks executing on the vehicle processing system in the context of vehicle operation to the overall vehicle safety performance.

26. The processing system according to claim 24, wherein the unit for determining a priority order of each of the plurality of neural networks executed on the vehicle processing system comprises: A unit for determining the relative performance of one or more neural networks among a plurality of neural networks based on one-second output inference and one output accuracy of each neural network.

27. The processing system according to claim 24, wherein the unit for allocating computing resources to the plurality of neural networks based on the determined priority of each neural network includes: A unit for adjusting one or more hyperparameters of one or more neural networks among the plurality of neural networks based on the determined priority order of one or more neural networks.

28. The processing system according to claim 27, wherein the unit for adjusting one or more hyperparameters of one or more neural networks among the plurality of neural networks based on the determined priority order of one or more neural networks among the plurality of neural networks includes: A unit used to adjust one or more hyperparameters based on a performance-efficiency curve of one or more of the plurality of neural networks.

29. The processing system according to request item 24 also includes: A unit for determining the performance of one or more of the plurality of neural networks using the allocated computing resources; And a unit for reallocating computing resources to one or more of the plurality of neural networks based on the performance of one or more of the determined neural networks.

30. The processing system according to request item 29 also includes: A unit for monitoring a dynamic availability and an actual use of the computing resource, wherein the unit for reallocating computing resources to one or more neural networks among the plurality of neural networks based on the determined performance of one or more neural networks includes: a unit for reallocating computing resources to one or more neural networks among the plurality of neural networks based on the dynamic availability and the actual use of the computing resource.

31. The processing system according to claim 29, wherein the unit for reallocating computing resources to one or more neural networks among the plurality of neural networks based on the determined performance of one or more neural networks includes: A unit for readjusting one or more hyperparameters of one or more neural networks among the plurality of neural networks based on the performance of each of the plurality of neural networks.

32. The processing system according to request item 24 also includes: Units used to determine whether available computing resources have been increased, decreased, or kept the same; Units for adjusting one or more neural network hyperparameters that have a relatively large impact on overall vehicle safety performance in response to a decision that the available computing resources have increased; and units for adjusting one or more neural network hyperparameters that have a relatively small impact on overall vehicle safety performance in response to a decision that the available computing resources have decreased.

Images