State pause for optimizing start-up process of autonomous vehicle

Abstract

This disclosure relates to state suspension for optimizing a startup process of an autonomous vehicle. Diagnosis and startup of AV hardware and software of a computer system of an autonomous vehicle can be performed based at least on receiving a shutdown or power-off indication, then a computing state of the computer system can be suspended when the computer system enters a low-power mode. The suspended computing state can be quickly restored without a need to restart and diagnosis for turning on. To ensure the integrity of the saved computing state, the computer system can exit the low-power mode, re-run the diagnosis, reload the programs, and then re-enter the low-power mode. The restoration of the suspended computing state can be triggered by a user inserting an ignition key, pressing a button to turn on the vehicle, opening a door of the vehicle, remotely unlocking the vehicle, remotely starting the vehicle, and the like.

Description

BACKGROUND

[0001] It is very difficult to design a system for autonomous driving of a vehicle without supervision at the required level of safety. An autonomous vehicle (AV) should at least be able to perform as a functionally equivalent of an attentive driver who utilizes a perception and action system with an amazing ability to recognize moving and static obstacles in a complex environment and react to them to avoid collisions with other objects or structures along its path. To meet these standards, autonomous vehicles are running increasingly complex programs. As this complexity increases, the startup time of the autonomous driving system also increases, in part due to the increase in software data that must be loaded and configured. At the same time, there is an increasing demand for reducing the startup time of a vehicle to start (e.g., the time from starting the vehicle to being able to drive safely).

[0002] Prior to running the autonomous driving software, the autonomous driving system can run initial diagnostics (e.g., potential fault tests). This typically requires a system reboot followed by AV hardware and AV software initialization, where the AV software is loaded from persistent storage to random access memory (RAM) and performs diagnostics. Traditionally, this process is triggered by the vehicle being started or a door being opened. Due to the complexity of modern AV systems, this process can take a long time to complete (e.g., 20 seconds or more). One way to reduce the startup time is to split the vehicle architecture, e.g., using a dedicated electronic control unit (ECU) for the cluster (rear view camera), another ECU for vehicle networking, another for ADAS, so that each of these ECUs can be built specifically to facilitate faster loading of the corresponding functionality. However, the more the vehicle architecture is split, the more dedicated hardware and software is needed, thereby increasing the complexity of the system. Moreover, as AV software becomes more complex, even this approach can not be sufficient to meet the startup time requirements. SUMMARY

[0003] Embodiments of the present disclosure relate to a suspended state for fast startup of an autonomous vehicle. The disclosed systems and methods reduce the startup time required to bring a computer system (and, by extension, an autonomous vehicle) into a fully operational mode after being started.

[0004] In contrast to traditional systems such as those described above, diagnostics and startup of the AV hardware and software of a computer system of an autonomous vehicle can be performed based at least on receiving a shutdown or power-off indication, and then the computing state of the computer system can be suspended when the computer system enters a low-power mode. When the autonomous vehicle is restarted, the suspended computing state can be quickly restored without the need for a reboot and diagnostics start.

[0005] To enter the low power mode, the computer system can perform various diagnostic functions, run safety mechanisms, and then reload the program into a memory storage medium, such as random access memory (RAM). While in the low power mode, certain components of the computer system can be fully powered, certain components can be partially powered, and certain components can be powered off. To ensure the integrity of the saved computing state, the computer system can exit the low power mode after a certain time interval. At the termination of the interval, the computer system can re-run the diagnostics, reload the program into the RAM, and then re-enter the low power mode. When the driver returns to the vehicle, the computer system can exit the low power mode (e.g., due to receiving an indication of an on event). Because the diagnostics were performed prior to entering the low power mode, further diagnostics can not be required. Additionally, because the program is already in the RAM, the time before the program is ready to run can be greatly reduced. Thus, the computer system can have a faster start-up time, and a simpler and more cost effective system can be implemented even with more general purpose system architectures (e.g., a single ECU). BRIEF DESCRIPTION OF DRAWINGS

[0006] The present systems and methods for a suspended state of a computer system in an autonomous vehicle are described in detail below with reference to the attached drawing figures, wherein:

[0007] Figure 1 is a flowchart illustrating a method for entering and exiting a suspended state according to some embodiments of the present disclosure;

[0008] Figure 2 is a system diagram illustrating components of a computer system of an autonomous vehicle according to some embodiments of the present disclosure;

[0009] Figure 3A is a flowchart illustrating processing of a request to enter a suspended state performed by a processing system and a controller according to some embodiments of the present disclosure;

[0010] Figure 3B is a flowchart illustrating entering a suspended state performed by a processing system and a controller according to some embodiments of the present disclosure;

[0011] Figure 4A is a flowchart illustrating exiting a suspended state to resume autonomous control performed by a processing system and a controller according to some embodiments of the present disclosure;

[0012] Figure 4B is a flowchart illustrating exiting a suspended state to a powered off state performed by a processing system and a controller according to some embodiments of the present disclosure;

[0013] Figure 5is a flowchart illustrating a method of entering and exiting a low power mode to resume autonomous control according to some embodiments of the present disclosure;

[0014] Figure 6 is a flowchart illustrating a method of controlling a processing system to enter and exit a low power mode according to some embodiments of the present disclosure;

[0015] Figure 7 is a flowchart illustrating a method of indicating entering and exiting a low power mode according to some embodiments of the present disclosure;

[0016] Figure 8A is an illustration of an example autonomous vehicle according to some embodiments of the present disclosure;

[0017] Figure 8B is an illustration of an example autonomous vehicle according to some embodiments of the present disclosure; Figure 8A is an example camera position and field of view of the example autonomous vehicle of

[0018] Figure 8C is an illustration of an example autonomous vehicle according to some embodiments of the present disclosure; Figure 8A is a block diagram of an example system architecture of the example autonomous vehicle of

[0019] Figure 8D is a system diagram for communicating between a cloud-based server and the example autonomous vehicle of Figure 8A

[0020] Figure 9 is a block diagram of an example computing device suitable for implementing some embodiments of the present disclosure; and

[0021] Figure 10 is a block diagram of an example data center suitable for implementing some embodiments of the present disclosure. DETAILED DESCRIPTION

[0022] Systems and methods related to a suspended state for fast start of an autonomous vehicle are disclosed. Although the present disclosure can be described with respect to an exemplary autonomous vehicle 800 (or referred to herein as "vehicle 800" or "self vehicle 800", an example of which is illustrated in FIG. 1, other embodiments are contemplated. Figures 8A-8D ​The related description), but this is not intended to limit the present disclosure. For example, the systems and methods described herein can be used by, without limitation, non-autonomous vehicles, semi-autonomous vehicles (e.g., in one or more adaptive driver assistance systems (ADAS)), manned and unmanned robots or robotic platforms, warehouse vehicles, off-road vehicles, vehicles connected to one or more trailers, airships, watercraft, shuttles, emergency response vehicles, motorcycles, electric or motorized bicycles, aircraft, construction vehicles, underwater vehicles, drones, and / or other vehicle types. Moreover, although the present disclosure can be described with respect to the start and halt computer systems of autonomous vehicles, this is not intended to limit the present disclosure, the systems and methods described herein can be used for augmented reality, virtual reality, mixed reality, robotics, security and surveillance, autonomous or semi-autonomous machine applications, and / or any other technology space that can use a start and halt of a computer.

[0023] In one or more embodiments, the computer system can enter a low power mode after an indication (e.g., from a driver and / or system components) that the vehicle is turned off (e.g., shut down or powered down). For example, a user can remove a key, press a physical or virtual button to turn off the vehicle, exit the vehicle, etc. In one or more embodiments, the computer system can end a drive cycle that at least partially operates the vehicle autonomously. This can include turning off various sensors and displays. In one or more embodiments, the computer system will run diagnostics on the hardware. There is a possibility of flipped bits or other errors that can accumulate during operation in any powered component. For example, these errors can be caused at least in part by radiation (such as solar radiation, alpha particles, etc.). The diagnostics can check for error accumulation during operation of the various components. These diagnostics can include a potential fault check.

[0024] In one or more embodiments, the computer system will at least partially reinitialize or reboot into the drive cycle prior to performing the diagnostics. In one or more embodiments, after the diagnostics and / or reboot, the computer system will halt the components and save the current state in RAM. The potential fault check, which can be performed as at least a portion of the diagnostics, can corrupt, jeopardize, or otherwise affect the current state of the processor. Upon completion of the potential fault check and prior to halting to RAM, the system can be secured by issuing a full reboot of the system. The computer system can then enter a low power mode. As described herein, the low power mode can refer to any of a variety of modes that reduce power usage of the computer system. Other components can be disabled or in standby. In one or more embodiments, the system can continue to perform diagnostics while performing entry into the low power mode, disabling the safety mechanism only at the point where the logic hosting that mechanism is disabled.

[0025] The computer system can remain in the low power mode until triggered to exit the low power mode. Three examples of triggers will be briefly discussed by way of example and not limitation. First, the computer system can detect or determine that an operator has returned to the vehicle, causing the computer system to exit the low power mode and enter a normal operating mode. The normal operating mode can include autonomous control and other functions that are performed at full power. Second, a certain time interval can elapse, causing the computer system to re-run diagnostics, reload programs, and re-enter the low power mode. Third, at a certain battery charge threshold (or other threshold), the computer system can exit the low power mode, shut down completely, or power off (to prevent further battery drain). While a battery is described herein, the description relating to the battery can apply to one or more energy storage media (e.g., capacitors, etc.).

[0026] In the low power mode, the most power-hungry components can be powered off. Examples of powered-off components can include a central processing unit (CPU) and a graphics processing unit (GPU). The CPU and / or GPU can partially power autonomous driving functionality, for example, computer vision using neural networks. In the low power mode, certain components can be fully powered (examples include the vehicle battery, power supply pre-regulation circuit, power supply sequencer, voltage regulator for the clock-on portion of the processor, voltage regulator for DRAM, etc.). The vehicle battery can power the computer system from a switch or directly from the vehicle battery. In the low power mode, certain components can be partially powered (examples include the power management integrated circuit, always-on portion of the processor, radar, Ethernet, etc.). Some of these can be powered directly from the battery and can be configured to provide wake-up signals to other components as needed to exit the low power mode. In addition, in the low power mode, certain components can be fully powered off (e.g., graphics processing unit, microcontroller unit, vehicle wiring harness, peripherals, sensors, and display). The microcontroller unit and / or other controllers can be used for safety, other aspects of the inspection system, and power control.

[0027] As described herein, the computer system can remain in the low power mode for a certain time interval. For example, the time interval can be 8 hours or 24 hours. The time interval can be a static number, a static number that is programmable (e.g., programmed by the vehicle manufacturer or end customer), or can be dynamic. After the interval, the computer system can briefly exit the low power mode, re-run diagnostics, re-initialize programs, reboot, and re-enter the low power mode. This cycle can continue until an exit event is detected, for example, a start instruction is received or the battery charge is below a certain level.

[0028] In one or more embodiments, entering the low power mode can be run at key-off (or some other indication that the drive cycle is ending). However, entering the low power mode can be configured to run when various potential criteria or conditions are met. If desired, the user can also be presented with an option to fully power down.

[0029] The computer system can exit the low power mode to enable normal autonomous driving operations. Exiting the low power mode can be triggered by an indication that a return to normal driving operations is about to begin or is imminent. For example, the user can insert a key, press a button to start the vehicle, open a door of the vehicle, remotely unlock the vehicle, remotely start the vehicle, etc.

[0030] To exit the low power mode, a wake-up signal can be sent to the microcontroller unit (and / or other controllers). The microcontroller unit can trigger the computer system to leave the low power mode based at least on the wake-up signal. Upon being triggered, the computer system can leave the low power mode. Leaving the low power mode can include (for example, but not limited to) tasks or operations such as: turning on clocks, sensors, and displays. Leaving the low power mode can also include (for example, but not limited to) tasks or operations such as: verifying, restoring saved states, and otherwise preparing the computer system for normal operations. Since the computing states are saved in RAM (and / or other storage media), the computer system can enter normal operations (e.g., the autonomous vehicle is ready for autonomous control) faster than traditional processes and solutions. For example, all application states and sensor states can already be in RAM.

[0031] In some cases, the computer system can enter a powered down mode or state from the low power mode. The computer system can enter the powered down mode based at least on one or more criteria being met. For example, the computer system can enter the powered down mode when the battery level is below a certain threshold. As another example, the computer system can enter the powered down mode based at least on a timeout of the total time in the low power mode. In these instances, the microcontroller unit can be woken up by the microcontroller power management circuit. The microcontroller unit can acknowledge the timeout and then instruct the various components in the low power mode to fully shut down or power down. The microcontroller can also place its power management circuit in standby mode and then shut itself down. Certain components, for example, the power management circuit, can remain powered in the powered down state to allow for a restoration. The computer system can remain in the powered down mode until a start or other indication is received (e.g., the battery has been sufficiently charged, the vehicle has been plugged into a power source, etc.).

[0032] Reference Figure 1 , Figure 1is a flowchart illustrating a method 100 for entering and exiting a suspended state, in accordance with some embodiments of the present disclosure. Each block of the method 100 (and other methods described herein) comprises a computational process that can be performed using any combination of hardware, firmware, and / or software. For example, various functions can be implemented by a processor executing instructions stored in memory. The method 100 can also be implemented as computer-usable instructions stored on a computer storage media. The method 100 can be provided by a standalone application, a service, or a hosted service (standalone or in combination with another hosted service) or a plug-in to another product, just to name a few. Further, the method 100 is described, by way of example, with reference to the computer system 200 of Figure 2 However, the method 100 can additionally or alternatively be performed by any one system or any combination of systems, including but not limited to the systems described herein.

[0033] The method 100 can be used to Figure 2 the startup, suspended state, and loaded state of the computer system 200 (e.g., ECU) to enable autonomous control of an autonomous vehicle (e.g., vehicle 800). However, the method 100 can also be used for other types of computer systems, or for implementing other types of functionality using computer systems. In embodiments of the present disclosure, the method 100 can be run when the autonomous vehicle is turned off. In addition to or instead of being turned off, the method 100 can be configured to run under certain criteria or conditions. Further, if desired, a user can choose to completely power down the computer system 200 of the vehicle, rather than running the method 100 and placing the computer system 200 in a low-power mode. For example, if the user knows that the autonomous vehicle will not be used for a long period of time, the user can choose to completely power down the vehicle, rather than allowing the autonomous vehicle to enter the method 100, which can be executed by default.

[0034] At block B102, the method 100 includes receiving or detecting a power-off indication. The computer system can begin entering a low-power mode based at least on the shutdown or power-off indication, such as an indication from the driver (and / or system components) that the vehicle has stopped running or is about to stop running (and / or that autonomous control is to be disabled). For example, the user can remove the ignition key, press a button to shut down the vehicle, exit the vehicle, etc. Based on the shutdown or power-off indication, the computer system can end a drive cycle of autonomously operating the vehicle. This can include shutting down various sensors and displays of the vehicle.

[0035] At block B104, the method 100 includes running a diagnostic on the computer system, such as an ECU. The computer system can run any of a variety of diagnostics on the hardware. In one or more embodiments, the diagnostics can be performed using in-system testing (IST). During normal computer operation, bits in the computer system can flip or encounter other errors or faults. These errors and faults can be caused by radiation (solar radiation, alpha particles, etc.) or other external sources, or can be caused at power-up. The diagnostic performed can include a potential fault check to determine and correct such errors. In one or more embodiments, while Figure 1 not shown in FIG. 1, the computer system can at least partially reboot prior to the diagnostic (e.g., based on the shut down or power off indication of block B102, and / or based on exiting the low power mode at block B124).

[0036] At block B106, the method 100 includes rebooting the computer system. The computer system can at least partially reboot after the diagnostic, back to the drive cycle. The computer system can at least partially perform a reboot because the potential fault check corrupted the current state of the processor. The reboot can create a new computing state that includes a current loaded instance of one or more programs for autonomous control and / or other functions of the autonomous vehicle.

[0037] At block B108, the method 100 includes storing the computing state. For example, the computing state can be stored in one or more computer storage media, which can include at least one volatile memory, such as random access memory (RAM). Storing the computing state prior to entering the low power mode can allow the computing state to be quickly called upon when exiting the low power mode, without having to perform a full reboot.

[0038] At block B110, the method 100 includes entering a low power state. To enter the low power state, the computer system can suspend one or more components discussed herein. Other components can be disabled or in a standby state. Reference is made to Figure 2 Examples of such components are further discussed. In some embodiments, the low power mode is system control 7 (SC7), where system control can refer to a system power state. Other system controls can include system control 0 where the computer system is at full power or system control 8 where the system is completely powered off (e.g., save wake circuitry). Other levels and configurations of low power modes can also be used.

[0039] In one or more embodiments, after entering the low power mode at block B110, various other functions can be performed according to various criteria being met. For example, blocks B112-B116, B118-B120, B122-B124, or B126-128 can occur. With respect to blocks B112-B116, at block B112, the method 100 includes an occurrence of a power-up or start indication. The power-up or start indication can trigger an exit from the low power mode and can include, by way of example and not limitation, a return to normal driving operations being imminent, about to occur, or otherwise expected. The power-up or start indication can be received from a component external to a computer system (e.g., an ECU) located in the autonomous vehicle, from a sensor communicatively coupled with the computer system, and / or from some other electronic device (e.g., a key fob). In one or more embodiments, a user can trigger the power-up or start indication by interacting with the autonomous vehicle in some manner. For example, the user can insert an ignition key, press a button to turn on the vehicle, open a door of the vehicle, remotely unlock the vehicle, remotely start the vehicle, approach the vehicle using a registered wireless communication device, etc.

[0040] At block B114, the method 100 includes exiting the low power mode. For example, the controller can trigger the processing system and other components of the computer system to leave the low power mode. This can include the controller instructing a power manager to power various components, as described herein. The controller and / or processing system can also turn on clocks, sensors, and displays. This can include verifying, initializing, and / or otherwise preparing the computer system for normal operation. Because the previous computing state was saved in memory (e.g., according to block B108), the computer system can enter a runtime without needing to load and initialize all AV software and hardware configurations. For example, all application states and sensor states can already be loaded into the computing state saved in memory.

[0041] At block B116, the method 100 includes enabling autonomous control. For example, when the computing state has been restored from memory, the computer system can take autonomous control of the vehicle. The computer system can maintain autonomous control of the vehicle until a shutdown or power-off indication is received or some other event occurs. For example, the method 100 can loop back to block B102 and continue another cycle.

[0042] With respect to blocks B118-B120, at block B118, the method 100 includes an occurrence of a low power supply. For example, a low battery power can be determined by the computer system, a component external to the computer system, or some other device. In one or more embodiments, the low power supply can be determined based at least on a power level being below a threshold.

[0043] At block B120, the method 100 includes the computer system being completely powered off. With a complete power off, consumption on the power supply will stop or substantially decrease.

[0044] Continuing to operate in the low power mode when the power source falls below a certain threshold can be disadvantageous because the power source can be too low to allow the autonomous vehicle to start or operate for a sufficient period of time. For example, for an internal combustion engine powered vehicle, a low battery can prevent the starter from successfully cranking the engine. Similarly, in the case of an electric vehicle, a low battery can reduce the range available to the electronic vehicle. Blocks B122-B124 can be used to improve such situations. With respect to blocks B122-B124, at block B122, the method 100 includes having passed a certain time interval (and / or other conditions being met). The time interval can indicate that the diagnostics should be performed again to remain in the low power mode.

[0045] The computer system can remain in the low power state for a certain time interval. For example, the time interval can be 8 hours or 24 hours. The time interval can be a pre-determined number, a static number that is programmable (e.g., by the vehicle manufacturer or end customer), or can be dynamic. After the interval, the computer system can briefly exit the low power state, re-run the diagnostics, re-initialize the program, and re-enter the low power mode (e.g., according to blocks B104-B110). In one or more embodiments, the computer system can also restart before block B104.

[0046] Accordingly, at block B124, the method 100 includes exiting the low power mode, after which the method 100 can return to block B104 (e.g., after a restart), such that the diagnostics can be performed again. The computer system can then restart, store the computing state, and re-enter the low power mode (e.g., at block B110). For example, the computer system can repeat the cycle as long as the power source (e.g., battery) has sufficient power and does not receive a power-on or start indication. The computer system can repeat the cycle while the power source is externally charging until a power-on or start indication is received (e.g., as described in block B112).

[0047] At block B126, the method 100 can include detecting one or more faults while the computer system is in the low power mode. For example, while the processing system and other components of the computer system are in the low power mode, the controller can perform diagnostics. This can include the controller instructing the processor, power manager, or other component to test for one or more faults periodically or upon a certain trigger. The diagnostics can be the same as or different from the diagnostics performed in block B104.

[0048] At block B128, the method 100 can include recording the fault after detecting the fault in block B126. The recording of the fault can create or update a record indicating the detected component and type of fault, and can include various sensor readings related to the fault, etc. In some embodiments of the present disclosure, after detecting and / or recording the fault, the controller can instruct a full power down. This full power down can include one or more of the steps described herein in block B120. In other embodiments of the present disclosure, after detecting and / or recording the fault, the computer system can remain in a low power mode for a time interval (as discussed in block B122) and / or receive a power up or start indication (as discussed in block B112). In these embodiments, the recorded fault can be further evaluated and tested after the computer system returns to full power mode (e.g., as discussed in blocks B114 and / or B124). If the fault is verified after returning to full power mode, the controller can instruct the computer system to fully power down (as discussed in block B120).

[0049] Referring to Figure 2 , Figure 2 is an example computer system 200 for an autonomous vehicle 800 or other machine. The computer system 200 can have one or more processing systems 202. The one or more processing systems 202 can load and run autonomous control software for the autonomous vehicle, and / or can perform other functions. The computer system 200 can also have one or more controllers 204. The one or more controllers can include a microcontroller unit (MCU) or other controller. For safety, the one or more controllers 204 can be used to check, instruct, and monitor other components of the computer system 200. The one or more controllers 204 can also control the power to other components of the computer system 200. The computer system 200 can also have computer storage 206 (e.g., volatile memory storage). In one or more embodiments, the computer storage 206 can include one or more computer storage media, such as RAM, and can include dynamic random access memory (DRAM). The computer system 200 can also have or have access to non-volatile memory (not shown) for storing various programs and configurations of the processing system 202. As described herein, this memory can be used to load programs and configurations during a restart of the computer system 200.

[0050] The computer system 200 can also have, or be otherwise associated with, a power source 208. Examples of the power source 208 include a vehicle battery, such as a battery that powers an electric vehicle, a battery that provides electrical functionality to an internal combustion powered vehicle, a dedicated battery for an electronic control unit, and / or other batteries. The computer system 200 can additionally have an interface manager 210. The interface manager can manage one or more communication interfaces, such as a controller area network (CAN), an Ethernet, a FlexRay network, and / or other network or interface type communication interfaces. In one or more embodiments, the interface manager 210 can include a CAN controller, an Ethernet physical layer (such as a chip or software that transmits and receives Ethernet frames), a FlexRay communication bus, and / or other components. The computer system 200 can also have a power manager 212. By way of example, the power manager can include a power management integrated circuit (PMIC). The power manager 212 can interface with a vehicle wiring harness 214 as well as the controller 204.

[0051] The computer system 200 can also have one or more pre-regulators 216. The one or more pre-regulators 216 can be configured to reduce ripple present in output power from the power source 208. The one or more pre-regulators 216 can additionally or alternatively reduce or minimize power dissipation at the voltage regulator 220. The one or more pre-regulators 216 can interface with a peripheral power source 218. The peripheral power source 218 can provide power, directly or indirectly, to any of various peripheral components, such as the sensors described herein.

[0052] The voltage regulator 220 can provide power to various other components of the computer system 200, including a wake module 222 on the processing system 202. The wake module 222 can include an always-on portion of the processing system 202 and / or a portion that can be turned on when the processing system 202 is in a low power mode. The voltage regulator 220 can provide power to the wake module 222 via a wake module power source 224, which can include a voltage regulator for the always-on portion of the processing system 202. The voltage regulator 220 can also provide power to the computer storage 206 via a storage power source 226. The voltage regulator can also provide power to the processing system 202 via a main power source 228 of the processing system 202 (e.g., when the processing system 202 is not in a low power mode and / or when in normal operation).

[0053] As described herein, in the low power mode, certain components can receive full power, partial power, or no power. In various embodiments, components can be selected for full power, partial power, or no power based on their functionality and the need during the low power mode and leaving the low power mode. To enter the low power mode, the most power-consuming components of the computer system 200 can be powered off. This can include the central processing unit (CPU) and / or GPU, which can be components of or otherwise related to the processing system 202. For example, the processing system 202 can be or include one or more of the SoCs 804 of Figure 8C In addition, the one or more processors 810, one or more CPUs 806, one or more accelerators 814, and / or one or more GPUs 808 can be powered off for the low power mode. In embodiments where the processing system 202 includes one or more logic units 920, they can also be powered off for the low power mode. Figure 9

[0054] When in the low power mode or state, certain components can be fully powered (examples include the power supply 208, pre-conditioner 216, wake module 222, wake module power supply 224, storage power supply 226, etc.), as shown in Figure 2 The vehicle battery or other power supply 208 supplies power to the electronic control unit, and can be from a switch or directly from the vehicle. When in the low power state, certain components can be partially powered (examples include the power manager 212, interface manager 210, computer storage 206, processing system 202, etc.). Some of the partially powered components can be powered from the power supply 208, and can be configured to provide wake-up signals to other components. For example, the power manager 212 can be configured to provide one or more wake-up signals to the wake module 222 in order to wake the processing system 202 from the low power mode. When in the low power mode, certain components can be fully powered off (examples include the most power-consuming components of the processing system 202 described herein, one or more controllers 204, vehicle wiring harness 214, and peripherals, sensors, and / or displays that can be powered by the peripheral power supply 218). During the low power mode, various safety mechanisms can be partially or fully powered. Examples of these safety mechanisms can involve monitoring and controlling power, clocks, temperature, and other variables to ensure safe operation of the computer system 200 when leaving the low power mode, which can further improve start-up time by reducing the amount of testing needed when waking up.

[0055] Referring now to Figures 3A-3B , a general flow between the processing system 202 and the controller 204 is shown. By way of example, the processing system 202 and the controller 204 can interact according to the general flow to implement at least some of the blocks B102-B110 of the method 100.​Figure 3A An example flowchart 300A is described that includes receiving and processing a shutdown or power-off request of a computer system 200. Figure 3B An example flowchart 300B is depicted that includes entering a low-power mode in response to a shutdown or power-off request received and processed in flowchart 300A.

[0056] The functions that can be performed by the processing system 202 are shown on the left side of Figure 3A and 3B and the steps that can be performed by the controller 204 are shown on the right side of Figure 3A and 3B . Figure 3A and 3B The time elapses shown primarily illustrate the downward movement in the example process. It should be understood that the length of the time elapses shown is merely an example, and the time elapses can vary in the amount of time in each block or between each block without departing from the scope currently disclosed. Similarly, the arrows shown between the sides can indicate one or more messages or status updates sent between the processing system 202 and the controller 204, or can indicate the next step taken without sending any information or messages between the processing system 202 and the controller 204. In particular, the arrows and time intervals are merely to explain the concepts involved to the reader.

[0057] In flowchart 300A, block B302 includes the controller 204 requesting a shutdown or power down of the processing system 202, for example, in response to a shutdown or power down indication from a vehicle component outside of the computer system 200. The controller 204 can send the shutdown or power down request to the processing system 202. Block B304 includes the controller 204 initiating a power down sequence of the processing system 202. Block B306 includes the controller 204 triggering a restart of the processing system 202. In accordance with the trigger by the controller 204, block B308 includes the processing system 202 preparing for a restart. The processing system 202 can report to the controller 204 that the restart is ready, or the controller 204 can wait for a certain time interval. Block B310 includes the controller 204 asserting a reset of the processing system 202. The controller 204 can also initiate diagnostics to be performed by the processing system 202. In accordance with the indication by the controller 204, block B312 includes the processing system 202 restarting and subsequently running the diagnostics. When the diagnostics are complete, the processing system 202 can report to the controller 204 of the completion. The report can include an indication of no faults found, an indication of any faults found, or other information. Block B314 includes the controller 204 asserting a reset of the processing system 202 upon an indication that the diagnostics have been completed. Block B316 includes the processing system 202 restarting into a functional mode of operation. The functional mode of operation can include loading all or part of the programs, applications, functions, and other computer instructions associated with autonomous control of the autonomous vehicle. Block B318 includes the controller 204 detecting (or otherwise receiving an indication of) the functional mode of the processing system 202 being entered.

[0058] In flowchart 300B, block B350 includes the processing system 202 requesting entry into a low power mode after expiration of a certain timer. The timer can allow for cancellation of the low power mode. Block B352 includes the controller 204 triggering the low power mode and responding to the processing system 202. Block B354 includes the processing system 202 suspending units and saving a computing state to the computer storage 206. The suspended units can include any of the various components of the computer system 200, and can additionally or alternatively include various other components external to the computer system 200, such as peripherals, sensors, displays, input devices, output devices, and / or other electronic devices. The computing state that can have been generated using block B316 of flowchart 300A and detected using block B318 of flowchart 300A can be saved to the computer storage 206. This can allow for the computing state to be recalled from the computer storage 206 at startup. Block B356 includes the controller reporting a status to one or more components of the autonomous vehicle external to the computer system 200. Block B356 can be performed by the controller 204 while the processing system 202 is suspending units and / or saving the computing state. Block B358 includes the processing system 202 starting a power down sequence. Block B360 includes the processing system 202 placing at least a portion of the computer system 200 into a low power mode.

[0059] Referring now to Figure 4A , Figure 4A is a flowchart 400A illustrating exiting a suspended state to resume autonomous control by the processing system 202 and the controller 204, according to some embodiments of the present disclosure. By way of example, the processing system 202 and the controller 204 can interact according to flowchart 400A to implement at least some of blocks B112-B114 of method 100.

[0060] Block B402 includes the controller 204 receiving a vehicle power on or start trigger or indication. The trigger can be received from the vehicle wiring harness 214 by the power manager 212. Block B404 includes powering the controller 204, such as by the power manager 212. For example, power can be supplied from the power source 208 via the power manager 212. Block B406 includes the controller 204 starting up. The controller 204 can begin any of various operations to process the power on or start trigger and begin a process of indicating to other components of the computer system 200 to exit a low power mode. Block B408 includes the controller 204 responding to the power on or start trigger. For example, the controller 204 can send a message to components external to the computer system 200 indicating that the power on or start trigger has been received and is being processed.

[0061] Block B410 includes the controller 204 instructing the processing system 202 to exit the low power mode and resume normal operation. As described herein, the controller 204 can instruct the pre-regulator 216 and / or the voltage regulator 220 to power (e.g., at full power) the respective components of the computer system 200. In addition, the pre-regulator 216 and / or the voltage regulator 220 can power the processing system 202, the peripherals, the computer storage 206, and other components.

[0062] Block B412 includes the processing system 202 preparing to exit the low power mode. This can include recovering the computing state from the computer storage 206. Block B414 includes the processing system 202 indicating to the controller 204 that the processing system is ready for autonomous control. Block B416 includes the controller 204 sending a ready report of the processing system 202 to components outside of the computer system 200. B418 includes the controller 204 instructing the processing system 202 to autonomously control the autonomous vehicle. The processing system can then proceed with autonomous control until a subsequent shutdown or power off instruction is received, or some other event occurs or is detected.

[0063] Referring now to Figure 4B , Figure 4B is a flowchart 400B illustrating exiting from a suspended state to a powered off state by the controller 204, according to some embodiments of the present disclosure. By way of example, the controller 204 can act according to the flowchart 400B to implement at least some of blocks B122-B124 of the method 100.

[0064] Block B452 includes the power manager 212 triggering a timeout from the low power mode due to low battery power, total time in the low power mode cycle, or other trigger. Block B454 includes the controller 204 being powered by the power manager 212 in response to the wake up by the power manager 212. Block B456 includes the controller 204 acknowledging the timeout due to the trigger. Block B458 includes the controller 204 instructing power off (e.g., powered by the low power mode rail) of other components in the computer system 200. Block B460 includes the controller 204 placing the power manager 212 in standby mode and powering itself off. During the power off, all rails of the computer system 200 can be off, except that the power manager 212 can be in standby or sleep mode. In addition, the interface manager 210 can be in a low power standby mode.

[0065] Figure 5 is a flowchart illustrating a method 500 of entering and leaving a low power mode to resume autonomous control, according to some embodiments of the present disclosure. By way of example, the method 500 is described with respect to the computer system 200 of Figure 2 However, the method 500 can additionally or alternatively be performed by any one system or any combination of systems, including but not limited to those described herein.

[0066] At block B502, the method 500 includes running a diagnostic on the computer system. For example, the processing system 202 can run a diagnostic on the computer system 200 for autonomous control of the machine (e.g., the vehicle 800) based at least on the machine’s shutdown or power-off indication.

[0067] At block B504, the method 500 includes restarting the computer system. For example, the processing system 202 can restart one or more portions of the computer system 200 based at least in part on the diagnostic to configure the computer system 200 in a computing state.

[0068] At block B506, the method 500 includes storing the computing state. For example, the processing system 202 can store the computing state in the computer storage 206 as a saved state.

[0069] At block B508, the method 500 includes entering a low-power mode. For example, the processing system 202 can enter a low-power mode when the saved state is stored in the computer storage 206.

[0070] At block B510, the method 500 includes exiting the low-power mode. For example, the processing system 202 can exit the low-power mode based at least on the machine’s power-on or start indication.

[0071] At block B512, the method 500 enables autonomous control. For example, the processing system can enable autonomous control of the machine by the computer system 200 based at least on restoring the saved state from the computer storage 206.

[0072] Reference is now made to Figure 6 , Figure 6 is a flowchart illustrating a method of controlling a processing system to enter and exit a low-power mode, according to some embodiments of the present disclosure. The method 600 is described by way of example for the computer system 200 of Figure 2 However, the method 600 can additionally or alternatively be performed by any one system or any combination of systems, including but not limited to those described herein.

[0073] At block B602, the method 600 includes initiating a power-off sequence. For example, the controller 204 can initiate a power-off sequence in response to detecting a shutdown or power-off indication of a vehicle (e.g., the vehicle 800).

[0074] At block B604, the method 600 includes indicating a diagnostic to be run. For example, the controller 204 can indicate that the computer system 200 for autonomous control of the vehicle execute a diagnostic.

[0075] At block B606, the method 600 includes instructing a restart to be performed. For example, the controller 204 can instruct one or more portions of the computer system 200 (e.g., the processing system 202) to restart to configure the computer system 200 to the computing state and store the computing state as a saved state in the computer storage 206.

[0076] At block B608, the method 600 includes triggering a low-power mode. For example, the controller 204 can trigger a low-power mode of one or more components within the computer system 200 when the saved state is stored in the computer storage 206.

[0077] At block B610, the method 600 includes triggering an exit from the low-power mode. For example, the controller 204 can trigger an exit from the low-power mode based at least on a power-on or start indication of the vehicle, the triggering of the exit enabling the computer system 200 to autonomously control the vehicle based at least on the saved state recovered from the computer storage 206.

[0078] Reference is now made to Figure 7 , Figure 7 is a flowchart illustrating a method 700 of indicating entry into and exit from a low-power mode, according to some embodiments of the present disclosure. By way of example, the method 700 is described with respect to the computer system 200 (discussed herein). Figure 2 The method 700 can additionally or alternatively be performed by any one system or any combination of systems, including but not limited to those described herein.

[0079] At block B702, the method 700 includes performing an in-system test. For example, the processing system 202 can perform an in-system test of the computer system 200 for autonomous control of the vehicle based at least on a first indication of the vehicle being turned off.

[0080] At block B704, the method 700 includes configuring a computing state. For example, the processing system 202 can configure the computer system 200 to a computing state that enables autonomous control based at least on completion of the in-system test.

[0081] At block B706, the method 700 includes operating in a low-power mode. For example, the processing system 202 can operate in a low-power mode when the computing state is stored as a saved state in the computer storage 206.

[0082] At block B708, the method 700 includes exiting the low-power mode. For example, the processing system 202 can exit the low-power mode based at least on a second indication of the vehicle being turned on.

[0083] At block B710, the method 700 includes enabling autonomous control. For example, the processing system 202 can implement autonomous control of the machine by the computer system 200 based at least on restoring the saved state from the computer storage 206.

[0084] Example autonomous vehicle

[0085] Figure 8A is an illustration of an example autonomous vehicle 800 in accordance with some embodiments of the present disclosure. The autonomous vehicle 800 (alternatively referred to herein as “vehicle 800”) can include, but is not limited to, a passenger vehicle such as a car, truck, bus, emergency vehicle, shuttle, electric or motorized bicycle, motorcycle, fire vehicle, police vehicle, ambulance, boat, construction vehicle, underwater vessel, drone, vehicle connected to a trailer, and / or another type of vehicle (e.g., a vehicle that is unmanned and / or that accommodates one or more passengers). Autonomous vehicles are often described in terms of levels of automation as defined by a department of the United States Department of Transportation, the National Highway Traffic Safety Administration (NHTSA), and the Society of Automotive Engineers (SAE) “Taxonomy and Definitions for Terms Related to Driving Automation Systems for On-Road Motor Vehicles” (Standard No. J3016-201806 published June 15, 2018, Standard No. J3016-201609 published September 30, 2016, and previous and future versions of this standard). The vehicle 800 is capable of implementing functionality that complies with one or more of Levels 1-5 of autonomous driving. For example, the vehicle 800 can have the capability of driver assistance (Level 1), partial automation (Level 2), conditional automation (Level 3), high automation (Level 4), and / or full automation (Level 5), according to embodiments. The term “autonomous” as used herein can include any and / or all types of autonomy of the vehicle 800 or other machine, such as fully autonomous, highly autonomous, conditionally autonomous, partially autonomous, assistance-providing autonomous, semi-autonomous, primarily autonomous, or other designation.

[0086] The vehicle 800 can include components such as a chassis, a body, wheels (e.g., 2, 4, 6, 8, 18, etc.), tires, axles, and other components of a vehicle. The vehicle 800 can include a propulsion system 850, such as an internal combustion engine, a hybrid power plant, an all-electric motor, and / or another type of propulsion system. The propulsion system 850 can be connected to a drivetrain of the vehicle 800 that can include a transmission in order to effectuate propulsion of the vehicle 800. The propulsion system 850 can be controlled in response to receiving a signal from a throttle / accelerator 852.

[0087] A steering system 854 can include a steering wheel, which can be used to steer the vehicle 800 (e.g., along a desired path or route) while the propulsion system 850 is operating (e.g., when the vehicle is in motion). The steering system 854 can receive signals from a steering actuator 856. For full automation (level 5) functionality, the steering wheel can be optional.

[0088] A braking sensor system 846 can be used to operate the vehicle brakes in response to receiving signals from the braking actuator 848 and / or braking sensors.

[0089] One or more controllers 836 can include one or more system on a chip (SoC) 804 Figure 8C ) and / or one or more GPUs, which can provide signals (e.g., representing commands) to one or more components and / or systems of the vehicle 800. For example, the one or more controllers can send signals to operate the vehicle brakes via one or more braking actuators 848, to operate the steering system 854 via one or more steering actuators 856, to operate the propulsion system 850 via one or more throttle / accelerators 852. The one or more controllers 836 can include one or more on-board (e.g., integrated) computing devices (e.g., supercomputers) that process sensor signals and output operational commands (e.g., signals representing commands) to enable autonomous driving and / or to assist a human driver in driving the vehicle 800. The one or more controllers 836 can include a first controller 836 for autonomous driving functionality, a second controller 836 for functional safety functionality, a third controller 836 for artificial intelligence functionality (e.g., computer vision), a fourth controller 836 for infotainment functionality, a fifth controller 836 for redundancy in emergency situations, and / or other controllers. In some examples, a single controller 836 can handle two or more of the above functionalities, two or more controllers 836 can handle a single functionality, and / or any combination thereof.

[0090] One or more controllers 836 can provide signals for controlling one or more components and / or systems of the vehicle 800 in response to sensor data (e.g., sensor inputs) received from one or more sensors. The sensor data can be received from, for example and without limitation, global navigation satellite system sensors 858 (e.g., global positioning system sensors), RADAR sensors 860, ultrasonic sensors 862, LIDAR sensors 864, inertial measurement unit (IMU) sensors 866 (e.g., accelerometers, gyroscopes, magnetic compasses, magnetometers, etc.), microphones 896, stereo cameras 868, wide-angle cameras 870 (e.g., fisheye cameras), infrared cameras 872, surround cameras 874 (e.g., 360 degree cameras), long and / or medium range cameras 898, speed sensors 844 (e.g., to measure the speed of the vehicle 800), vibration sensors 842, steering sensors 840, brake sensors (e.g., as part of a brake sensor system 846), and / or other sensor types.

[0091] One or more of the controllers 836 can receive inputs (e.g., represented by input data) from the instrument cluster 832 of the vehicle 800 and provide outputs (e.g., represented by output data, display data, etc.) via a human-machine interface (HMI) display 834, audible annunciators, speakers, and / or via other components of the vehicle 800. These outputs can include information such as vehicle speed, velocity, time, map data (e.g., an HD map 822 of the controllers 836), location data (e.g., the location of the vehicle 800, e.g., on a map), direction, locations of other vehicles (e.g., an occupancy grid), information about objects and object states as perceived by the controllers 836, and so on. For example, the HMI display 834 can display information about the presence of one or more objects (e.g., street signs, warning signs, traffic light changes, etc.) and / or information about driving maneuvers that the vehicle has made, is making, or will make (e.g., change lanes now, exit 34B in two miles, etc.). Figure 8C

[0092] The vehicle 800 also includes a network interface 824 that can communicate over one or more networks using one or more wireless antennas 826 and / or modems. For example, the network interface 824 can be capable of communicating over LTE, WCDMA, UMTS, GSM, CDMA2000, etc. The one or more wireless antennas 826 can also enable communication between objects (e.g., vehicles, mobile devices, etc.) in an implementation environment using one or more local area networks such as Bluetooth, Bluetooth LE, Z-Wave, ZigBee, etc. and / or one or more low power wide area networks (LPWANs) such as LoRaWAN, SigFox, etc.

[0093] Figure 8B ​To provide an example autonomous vehicle 800 according to some embodiments of the present disclosure Figure 8A An example of camera locations and fields of view of the example autonomous vehicle 800. The cameras and respective fields of view are one example embodiment and are not intended to be limiting. For example, additional and / or alternative cameras can be included, and / or these cameras can be located at different locations on the vehicle 800.

[0094] Camera types for the cameras can include, but are not limited to, digital cameras that can be suitable for use with components and / or systems of the vehicle 800. The cameras can operate at Automotive Safety Integrity Level (ASIL) B and / or at another ASIL. The camera types can have any image capture rate, such as 60 frames per second (fps), 120 fps, 240 fps, and so on, depending on the embodiment. The cameras can be capable of using a rolling shutter, a global shutter, another type of shutter, or a combination thereof. In some examples, a color filter array can include a red-white-white-white (RCCC) color filter array, a red-white-white-blue (RCCB) color filter array, a red-blue-green-white (RBGC) color filter array, a Foveon X3 color filter array, a Bayer sensor (RGGB) color filter array, a monochrome sensor color filter array, and / or another type of color filter array. In some embodiments, clear pixel cameras, such as cameras with a

[0095] In some examples, one or more of the cameras can be used to perform advanced driver assistance system (ADAS) functions (e.g., as part of a redundant or fail-safe design). For example, a multi-functional mono camera can be installed to provide functions including lane departure warning, traffic sign assist, and intelligent headlamp control. One or more (e.g., all) of the cameras can simultaneously record and provide image data (e.g., video).

[0096] One or more of the cameras can be installed in mounting assemblies, such as custom designed (3-D printed) assemblies, in order to cut off stray light and reflections from within the car (e.g., reflections from the dashboard reflected in the windshield mirror) that can interfere with the image data capture capabilities of the cameras. With regard to wing mirror mounting assemblies, the wing mirror assemblies can be custom 3-D printed such that the camera mounting plates match the shape of the wing mirrors. In some examples, one or more cameras can be integrated into the wing mirrors. For side view cameras, one or more cameras can also be integrated into the four pillars at each corner of the cab.

[0097] Cameras with fields of view that include portions of the environment in front of the vehicle 800 (e.g., front-facing cameras) can be used for surround view to help identify a forward path and obstacles, and to assist in providing information critical to generating an occupancy grid and / or determining a preferred vehicle path with the help of one or more controllers 836 and / or control SoCs. Front-facing cameras can be used to perform many of the same ADAS functions as LIDAR, including emergency braking, pedestrian detection, and collision avoidance. Front-facing cameras can also be used for ADAS functions and systems, including lane departure warning (LDW), adaptive cruise control (ACC), and / or other functions such as traffic sign recognition.

[0098] A wide variety of cameras can be used in a front-facing configuration, including, for example, monocular camera platforms including CMOS (complementary metal-oxide semiconductor) color imagers. Another example can be a wide-angle camera 870, which can be used to perceive objects (e.g., pedestrians, intersection traffic, or bicycles) entering the field of view from the periphery. Although Figure 8B Although only one wide-angle camera is illustrated in FIG. 8, any number of wide-angle cameras 870 can be present on the vehicle 800. In addition, long-range cameras 898 (e.g., a pair of long-range stereo cameras) can be used for depth-based object detection, especially for objects for which a neural network has not been trained. Long-range cameras 898 can also be used for object detection and classification and basic object tracking.

[0099] One or more stereo cameras 868 can also be included in a front-facing configuration. Stereo cameras 868 can include an integrated control unit that includes a scalable processing unit that can provide a multi-core microprocessor with integrated CAN or Ethernet interfaces and programmable logic (FPGA) on a single chip. Such a unit can be used to generate a 3-D map of the vehicle’s environment, including distance estimates for all points in the image. Alternative stereo cameras 868 can include a compact stereo vision sensor that can include two camera lenses (one on the left and one on the right) and an image processing chip that can measure the distance from the vehicle to a target object and use the generated information (e.g., metadata) to activate autonomous emergency braking and lane departure warning functions. Other types of stereo cameras 868 can be used in addition to or instead of those described herein.

[0100] Cameras with fields of view that include portions of the environment on the sides of the vehicle 800 (e.g., side-view cameras) can be used for surround view to provide information used to create and update an occupancy grid and to generate side impact collision warnings. For example, surround cameras 874 (e.g., as illustrated in FIG. 8) can be used to provide a 360-degree view of the vehicle’s surroundings. The surround cameras 874 can be used to provide a 360-degree view of the vehicle’s surroundings, which can be used to create and update an occupancy grid and to generate side impact collision warnings. The surround cameras 874 can also be used for ADAS functions and systems, including LDW, ACC, and / or other functions such as traffic sign recognition. Figure 8BFour surround cameras 874) shown in FIG. 8 can be placed on the vehicle 800. The surround cameras 874 can include wide-view cameras 870, fisheye cameras, 360-degree cameras, and / or the like. In one example, four fisheye cameras can be placed on the front, back, and sides of the vehicle. In an alternative arrangement, the vehicle can use three surround cameras 874 (e.g., left, right, and back) and can utilize one or more other cameras (e.g., a forward-facing camera) as a fourth surround camera.

[0101] Cameras with a field of view that includes a portion of the environment behind the vehicle 800 (e.g., rearview cameras) can be used to assist with parking, surround view, rear collision warnings, and creating and updating the occupancy grid. A wide variety of cameras can be used, including but not limited to cameras that are also suitable as front-facing cameras (e.g., long and / or mid-range cameras 898, stereo cameras 868, infrared cameras 872, etc.) as described herein.

[0102] Figure 8C FIG. 8 is a block diagram of an example system architecture for an example autonomous vehicle 800 in accordance with some embodiments of the present disclosure. Figure 8A FIG. 8 is a block diagram of an example system architecture for an example autonomous vehicle 800 in accordance with some embodiments of the present disclosure. It should be understood that this arrangement and other arrangements described herein are set forth merely as examples. Other arrangements and elements (e.g., machines, interfaces, functions, orders, groupings of functions, etc.) can be used in addition to or instead of those shown, and some elements can be omitted altogether. Further, many of the elements described herein are functional entities that can be implemented as discrete or distributed components or in conjunction with other components, and in any suitable combinations and locations. Various functions described herein as being performed by an entity can be implemented if hardware, firmware, and / or software. For instance, various functions can be implemented by a processor executing instructions stored in a memory.

[0103] Figure 8C Each of the components, features, and systems of the vehicle 800 in FIG. 8 is illustrated as being connected via a bus 802. The bus 802 can include a controller area network (CAN) data interface (alternatively referred to herein as a “CAN bus”). The CAN can be a network within the vehicle 800 that is used to assist in controlling various features and functions of the vehicle 800, such as the actuation of brakes, acceleration, braking, steering, windshield wipers, and the like. The CAN bus can be configured to have tens or even hundreds of nodes, each with its own unique identifier (e.g., CAN ID). The CAN bus can be read to find steering wheel angle, ground speed, revolutions per minute (RPM) of the engine, button positions, and / or other vehicle status indicators. The CAN bus can be ASIL B compliant.

[0104] Although bus 802 is described herein as a CAN bus, this is not intended to be limiting. For example, FlexRay and / or Ethernet can be used in addition to or instead of a CAN bus. Further, although bus 802 is represented with a single line, this is not intended to be limiting. For example, there can be any number of buses 802, which can include one or more CAN buses, one or more FlexRay buses, one or more Ethernet buses, and / or one or more other types of buses that use different protocols. In some examples, two or more buses 802 can be used to perform different functions, and / or can be used for redundancy. For example, a first bus 802 can be used for collision avoidance functions, and a second bus 802 can be used for drive control. In any example, each bus 802 can communicate with any component of vehicle 800, and two or more buses 802 can communicate with the same components. In some examples, each SoC 804, each controller 836, and / or each computer within the vehicle can have access to the same input data (e.g., input from sensors of vehicle 800), and can be connected to a common bus, such as a CAN bus.

[0105] Vehicle 800 can include one or more controllers 836, such as those described herein with respect to Figure 8A controllers. Controllers 836 can be used for a wide variety of functions. Controllers 836 can be coupled to any other distinct component and system of vehicle 800, and can be used for control of vehicle 800, artificial intelligence of vehicle 800, infotainment for vehicle 800, and / or the like.

[0106] Vehicle 800 can include one or more system on chips (SoCs) 804. SoCs 804 can include CPUs 806, GPUs 808, processors 810, caches 812, accelerators 814, data stores 816, and / or other components and features not illustrated. In a wide variety of platforms and systems, SoCs 804 can be used to control vehicle 800. For example, one or more SoCs 804 can be in a system (e.g., a system of vehicle 800) in conjunction with HD map 822, which can obtain map refreshes and / or updates from one or more servers (e.g., one or more servers 878) via network interface 824. Figure 8D

[0107] ​CPU 806 can include a CPU cluster or CPU complex (alternatively referred to herein as a“CCPLEX”). CPU 806 can include multiple cores and / or L2 caches. For example, in some embodiments, CPU 806 can include eight cores in a coherent multi-processor configuration. In some embodiments, CPU 806 can include four dual-core clusters with each cluster having a dedicated L2 cache (e.g., a 2 MB L2 cache). CPU 806 (e.g., the CCPLEX) can be configured to support simultaneous cluster operation such that any combination of clusters of CPU 806 can be active at any given time.

[0108] CPU 806 can implement power management capabilities including one or more of the following features: individual hardware blocks can be automatically clock-gated when idle to save dynamic power; each core clock can be gated when the core is not actively executing instructions due to execution of WFI / WFE instructions; each core can be independently power-gated; when all cores are clock-gated or power-gated, each core cluster can be independently clock-gated; and / or when all cores are power-gated, each core cluster can be independently power-gated. CPU 806 can further implement enhanced algorithms for managing power states, with specified allowed power states and desired wake-up times, and the hardware / microcode determines the best power state for the cores, clusters, and CCPLEX to enter. The processing cores can support a simplified power state entry sequence in software, with the work offloaded to microcode.

[0109] GPU 808 can include an integrated GPU (alternatively referred to herein as an“iGPU”). GPU 808 can be programmable and efficient for parallel workloads. In some examples, GPU 808 can use an enhanced tensor instruction set. GPU 808 can include one or more streaming microprocessors, where each streaming microprocessor can include an LI cache (e.g., an LI cache having at least 96 KB of storage capacity), and two or more of the streaming microprocessors can share an L2 cache (e.g., an L2 cache having 512 KB of storage capacity). In some embodiments, GPU 808 can include at least eight streaming microprocessors. GPU 808 can use a compute application programming interface (API). In addition, GPU 808 can use one or more parallel computing platforms and / or programming models (e.g., NVIDIA’s CUDA).

[0110] In automotive and embedded use cases, the GPU 808 can be power-optimized for best performance. For example, the GPU 808 can be fabricated on a fin field-effect transistor (FinFET). However, this is not intended to be limiting, and the GPU 808 can be fabricated using other semiconductor fabrication processes. Each streaming microprocessor can incorporate several mixed-precision processing cores divided into multiple blocks. For example, and without limitation, 64 PF32 cores and 32 PF64 cores can be divided into four processing blocks. In such an example, each processing block can be allocated 16 FP32 cores, 8 FP64 cores, 16 INT32 cores, two mixed-precision NVIDIA Tensor Cores for deep learning matrix arithmetic, an L0 instruction cache, a thread warp scheduler, a dispatch unit, and / or a 64 KB register file. Further, the streaming microprocessor can include independent parallel integer and floating point datapaths to exploit the mix of computation and address computation to provide efficient execution of workloads. The streaming microprocessor can include independent thread scheduling capabilities to allow for more fine-grained synchronization and cooperation between parallel threads. The streaming microprocessor can include a combined LI data cache and shared memory unit to improve performance while simplifying programming.

[0111] The GPU 808 can include a high bandwidth memory (HBM) and / or a 16 GB HBM2 memory subsystem that provides approximately 900 GB / s of peak memory bandwidth in some examples. In some examples, in addition to or alternatively from HBM memory, a synchronous graphics random access memory (SGRAM) such as a fifth generation graphics double data rate synchronous random access memory (GDDR5) can be used.

[0112] The GPU 808 can include a unified memory technology that includes access counters to allow memory pages to be migrated more precisely to the processors that access them most frequently, improving efficiency of memory ranges shared between processors. In some examples, address translation services (ATS) support can be used to allow the GPU 808 to access CPU 806 page tables directly. In such examples, when the GPU 808 memory management unit (MMU) experiences a miss, an address translation request can be transmitted to the CPU 806. In response, the CPU 806 can look up a virtual-to-physical mapping for the address in its page tables and transmit the translation back to the GPU 808. In this way, the unified memory technology can allow a single unified virtual address space for memory of both the CPU 806 and the GPU 808, simplifying GPU 808 programming and porting applications to the GPU 808.

[0113] In addition, GPU 808 can include an access counter that can track how often GPU 808 accesses memory of other processors. The access counter can help ensure that memory pages are moved to the physical memory of the processor that most frequently accesses those pages.

[0114] SoC 804 can include any number of caches 812, including those described herein. For example, caches 812 can include an L3 cache that is available to both CPU 806 and GPU 808 (e.g., connected to both CPU 806 and GPU 808). Caches 812 can include a write-back cache that can track the state of a line, for example, by using a cache coherency protocol (e.g., MEI, MESI, MSI, etc.). Depending on the embodiment, the L3 cache can include 4MB or more, although smaller cache sizes can also be used.

[0115] SoC 804 can include an arithmetic logic unit (ALU) that can be utilized in processing any of a variety of tasks or operations with respect to vehicle 800, such as processing a DNN. In addition, SoC 804 can include a floating point unit (FPU) (or other mathematical co-processor or digital co-processor type) for performing mathematical operations within the system. For example, SoC 104 can include one or more FPUs integrated as execution units within CPU 806 and / or GPU 808.

[0116] SoC 804 can include one or more accelerators 814 (e.g., hardware accelerators, software accelerators, or a combination thereof). For example, SoC 804 can include a hardware accelerator cluster that can include optimized hardware accelerators and / or large on-chip memory. The large on-chip memory (e.g., 4MB SRAM) can enable the hardware accelerator cluster to accelerate neural networks and other computations. The hardware accelerator cluster can be used to supplement GPU 808 and offload some of the tasks of GPU 808 (e.g., freeing up more cycles of GPU 808 for performing other tasks). As one example, accelerators 814 can be used for targeted workloads (e.g., perception, convolutional neural networks (CNNs), etc.) that are stable enough to accelerate easily. As used herein, the term “CNN” can include all types of CNNs, including region-based or region convolutional neural networks (RCNNs) and fast RCNNs (e.g., for object detection).

[0117] The accelerators 814 (e.g., hardware accelerator cluster) can include a deep learning accelerator (DLA). The DLA can include one or more tensor processing units (TPUs) that can be configured to provide an additional 100 billion operations per second for deep learning applications and inferencing. The TPU can be an accelerator that is configured to perform image processing functions (e.g., for CNNs, RCNNs, etc.) and is optimized for performing image processing functions. The DLA can be further optimized for a specific set of neural network types and floating point operations and inferencing. The design of the DLA can provide higher performance per mm than a general purpose GPU and far exceeds the performance of a CPU. The TPU can perform several functions including single instance convolution functions, support for INT8, INT16, and FP16 data types for both features and weights, for example, and post-processor functions.

[0118] The DLA can perform neural networks, especially CNNs, on processed or unprocessed data for any of a wide variety of functions, such as and not by way of limitation: CNNs for object recognition and detection using data from a camera sensor; CNNs for distance estimation using data from a camera sensor; CNNs for emergency vehicle detection and identification and detection using data from a microphone; CNNs for face recognition and vehicle owner identification using data from a camera sensor; and / or CNNs for safety and / or safety related events.

[0119] The DLA can perform any of the functions of the GPU 808, and by using an inferencing accelerator, the designer can target the DLA or the GPU 808 for any function, for example. For example, the designer can focus the processing and floating point operations of the CNNs on the DLA and leave other functions to the GPU 808 and / or other accelerators 814.

[0120] The accelerators 814 (e.g., hardware accelerator cluster) can include a programmable vision accelerator (PVA), which can be alternatively referred to herein as a computer vision accelerator. The PVA can be designed and configured to accelerate computer vision algorithms for advanced driver assistance systems (ADAS), autonomous driving, and / or augmented reality (AR) and / or virtual reality (VR) applications. The PVA can provide a balance between performance and flexibility. For example, each PVA can include any number of reduced instruction set computer (RISC) cores, direct memory access (DMA), and / or any number of vector processors, for example, and not by way of limitation.

[0121] The RISC cores can interact with image sensors (e.g., image sensors of any of the cameras described herein), image signal processors, and / or the like. Each of these RISC cores can include any number of memories. Depending on the embodiment, the RISC cores can use any of several protocols. In some examples, the RISC cores can execute a real-time operating system (RTOS). The RISC cores can be implemented using one or more integrated circuit devices, application specific integrated circuits (ASICs), and / or memory devices. For example, the RISC cores can include instruction caches and / or tightly coupled RAM.

[0122] The DMA can enable components of the PVA to access system memory independently of the CPU 806. The DMA can support any number of features to provide optimization to the PVA, including but not limited to supporting multi-dimensional addressing and / or circular addressing. In some examples, the DMA can support addressing up to six or more dimensions, which can include block width, block height, block depth, horizontal block step, vertical block step, and / or depth step.

[0123] The vector processor can be a programmable processor that can be designed to efficiently and flexibly execute programming for computer vision algorithms and provide signal processing capabilities. In some examples, the PVA can include a PVA core and two vector processing subsystem partitions. The PVA core can include a processor subsystem, one or more DMA engines (e.g., two DMA engines), and / or other peripherals. The vector processing subsystems can operate as the main processing engines of the PVA and can include a vector processing unit (VPU), an instruction cache, and / or a vector memory (e.g., VMEM). The VPU core can include a digital signal processor such as, for example, a single instruction multiple data (SIMD), very long instruction word (VLIW) digital signal processor. The combination of SIMD and VLIW can enhance throughput and speed.

[0124] Each of the vector processors can include an instruction cache and can be coupled to a dedicated memory. As a result, in some examples, each of the vector processors can be configured to execute independently of the other vector processors. In other examples, the vector processors included in a particular PVA can be configured to employ data parallelization. For example, in some embodiments, multiple vector processors included in a single PVA can execute the same computer vision algorithm, but on different regions of an image. In other examples, the vector processors included in a particular PVA can execute different computer vision algorithms on the same image simultaneously, or even different algorithms on a sequence of images or portions of an image. Any number of PVAs can be included in the hardware accelerator cluster, and any number of vector processors can be included in each of the PVAs, among other things. Furthermore, the PVAs can include additional error-correcting code (ECC) memory to enhance overall system security.

[0125] The accelerator 814 (e.g., hardware accelerator cluster) can include an on-chip computer vision network and SRAM to provide high bandwidth, low latency SRAM for the accelerator 814. In some examples, the on-chip memory can include at least 4 MB of SRAM composed of, for example and without limitation, eight field-programmable memory blocks, which can be accessed by both the PVA and the DLA. Each pair of memory blocks can include an advanced peripheral bus (APB) interface, configuration circuitry, a controller, and a multiplexer. Any type of memory can be used. The PVA and the DLA can access the memory via a backbone that provides high-speed memory access to the PVA and the DLA. The backbone can include an on-chip computer vision network that interconnects the PVA and the DLA to the memory, for example using an APB.

[0126] The on-chip computer vision network can include an interface that determines that both the PVA and the DLA provide ready and valid signals before transmitting any control signals / addresses / data. Such an interface can provide separate phases and separate channels for transmitting control signals / addresses / data, as well as burst communications for continuous data transmission. This type of interface can comply with ISO 26262 or IEC 61508 standards, but other standards and protocols can also be used.

[0127] In some examples, the SoC 804 can include a real-time ray tracing hardware accelerator, such as described in U.S. Patent Application No. 16 / 101,232, filed August 10, 2018. The real-time ray tracing hardware accelerator can be used to quickly and efficiently determine locations and extents of objects (e.g., within a world model) in order to generate real-time visualizations simulations for RADAR signal interpretation, for sound propagation synthesis and / or analysis, for SONAR system simulation, for general wave propagation simulation, for comparison with LIDAR data for purposes of localization and / or other functions, and / or for other uses. In some embodiments, one or more tree traversal units (TTUs) can be used to perform one or more ray tracing related operations.

[0128] The accelerator 814 (e.g., hardware accelerator cluster) has a wide range of autonomous driving uses. The PVA can be a programmable vision accelerator that can be used for key processing stages in ADAS and autonomous vehicles. The capabilities of the PVA are a good match for algorithm domains that require predictable processing, low power, and low latency. In other words, the PVA performs well on semi-dense or dense regular computations, and even on small data sets that require predictable runtimes with low latency and low power. Thus, in the context of a platform for autonomous vehicles, the PVA is designed to run classical computer vision algorithms because they are effective at object detection and integer math operations.

[0129] For example, according to one embodiment of the technology, the PVA is used to perform computer stereo vision. In some examples, a semi-global matching based algorithm can be used, although this is not intended to be limiting. Many applications for level 3-5 autonomous driving require instant motion estimation / stereo matching (e.g., structure from motion, pedestrian recognition, lane detection, etc.). The PVA can perform computer stereo vision functions on input from two monocular cameras.

[0130] In some examples, the PVA can be used to perform dense optical flow. Raw RADAR data is processed according to a process (e.g., using a 4D fast Fourier transform) to provide processed RADAR. In other examples, the PVA is used for time-of-flight depth processing, such as by processing raw time-of-flight data to provide processed time-of-flight data.

[0131] The DLA can be used to run any type of network to enhance control and driving safety, including, for example, a neural network that outputs a confidence metric for each object detection. Such a confidence value can be interpreted as a probability, or as providing a relative "weight" for each detection compared to other detections. The confidence value enables the system to make further decisions about which detections should be considered true positive detections and not false positive detections. For example, the system can set a threshold for confidence, and only consider detections that exceed the threshold as true positive detections. In an automatic emergency braking (AEB) system, false positive detections would cause the vehicle to automatically perform an emergency brake, which is obviously undesirable. Thus, only the most confident detections should be considered a trigger for AEB. The DLA can run a neural network for regression of a confidence value. The neural network can take as its input at least some subset of parameters, such as a bounding box dimension, a ground plane estimate obtained (e.g., from another subsystem), inertial measurement unit (IMU) sensor 866 outputs related to vehicle 800 orientation, distance, 3D position estimates of objects obtained from the neural network and / or other sensors (e.g., LIDAR sensor 864 or RADAR sensor 860), etc.

[0132] SoC 804 can include one or more data stores 816 (e.g., memory). Data stores 816 can be on-chip memory of SoC 804, which can store neural networks to be executed on the GPU and / or DLA. In some examples, for redundancy and safety, data stores 816 can be large enough in capacity to store multiple instances of a neural network. Data stores 812 can include L2 or L3 cache 812. References to data stores 816 can include references to memory associated with PVAs, DLAs, and / or other accelerators 814 as described herein.

[0133] SoC 804 can include one or more processors 810 (e.g., embedded processors). The processors 810 can include a boot and power management processor, which can be a specialized processor and subsystem for handling boot power and management functions, as well as security implementation. The boot and power management processor can be part of the SoC 804 boot sequence and can provide run-time power management services. The boot power and management processor can provide clock and voltage programming, auxiliary system low power state transitions, SoC 804 thermal and temperature sensor management, and / or SoC 804 power state management. Each temperature sensor can be implemented as a ring oscillator whose output frequency is proportional to temperature, and the SoC 804 can use the ring oscillator to detect the temperature of the CPU 806, GPU 808, and / or accelerator 814. If it is determined that the temperature exceeds a threshold, the boot and power management processor can enter a temperature fault routine and place the SoC 804 in a lower power state and / or place the vehicle 800 in a driver safe park mode (e.g., safely park the vehicle 800).

[0134] The processors 810 can also include a set of embedded processors that can be used as an audio processing engine. The audio processing engine can be an audio subsystem that allows for full hardware support for multi-channel audio over multiple interfaces, as well as a range of extensive and flexible audio I / O interfaces. In some examples, the audio processing engine is a specialized processor core with a digital signal processor with dedicated RAM.

[0135] The processors 810 can also include an always-on processor engine, which can provide the necessary hardware features to support low-power sensor management and wake-up use cases. The always-on processor engine can include a processor core, tightly coupled RAM, supporting peripherals (e.g., timers and interrupt controllers), various I / O controller peripherals, and routing logic.

[0136] The processors 810 can also include a safety cluster engine, which includes a specialized processor subsystem that handles safety management for automotive applications. The safety cluster engine can include two or more processor cores, tightly coupled RAM, supporting peripherals (e.g., timers, interrupt controllers, etc.), and / or routing logic. In safety mode, the two or more cores can operate in lockstep mode and act as a single core with comparison logic that detects any differences between their operations.

[0137] The processors 810 can also include a real-time camera engine, which can include a specialized processor subsystem for handling real-time camera management.

[0138] The processor 810 can further include a high dynamic range signal processor, which can include an image signal processor, which is a hardware engine that is part of the camera processing pipeline.

[0139] The processor 810 can include a video image compositor, which can be a processing block (e.g., implemented on a microprocessor), that implements video post-processing functions needed by the video playback application to produce the final image for the player window. The video image compositor can perform lens distortion correction on the wide-angle camera 870, surround camera 874, and / or on the cab-in monitor camera sensors. The cab-in monitor camera sensors are preferably monitored by a neural network running on another instance of the advanced SoC, configured to recognize cab-in events and respond accordingly. The cab-in system can perform lip reading to activate mobile phone services and place a call, dictate an email, change the vehicle destination, activate or change the vehicle's infotainment system and settings, or provide voice-activated web surfing. Certain functions are only available to the driver when the vehicle is operating in autonomous mode, and are disabled otherwise.

[0140] The video image compositor can include enhanced temporal noise reduction for spatial and temporal noise reduction. For example, where motion is present in the video, the noise reduction appropriately weights the spatial information, reducing the weight of information provided by neighboring frames. Where the image or portions of the image do not include motion, the temporal noise reduction performed by the video image compositor can use information from previous images to reduce noise in the current image.

[0141] The video image compositor can also be configured to perform stereo correction on input stereo lens frames. The video image compositor can further be used for user interface composition when the operating system desktop is in use and the GPU 808 does not need to continuously render new surfaces. Even when the GPU 808 is powered on and active, doing 3D rendering, the video image compositor can be used to offload the GPU 808 to improve performance and responsiveness.

[0142] The SoC 804 can further include a Mobile Industry Processor Interface (MIPI) camera serial interface for receiving video and input from cameras, a high-speed interface, and / or a video input block that can be used for camera and related pixel input functions. The SoC 804 can further include an input / output controller that can be controlled by software and can be used to receive I / O signals that are not committed to a particular role.

[0143] The SoC 804 can also include a wide range of peripheral device interfaces to enable communication with peripherals, audio codecs, power management, and / or other devices. The SoC 804 can be used to process data from cameras (connected over Gigabit Multimedia Serial Link and Ethernet), sensors (e.g., LIDAR sensors 864, RADAR sensors 860, etc. that can be connected over Ethernet), data from the bus 802 (e.g., speed of the vehicle 800, steering wheel position, etc.), data from GNSS sensors 858 (connected over Ethernet or CAN bus). The SoC 804 can also include dedicated high-performance mass storage controllers, which can include their own DMA engines, and which can be used to free up the CPU 806 from routine data management tasks.

[0144] The SoC 804 can be an end-to-end platform with a flexible architecture that spans automation levels 3-5, providing an integrated functional safety architecture for a platform that leverages and efficiently uses computer vision and ADAS technology to achieve diversity and redundancy, along with deep learning tools. The SoC 804 can be faster, more reliable, and even more energy and space efficient than conventional systems. For example, the accelerators 814, when combined with the CPU 806, GPU 808, and data storage 816, can provide a fast and efficient platform for level 3-5 autonomous vehicles.

[0145] The technology thus provides capabilities and functionality that cannot be achieved by conventional systems. For example, computer vision algorithms can be executed on CPUs that can be configured using high-level programming languages such as the C programming language to perform a wide variety of processing algorithms across a wide variety of visual data. However, CPUs often cannot meet the performance requirements of many computer vision applications, such as those related to, for example, execution time and power consumption. In particular, many CPUs cannot execute complex object detection algorithms in real time, which is a requirement for on-board ADAS applications and for practical level 3-5 autonomous vehicles.

[0146] In contrast to conventional systems, by providing a CPU complex, a GPU complex, and a cluster of hardware accelerators, the technology described herein allows multiple neural networks to be executed simultaneously and / or sequentially, and the results to be combined together to achieve level 3-5 autonomous driving functionality. For example, a CNN executed on a DLA or dGPU (e.g., GPU 820) can include text and word recognition, allowing a supercomputer to read and understand traffic signs, including signs for which a neural network has not been specifically trained. The DLA can also include a neural network that is able to recognize, interpret, and provide a semantic understanding of the sign, and pass that semantic understanding to a path planning module running on the CPU complex.

[0147] As another example, multiple neural networks can be run simultaneously as required for level 3, 4, or 5 driving. For example, a warning sign consisting of the words "Caution: flashing lights indicate icy conditions" along with electric lights can be interpreted by several neural networks independently or collectively. The sign itself can be recognized by a first deployed neural network (e.g., a trained neural network) as a traffic sign, the text "flashing lights indicate icy conditions" can be interpreted by a second deployed neural network that informs the vehicle's path planning software (preferably executing on the CPU complex) that icy conditions exist when flashing lights are detected. The flashing lights can be recognized by operating a third deployed neural network over multiple frames that informs the vehicle's path planning software of the presence (or absence) of flashing lights. All three neural networks can be run simultaneously, for example, within the DLA and / or on the GPU 808.

[0148] In some examples, a CNN for face recognition and owner recognition can use data from the camera sensor to recognize the presence of an authorized driver and / or owner of the vehicle 800. A processing engine always on the sensor can be used to unlock the vehicle and turn on the lights when the owner approaches the driver's door, and in a safe mode, disable the vehicle when the owner leaves the vehicle. In this way, the SoC 804 provides security against theft and / or carjacking.

[0149] In another example, a CNN for emergency vehicle detection and recognition can use data from the microphones 896 to detect and recognize emergency vehicle sirens. In contrast to conventional systems that detect sirens using a general classifier and manually extract features, the SoC 804 uses a CNN to classify ambient and urban sounds as well as to classify visual data. In a preferred embodiment, a CNN running on the DLA is trained to recognize the relative closing speed of an emergency vehicle (e.g., by using the Doppler effect). The CNN can also be trained to recognize emergency vehicles specific to the local area in which the vehicle is operating as recognized by the GNSS sensor 858. Thus, for example, when operating in Europe, the CNN will seek to detect European sirens, and when in the United States, the CNN will seek to recognize sirens that are only North American. Once an emergency vehicle is detected, a control program can be used to execute an emergency vehicle safety routine, slow the vehicle down, pull over to the side of the road, stop the vehicle, and / or idle the vehicle until the emergency vehicle passes, with the assistance of the ultrasonic sensors 862.

[0150] The vehicle can include a CPU 818 (e.g., a discrete CPU or dCPU) that can be coupled to the SoC 804 via a high-speed interconnect (e.g., PCIe). The CPU 818 can include, for example, an X86 processor. The CPU 818 can be used to perform any of a wide variety of functions, including, for example, arbitrating potentially inconsistent results between ADAS sensors and the SoC 804, and / or monitoring the status and health of the controller 836 and / or infotainment SoC 830.

[0151] The vehicle 800 can include a GPU 820 (e.g., a discrete GPU or dGPU) that can be coupled to the SoC 804 via a high-speed interconnect (e.g., NVIDIA’s NVLINK). The GPU 820 can provide additional artificial intelligence functionality, for example, by executing redundant and / or different neural networks, and can be used to train and / or update neural networks based at least in part on input (e.g., sensor data) from sensors of the vehicle 800.

[0152] The vehicle 800 can also include a network interface 824 that can include one or more wireless antennas 826 (e.g., one or more wireless antennas for different communication protocols, such as cellular antennas, Bluetooth antennas, etc.). The network interface 824 can be used to enable wireless connections over the Internet with a cloud (e.g., with a server 878 and / or other network devices), with other vehicles, and / or with computing devices (e.g., client devices of passengers). For communication with other vehicles, a direct link can be established between the two vehicles, and / or an indirect link can be established (e.g., across a network and through the Internet). The direct link can be provided using a car-to-car communication link. The car-to-car communication link can provide the vehicle 800 with information about vehicles that are approaching the vehicle 800 (e.g., vehicles in front of, to the side of, and / or behind the vehicle 800). This functionality can be part of a cooperative adaptive cruise control functionality of the vehicle 800.

[0153] The network interface 824 can include a SoC that provides modulation and demodulation functionality and enables the controller 836 to communicate over a wireless network. The network interface 824 can include a radio frequency front end for up-conversion from baseband to radio frequency and down-conversion from radio frequency to baseband. The frequency conversion can be performed through well-known processes, and / or can be performed using a super-heterodyne process. In some examples, the radio frequency front end functionality can be provided by a separate chip. The network interface can include wireless functionality for communication over LTE, WCDMA, UMTS, GSM, CDMA2000, Bluetooth, Bluetooth LE, Wi-Fi, Z-Wave, ZigBee, LoRaWAN, and / or other wireless protocols.

[0154] The vehicle 800 can also include a data store 828, which can include off-chip (e.g., off-SoC 804) storage. The data store 828 can include one or more storage elements, including RAM, SRAM, DRAM, VRAM, flash memory, hard disks, and / or other components and / or devices that can store data for at least one bit.

[0155] The vehicle 800 can also include a GNSS sensor 858. The GNSS sensor 858 (e.g., GPS, assisted GPS sensor, differential GPS (DGPS) sensor, etc.) is used to assist in mapping, perception, occupancy grid generation, and / or path planning functions. Any number of GNSS sensors 858 can be used, including, for example and without limitation, a GPS using a USB connector with an Ethernet-to-serial (RS-232) bridge.

[0156] The vehicle 800 can also include a RADAR sensor 860. The RADAR sensor 860 can be used by the vehicle 800 for long-range vehicle detection, even in darkness and / or adverse weather conditions. The RADAR functional safety level can be ASIL B. The RADAR sensor 860 can use the CAN and / or the bus 802 (e.g., to transmit data generated by the RADAR sensor 860) for control as well as access to object tracking data, in some examples, Ethernet for access to raw data. A wide variety of RADAR sensor types can be used. For example and without limitation, the RADAR sensor 860 can be suitable for front, rear, and side RADAR use. In some examples, a pulsed Doppler RADAR sensor is used.

[0157] The RADAR sensor 860 can include different configurations, such as long-range with a narrow field of view, short-range with a wide field of view, short-range side coverage, and so on. In some examples, long-range RADAR can be used for adaptive cruise control functionality. A long-range RADAR system can provide a wide field of view (e.g., 250 m range) implemented through two or more independent scans. The RADAR sensor 860 can help distinguish between static and moving objects, and can be used by an ADAS system for emergency brake assist and forward collision warning. A long-range RADAR sensor can include a single-station multi-mode RADAR with multiple (e.g., six or more) fixed RADAR antennas, as well as a high-speed CAN and FlexRay interface. In examples with six antennas, the central four antennas can create focused beam patterns designed to record the surroundings of the vehicle 800 at higher speed with minimal traffic interference from adjacent lanes. The other two antennas can extend the field of view, making it possible to quickly detect vehicles entering or leaving the lane of the vehicle 800.

[0158] As one example, a mid-range RADAR system can include a range of up to 860 m (front) or 80 m (rear) and a field of view of up to 42 degrees (front) or 850 degrees (rear). A short-range RADAR system can include, but is not limited to, RADAR sensors designed to be mounted at both ends of the rear bumper. When mounted at both ends of the rear bumper, such a RADAR sensor system can create two beams that continuously monitor the rear and the blind spot next to the vehicle.

[0159] A short-range RADAR system can be used in an ADAS system for blind spot detection and / or lane change assist.

[0160] The vehicle 800 can also include ultrasonic sensors 862. The ultrasonic sensors 862, which can be placed on the front, rear, and / or sides of the vehicle 800, can be used for parking assist and / or to create and update an occupancy grid. A wide variety of ultrasonic sensors 862 can be used, and different ultrasonic sensors 862 can be used for different detection ranges (e.g., 2.5 m, 4 m). The ultrasonic sensors 862 can operate at an ASIL B functional safety level.

[0161] The vehicle 800 can include LIDAR sensors 864. The LIDAR sensors 864 can be used for object and pedestrian detection, emergency braking, collision avoidance, and / or other functions. The LIDAR sensors 864 can be at an ASIL B functional safety level. In some examples, the vehicle 800 can include multiple LIDAR sensors 864 (e.g., two, four, six, etc.) that can use Ethernet (e.g., to provide data to a Gigabit Ethernet switch).

[0162] In some examples, the LIDAR sensors 864 can be capable of providing a list of objects and their distances for a 360-degree field of view. A commercially available LIDAR sensor 864 can have, for example, an advertised range of approximately 800 m, a precision of 2 cm - 3 cm, and support for an 800 Mbps Ethernet connection. In some examples, one or more flush-mounted LIDAR sensors 864 can be used. In such examples, the LIDAR sensors 864 can be implemented as small devices that can be embedded into the front, rear, sides, and / or corners of the vehicle 800. In such examples, the LIDAR sensors 864 can provide a field of view of up to 120 degrees horizontal and 35 degrees vertical, with a range of 200 m, even for low reflectivity objects. Front-mounted LIDAR sensors 864 can be configured for a horizontal field of view between 45 degrees and 135 degrees.

[0163] In some examples, LIDAR technology such as 3D Flash LIDAR can also be used. 3D Flash LIDAR uses a flash of laser light as a source of emission to illuminate the vehicle’s surroundings up to about 200 m. The flash LIDAR unit includes a receptor that records the laser pulse transmission time and reflected light on each pixel, which in turn corresponds to the range from the vehicle to the object. Flash LIDAR can allow for the generation of highly accurate and distortion-free images of the surroundings with each laser flash. In some examples, four flash LIDAR sensors can be deployed, one on each side of the vehicle 800. Available 3D flash LIDAR systems include solid-state 3D staring array LIDAR cameras (e.g., non-scanning LIDAR devices) that have no moving parts other than fans. The flash LIDAR device can use 5 nanosecond Class I (eye-safe) laser pulses per frame and can capture the reflected laser light in the form of 3D range point clouds and co-registered intensity data. By using flash LIDAR, and because flash LIDAR is a solid-state device with no moving parts, the LIDAR sensor 864 can be less susceptible to motion blur, vibration, and / or jostling.

[0164] The vehicle can also include an IMU sensor 866. In some examples, the IMU sensor 866 can be located at the center of the rear axle of the vehicle 800. The IMU sensor 866 can include, for example and without limitation, an accelerometer, a magnetometer, a gyroscope, a magnetic compass, and / or other sensor types. In some examples, such as in six-axis applications, the IMU sensor 866 can include an accelerometer and a gyroscope, while in nine-axis applications, the IMU sensor 866 can include an accelerometer, a gyroscope, and a magnetometer.

[0165] In some embodiments, the IMU sensor 866 can be implemented as a microelectromechanical systems (MEMS) inertial navigation system (INS) that combines a microelectromechanical systems (MEMS) inertial sensor, a high-sensitivity GPS receiver, and an advanced Kalman filter algorithm to provide estimates of position, velocity, and attitude. As such, in some examples, the IMU sensor 866 can enable the vehicle 800 to estimate heading without input from a magnetic sensor by directly observing the change in velocity from GPS to the IMU sensor 866 and correlating it. In some examples, the IMU sensor 866 and the GNSS sensor 858 can be combined into a single integrated unit.

[0166] The vehicle can include a microphone 896 placed in and / or around the vehicle 800. The microphone 896 can be used for emergency vehicle detection and identification, among other things.

[0167] The vehicle can also include any number of camera types, including stereo cameras 868, wide-view cameras 870, infrared cameras 872, surround-view cameras 874, long and / or mid-range cameras 898, and / or other camera types. These cameras can be used to capture image data around the entire periphery of the vehicle 800. The types of cameras used depend on the embodiment and requirements of the vehicle 800, and any combination of camera types can be used to provide the necessary coverage around the vehicle 800. Further, the number of cameras can vary depending on the embodiment. For example, the vehicle can include six cameras, seven cameras, ten cameras, twelve cameras, and / or another number of cameras. As one example and without limitation, the cameras can support Gigabit Multimedia Serial Link (GMSL) and / or Gigabit Ethernet. Each of the cameras is described in more detail herein with respect to Figure 8A and Figure 8B are described in more detail.

[0168] The vehicle 800 can also include vibration sensors 842. The vibration sensors 842 can measure vibrations of components of the vehicle, such as axles. For example, changes in vibration can indicate changes in the road surface. In another example, when two or more vibration sensors 842 are used, differences between the vibrations can be used to determine the friction or slip of the road surface (e.g., when there is a difference in vibration between a power driven axle and a free spinning axle).

[0169] The vehicle 800 can include an ADAS system 838. In some examples, the ADAS system 838 can include a SoC. The ADAS system 838 can include adaptive / automatic / autonomous cruise control (ACC), cooperative adaptive cruise control (CACC), forward collision warning (FCW), automatic emergency braking (AEB), lane departure warning (LDW), lane keep assist (LKA), blind spot warning (BSW), rear cross-traffic warning (RCTW), collision warning system (CWS), lane centering (LC), and / or other features and functionality.

[0170] The ACC system can use RADAR sensors 860, LIDAR sensors 864, and / or cameras. The ACC system can include longitudinal ACC and / or lateral ACC. Longitudinal ACC monitors and controls the distance to the vehicle immediately ahead of the vehicle 800 and automatically adjusts the vehicle speed to maintain a safe distance from the vehicle ahead. Lateral ACC performs distance keeping and, if necessary, suggests a lane change for the vehicle 800. Lateral ACC is related to other ADAS applications such as LCA and CWS.

[0171] CACC uses information from other vehicles, which can be received from other vehicles via a wireless link via the network interface 824 and / or wireless antenna 826 or indirectly through a network connection, such as through the Internet. Direct links can be provided by a vehicle-to-vehicle (V2V) communication link, while indirect links can be an infrastructure-to-vehicle (I2V) communication link. Generally, the V2V communication concept provides information about the immediately preceding vehicles, such as vehicles immediately ahead of and in the same lane as the vehicle 800, while the I2V communication concept provides information about traffic further ahead. A CACC system can include either or both of I2V and V2V information sources. Given information about vehicles ahead of the vehicle 800, CACC can be more reliable, and it has the potential to improve traffic flow and reduce road congestion.

[0172] FCW systems are designed to alert the driver to a hazard so that the driver can take corrective action. FCW systems use a front-facing camera and / or RADAR sensor 860 coupled to a dedicated processor, DSP, FPGA, and / or ASIC that is electrically coupled to driver feedback such as a display, speaker, and / or vibrating component. FCW systems can provide warnings in the form of, for example, sound, visual warnings, vibrations, and / or quick brake pulses.

[0173] AEB systems detect an impending forward collision with another vehicle or other object and can automatically apply the brakes if the driver does not take corrective action within specified time or distance parameters. AEB systems can use a front-facing camera and / or RADAR sensor 860 coupled to a dedicated processor, DSP, FPGA, and / or ASIC. When an AEB system detects a hazard, it typically first alerts the driver to take corrective action to avoid a collision, and if the driver does not take corrective action, the AEB system can automatically apply the brakes in an effort to prevent or at least mitigate the effects of a predicted collision. AEB systems can include technologies such as dynamic brake support and / or crash imminent braking.

[0174] LDW systems provide visual, audible, and / or tactile warnings, such as steering wheel or seat vibrations, to alert the driver when the vehicle 800 is crossing lane markers. The LDW system is not activated when the driver indicates an intentional lane departure by activating a turn signal. LDW systems can use a front-side facing camera coupled to a dedicated processor, DSP, FPGA, and / or ASIC that is electrically coupled to driver feedback such as a display, speaker, and / or vibrating component.

[0175] An LKA system is a variation of the LDW system. If the vehicle 800 begins to leave the lane, the LKA system provides a steering input or brake to correct the vehicle 800.

[0176] A BSW system detects and warns the driver of vehicles in the car's blind spot. The BSW system can provide visual, audible, and / or tactile alerts to indicate that merging or changing lanes is unsafe. The system can provide additional warnings when the driver uses a turn signal. The BSW system can use rear side-facing cameras and / or RADAR sensors 860 coupled to a dedicated processor, DSP, FPGA, and / or ASIC that is electrically coupled to driver feedback such as a display, speaker, and / or vibrating component.

[0177] A RCTW system can provide visual, audible, and / or tactile notifications when objects are detected outside the range of the rear-facing camera while the vehicle 800 is backing up. Some RCTW systems include AEB to ensure that vehicle brakes are applied to avoid a collision. The RCTW system can use one or more rear-facing RADAR sensors 860 coupled to a dedicated processor, DSP, FPGA, and / or ASIC that is electrically coupled to driver feedback such as a display, speaker, and / or vibrating component.

[0178] Conventional ADAS systems can be prone to false positive results, which can annoy and distract the driver, but typically are not catastrophic because the ADAS system alerts the driver and allows the driver to decide whether the safety condition is truly present and act accordingly. However, in an autonomous vehicle 800, in the case of conflicting results, the vehicle 800 itself must decide whether to heed the results from the primary computer or the secondary computer (e.g., the first controller 836 or the second controller 836). For example, in some embodiments, the ADAS system 838 can be a secondary and / or auxiliary computer for providing perception information to a backup computer plausibility module. The backup computer plausibility monitor can run redundant diverse software on hardware components to detect faults in perception and dynamic driving tasks. The output from the ADAS system 838 can be provided to a supervisory MCU. If the outputs from the primary computer and the secondary computer conflict, the supervisory MCU must determine how to reconcile the conflict to ensure safe operation.

[0179] In some examples, the host computer can be configured to provide a confidence score to the supervisory MCU indicating the host computer's confidence in the selected result. If the confidence score exceeds a threshold, then the supervisory MCU can follow the host computer's direction, regardless of whether the secondary computer provides conflicting or inconsistent results. In the event that the confidence score does not satisfy the threshold and in the event that the host computer and the secondary computer indicate different results (e.g., a conflict), the supervisory MCU can arbitrate between the computers to determine the appropriate result.

[0180] The supervisory MCU can be configured to run a neural network that is trained and configured to determine conditions under which the secondary computer provides false alarms based at least in part on the output from the host computer and the secondary computer. Thus, the neural network in the supervisory MCU can learn when the output of the secondary computer can be trusted and when it cannot. For example, when the secondary computer is a RADAR-based FCW system, the neural network in the supervisory MCU can learn when the FCW system is identifying metal objects that are not in fact dangerous, such as drain grates or manhole covers that trigger false alarms. Similarly, when the secondary computer is a camera-based LDW system, the neural network in the supervisory MCU can learn to disregard the LDW when a cyclist or pedestrian is present and lane departure is in fact the safest strategy. In embodiments that include a neural network running on the supervisory MCU, the supervisory MCU can include at least one of a DLA or a GPU suitable for running the neural network with associated memory. In preferred embodiments, the supervisory MCU can include and / or be included as a component of the SoC 804.

[0181] In other examples, the ADAS system 838 can include a secondary computer that performs ADAS functions using traditional computer vision rules. In this way, the secondary computer can use classic computer vision rules (if-then), and the presence of a neural network in the supervisory MCU can improve reliability, safety, and performance. For example, the diverse implementation and intentional non-identity make the overall system more fault-tolerant, especially with respect to faults caused by software (or software-hardware interface) functions. For example, if there is a software bug or error in the software running on the host computer and the non-identical software code running on the secondary computer provides the same overall result, then the supervisory MCU can be more confident that the overall result is correct and that the bug in the software or hardware on the host computer did not cause a substantial error.

[0182] In some examples, the output of the ADAS system 838 can be fed to a perception block of the host computer and / or a dynamic driving task block of the host computer. For example, if the ADAS system 838 indicates a forward collision warning due to an object immediately ahead, the perception block can use this information in identifying the object. In other examples, the secondary computer can have its own neural network that is trained and thus reduces the risk of false positives as described herein.

[0183] The vehicle 800 can also include an infotainment SoC 830 (e.g., an in-vehicle infotainment system (IVI)). Although illustrated and described as a SoC, the infotainment system can not be a SoC and can include two or more discrete components. The infotainment SoC 830 can include a combination of hardware and software that can be used to provide audio (e.g., music, personal digital assistant, navigation instructions, news, radio, etc.), video (e.g., TV, movies, streaming, etc.), telephony (e.g., hands-free calling), network connectivity (e.g., LTE, WiFi, etc.), and / or information services (e.g., navigation system, park assist, radio data system, vehicle related information such as fuel level, total distance covered, brake fuel level, oil level, doors open / closed, air filter information, etc.) to the vehicle 800. For example, the infotainment SoC 830 can include a radio, a disc player, a navigation system, a video player, USB and Bluetooth connectivity, an in-car computer, in-car entertainment, WiFi, steering wheel audio controls, hands-free voice controls, a heads-up display (HUD), the HMI display 834, a telematics device, a control panel (e.g., for controlling and / or interacting with various components, features, and / or systems), and / or other components. The infotainment SoC 830 can further be used to provide information (e.g., visual and / or audible) to a user of the vehicle, such as information from the ADAS system 838, autonomous driving information such as planned vehicle maneuvers, trajectories, surrounding environment information (e.g., intersection information, vehicle information, road information, etc.), and / or other information.

[0184] The infotainment SoC 830 can include GPU functionality. The infotainment SoC 830 can communicate with other devices, systems, and / or components of the vehicle 800 over the bus 802 (e.g., a CAN bus, Ethernet, etc.). In some examples, the infotainment SoC 830 can be coupled to a supervisory MCU such that, in the event of a failure of the host controller 836 (e.g., a primary and / or backup computer of the vehicle 800), the GPU of the infotainment system can perform some autonomous driving functions. In such examples, the infotainment SoC 830 can place the vehicle 800 in a driver safe park mode as described herein.

[0185] Vehicle 800 may also include an instrument cluster 832 (e.g., a digital instrument panel, electronic instrument cluster, digital instrument panel, etc.). The instrument cluster 832 may include a controller and / or a supercomputer (e.g., a discrete controller or supercomputer). The instrument cluster 832 may include a set of instruments such as a speedometer, fuel level, oil pressure, tachometer, odometer, turn indicator, shift position indicator, seatbelt warning light, parking brake warning light, engine malfunction indicator, airbag (SRS) system information, lighting controls, safety system controls, navigation information, etc. In some examples, information may be displayed and / or shared between the infotainment SoC 830 and the instrument cluster 832. In other words, the instrument cluster 832 may be included as part of the infotainment SoC 830, or vice versa.

[0186] Figure 8D For cloud-based servers and according to some embodiments of this disclosure Figure 8A This is a system diagram illustrating communication between example autonomous vehicles 800. System 876 may include server 878, network 890, and vehicles including vehicle 800. Server 878 may include multiple GPUs 884(A)-884(H) (collectively referred to herein as GPU 884), PCIe switches 882(A)-882(H) (collectively referred to herein as PCIe switch 882), and / or CPUs 880(A)-880(B) (collectively referred to herein as CPU 880). GPUs 884, CPUs 880, and PCIe switches may be interconnected with high-speed interconnects and / or PCIe connections 886, such as, but not limited to, NVLink interface 888 developed by NVIDIA. In some examples, GPUs 884 are connected via NVLink and / or NVSwitch SoCs, and GPUs 884 and PCIe switches 882 are connected via PCIe interconnects. Although eight GPUs 884, two CPUs 880, and two PCIe switches are shown in the diagram, this is not intended to be limiting. Depending on the embodiment, each of the servers 878 may include any number of GPUs 884, CPUs 880, and / or PCIe switches. For example, each of the servers 878 may include eight, sixteen, thirty-two, and / or more GPUs 884.

[0187] The server 878 can receive image data from vehicles over the network 890 and representing images showing unexpected or changing road conditions such as a road work that recently started. The server 878 can transmit neural networks 892, updated neural networks 892, and / or map information 894, including information about traffic and road conditions, to vehicles over the network 890. Updates to the map information 894 can include updates to the HD map 822, e.g., information about construction sites, potholes, curves, floods, or other obstacles. In some examples, the neural networks 892, updated neural networks 892, and / or map information 894 can have been generated from experience using training performed at a data center (e.g., using the server 878 and / or other servers) and / or from data received from any number of vehicles in the environment.

[0188] The server 878 can be used to train machine learning models (e.g., neural networks) based on training data. The training data can be generated by vehicles and / or can be generated in simulations (e.g., using game engines). In some examples, the training data is labeled (e.g., in cases where the neural network benefits from supervised learning) and / or undergoes other pre-processing, while in other examples, the training data is not labeled and / or pre-processed (e.g., in cases where the neural network does not require supervised learning). The training can be performed according to any one or more categories of machine learning techniques, including but not limited to categories such as: supervised training, semi-supervised training, unsupervised training, self-learning, reinforcement learning, federated learning, transfer learning, feature learning (including principal component and cluster analysis), multilinear subspace learning, manifold learning, representation learning (including spare dictionary learning), rule-based machine learning, anomaly detection, and any variants or combinations thereof. Once the machine learning models are trained, the machine learning models can be used by vehicles (e.g., transmitted to vehicles over the network 890), and / or the machine learning models can be used by the server 878 to remotely monitor vehicles.

[0189] In some examples, the server 878 can receive data from vehicles and apply the data to the latest real-time neural networks for real-time intelligent inference. The server 878 can include deep learning supercomputers and / or specialized AI computers powered by GPUs 884, such as the DGX and DGX Station machines developed by NVIDIA. However, in some examples, the server 878 can include deep learning infrastructure of a data center that is powered by CPUs only.

[0190] The deep learning infrastructure of the server 878 can be capable of fast real-time inference, and can use this capability to assess and validate the health of the processors, software, and / or associated hardware in the vehicle 800. For example, the deep learning infrastructure can receive periodic updates from the vehicle 800, such as a sequence of images and / or objects located in the sequence of images that the vehicle 800 has located (e.g., via computer vision and / or other machine learning object classification techniques). The deep learning infrastructure can run its own neural network to identify the objects and compare them to the objects identified by the vehicle 800, and if the results do not match and the infrastructure concludes that the AI in the vehicle 800 is malfunctioning, then the server 878 can transmit a signal to the vehicle 800 instructing the fail-safe computer of the vehicle 800 to take control, notify the passengers, and complete a safe parking operation.

[0191] For inference, the server 878 can include GPUs 884 and one or more programmable inference accelerators (such as NVIDIA’s TensorRT). The combination of GPU-powered servers and inference-accelerated can make real-time responses possible. In other examples, such as where performance is less important, CPU, FPGA, and other processor-powered servers can be used for inference.

[0192] Example Computing Device

[0193] Figure 9 is a block diagram of an example computing device 900 suitable for implementing some embodiments of the present disclosure. The computing device 900 can include an interconnection system 902 coupling the following devices: a memory 904, one or more central processing units (CPUs) 906, one or more graphics processing units (GPUs) 908, a communication interface 910, input / output (I / O) ports 912, I / O components 914, a power supply 916, one or more presentation components 918 (e.g., display(s)), and one or more logic units 920. In at least one embodiment, the computing device(s) 900 can include one or more virtual machines (VMs), and / or any component thereof can include a virtual component (e.g., a virtual hardware component). For non-limiting examples, one or more of the GPUs 908 can include one or more vGPUs, one or more of the CPUs 906 can include one or more vCPUs, and / or one or more of the logic units 920 can include one or more virtual logic units. As such, the computing device(s) 900 can include discrete components (e.g., a full GPU dedicated to the computing device 900), virtual components (e.g., a portion of a GPU dedicated to the computing device 900), or a combination thereof.

[0194] AlthoughFigure 9 Various blocks of the computing device are shown by way of example as connected one to the other by way of the interconnect system 902, but this is not intended to be limiting and is for clarity only. For example, in some embodiments, a presentation component 918, such as a display device, can be considered an I / O component 914 (e.g., if the display is a touch screen). As another example, the CPU 906 and / or GPU 908 can include memory (e.g., the memory 904 can represent a storage device in addition to the memory of the GPU 908, CPU 906, and / or other components). In other words, Figure 9 The computing device of FIG. 1 is illustrative. No distinction is made between devices of such categories as “workstations,” “servers,” “laptops,” “desktops,” “tablet computers,” “client devices,” “mobile devices,” “handheld devices,” “gaming consoles,” “electronic control units (ECUs),” “virtual reality systems,” and / or other device or system types, as all are considered within the scope of the computing device of FIG. 1. Figure 9 The computing device of FIG. 1 is illustrative. No distinction is made between devices of such categories as “workstations,” “servers,” “laptops,” “desktops,” “tablet computers,” “client devices,” “mobile devices,” “handheld devices,” “gaming consoles,” “electronic control units (ECUs),” “virtual reality systems,” and / or other device or system types, as all are considered within the scope of the computing device of FIG. 1.

[0195] The interconnect system 902 can represent one or more links or buses, such as an address bus, a data bus, a control bus, or a combination thereof. The interconnect system 902 can include one or more bus or link types, such as an Industry Standard Architecture (ISA) bus, an Extended Industry Standard Architecture (EISA) bus, a Video Electronics Standards Association (VESA) bus, a Peripheral Component Interconnect (PCI) bus, a Peripheral Component Interconnect Express (PCIe) bus, and / or another type of bus or link. In some embodiments, there are direct connections between components. As an example, the CPU 906 can be directly connected to the memory 904. Further, the CPU 906 can be directly connected to the GPU 908. Where there are direct or point-to-point connections between components, the interconnect system 902 can include a PCIe link to perform the connection. In these examples, a PCI bus need not be included in the computing device 900.

[0196] The memory 904 can include any of a variety of computer-readable media. Computer-readable media can be any available media that can be accessed by computing device 900. By way of example, and not limitation, computer-readable media can comprise computer storage media and communication media.

[0197] Computer storage media can include volatile and nonvolatile media and / or removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules, and / or other data types. For example, memory 904 can store computer readable instructions (e.g., representing program(s) and / or program element(s), such as an operating system). Computer storage media can include, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can be accessed by computing device 900. As used herein, computer storage media does not include signals per se.

[0198] Computer storage media can embody computer readable instructions, data structures, program modules, and / or other data types in a modulated data signal, such as a carrier wave or other transport mechanism, and includes any information delivery media. The term "modulated data signal" can refer to a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. By way of example, and not limitation, computer storage media can include wired media such as a wired network or direct-wired connection, and wireless media such as acoustic, RF, infrared, and other wireless media. Combinations of the any of the above should also be included within the scope of computer readable media.

[0199] CPUs 906 can be configured to execute at least some of the computer readable instructions to control one or more components of computing device 900 to perform one or more of the methods and / or processes described herein. CPUs 906 can each include one or more cores (e.g., one, two, four, eight, twenty-eight, seventy-two, etc.) capable of handling multiple software threads concurrently. CPUs 906 can include any type of processors and can include different types of processors depending on the type of computing device 900 being implemented (e.g., a mobile device having fewer cores for processors and a server having more cores for processors). For example, depending on the type of computing device 900, the processors can be Advanced RISC Machines (ARM) processors implemented using Reduced Instruction Set Computing (RISC) or x86 processors implemented using Complex Instruction Set Computing (CISC). Computing device 900 can include one or more CPUs 906 in addition to, or as an alternative to, one or more microprocessors or co-processors such as mathematical co-processors.

[0200] In addition to or in place of the CPU(s) 906, the GPU(s) 908 can be configured to execute at least some of the computer-readable instructions to control one or more components of the computing device 900 to perform one or more of the methods and / or processes described herein. One or more of the GPU(s) 908 can be integrated GPUs (e.g., with one or more of the CPU(s) 906) and / or one or more of the GPU(s) 908 can be discrete GPUs. In embodiments, one or more of the GPU(s) 908 can be a co-processor to one or more of the CPU(s) 906. The GPU(s) 908 can be used by the computing device 900 to render graphics (e.g., 3D graphics) or to perform general-purpose computing. For example, the GPU(s) 908 can be used for general-purpose computing on GPUs (GPGPU). The GPU(s) 908 can include hundreds or thousands of cores capable of handling hundreds or thousands of software threads concurrently. The GPU(s) 908 can generate pixel data for an output image in response to rendering commands (e.g., received from the CPU(s) 906 via a host interface). The GPU(s) 908 can include graphics memory (e.g., display memory) for storing pixel data or any other suitable data (e.g., GPGPU data). The display memory can be included as part of the memory 904. The GPU(s) 908 can include two or more GPUs operating in parallel (e.g., via a link). The link can connect the GPUs directly (e.g., using NVLINK) or can connect the GPUs through a switch (e.g., using an NVSwitch). When combined together, each GPU 908 can generate pixel data or GPGPU data for a different portion of an output or for a different output (e.g., a first GPU for a first image and a second GPU for a second image). Each GPU can include its own memory or can share memory with other GPUs.

[0201] In addition to or in place of CPU 906 and / or GPU 908, logic units 920 can be configured to execute at least some of the computer-readable instructions to control one or more components of computing device 900 to perform one or more of the methods and / or processes described herein. In embodiments, CPU(s) 906, GPU(s) 908, and / or logic unit(s) 920 can perform any combination of the methods, processes, and / or portions thereof, discretely or jointly. One or more of logic units 920 can be part of and / or integrated with one or more of CPU 906 and / or GPU 908, and / or one or more of logic units 920 can be discrete components or otherwise external to CPU 906 and / or GPU 908. In embodiments, one or more of logic units 920 can be a co-processor of one or more of CPU 906 and / or GPU 908.

[0202] Examples of logic units 920 include one or more processing cores and / or components thereof, such as tensor cores (TCs), tensor processing units (TPUs), pixel visual cores (PVCs), visual processing units (VPUs), graphics processing clusters (GPCs), texture processing clusters (TPCs), streaming multi-processors (SMs), tree traversal units (TTUs), artificial intelligence accelerators (AIAs), deep learning accelerators (DLAs), arithmetic logic units (ALUs), application-specific integrated circuits (ASICs), floating point units (FPUs), input / output (I / O) elements, peripheral component interconnect (PCI) or peripheral component interconnect express (PCIe) elements, and the like.

[0203] Communication interface 910 can include one or more receivers, transmitters, and / or transceivers that enable computing device 900 to communicate with other computing devices via electronic communication networks, including wired and / or wireless communications. Communication interface 910 can include components and functionality to enable communication over any of a number of different networks, such as wireless networks (e.g., Wi-Fi, Z-Wave, Bluetooth, Bluetooth LE, ZigBee, etc.), wired networks (e.g., through Ethernet or InfiniBand), low power wide area networks (e.g., LoRaWAN, SigFox, etc.), and / or the Internet.

[0204] I / O ports 912 can enable the computing device 900 to logically couple to other devices including I / O components 914, presentation components 918, and / or other components, some of which can be built in to (e.g., integral with) the computing device 900. Illustrative I / O components 914 include a microphone, mouse, keyboard, joystick, game pad, game controller, satellite dish, scanner, printer, wireless device, etc. The I / O components 914 can provide a natural user interface (NUI) that processes air gestures, voice, or other physiological inputs generated by a user. In some instances, inputs can be transmitted to appropriate network elements for further processing. A NUI can implement any combination of speech recognition, gesture recognition, facial recognition, biometric recognition, posture recognition, gesture recognition within a

[0205] A power supply 916 can include a hard-wired power supply, a battery power supply, or a combination thereof. The power supply 916 can provide power to the computing device 900 to enable the components of the computing device 900 to operate.

[0206] The presentation components 918 can include a display (e.g., a monitor, a touch screen, a television, a heads-up display (HUD), other display types, or a combination thereof), speakers, and / or other presentation components. The presentation components 918 can receive data from other components (e.g., the GPU 908, the CPU 906, etc.) and output the data (e.g., as a

[0207] Example data center

[0208] Figure 10 An example data center 1000 that can be used in at least one embodiment of the present disclosure is shown. The data center 1000 can include a data center infrastructure layer 1010, a framework layer 1020, a software layer 1030, and / or an application layer 1040.

[0209] As Figure 10As shown, the data center infrastructure layer 1010 can include a resource orchestrator 1012, grouped computing resources 1014, and node computing resources ("node C.R.s") 1016(1)-1016(N), where "N" represents any whole, positive integer. In at least one embodiment, the node C.R.s 1016(1)-1016(N) can include, but are not limited to, any number of central processing units ("CPUs" or "processors") including accelerators, field programmable gate arrays (FPGAs), graphics processors or graphics processing units (GPUs), memory devices (e.g., dynamic random access memory), storage devices (e.g., solid state or disk drives), network input / output ("NW I / O") devices, network switches, virtual machines ("VMs"), power modules, and / or cooling modules, and the like. In some embodiments, one or more of the node C.R.s 1016(1)-1016(N) can correspond to a server having one or more of the above-described computing resources. Moreover, in some embodiments, the node C.R.s 1016(1)-1016(N) can include one or more virtual components, such as a vGPU, a vCPU, etc., and / or one or more of the node C.R.s 1016(1)-1016(N) can correspond to a virtual machine (VM).

[0210] In at least one embodiment, the grouped computing resources 1014 can include individual groups of node C.R.s 1016 housed within one or more racks (not shown) or housed within a number of racks at different geographic locations (also not shown) within a data center. The individual groups of node C.R.s 1016 within the grouped computing resources 1014 can include grouped computing, network, memory, or storage resources that can be configured or allocated to support one or more workloads. In at least one embodiment, a number of node C.R.s 1016 including CPUs, GPUs, and / or other processors can be grouped within one or more racks to provide computing resources to support one or more workloads. The one or more racks can also include any quantity of power modules, cooling modules, and / or network switches in any combination.

[0211] The resource orchestrator 1022 can configure or otherwise control the one or more node C.R.s 1016(1)-1016(N) and / or the grouped computing resources 1014. In at least one embodiment, the resource orchestrator 1022 can include a software design infrastructure ("SDI") management entity for the data center 1000. The resource orchestrator 1022 can comprise hardware, software, or some combination thereof.

[0212] In at least one embodiment, as Figure 10As shown, framework layer 1020 may include a job scheduler 1048, a configuration manager 1034, a resource manager 1036, and / or a distributed file system 1038. Framework layer 1020 may include a framework for software 1048 supporting software layer 1030 and / or one or more applications 1042 supporting application layer 1040. Software 1048 or application 1042 may respectively contain web-based service software or applications, such as those provided by Amazon Web Services, Google Cloud, and Microsoft Azure. Framework layer 1020 may be, but is not limited to, free and open-source software web application frameworks (such as Apache Spark) that can utilize distributed file system 1038 for large-scale data processing (e.g., "big data"). TM (Hereinafter referred to as "Spark") is a type of resource. In at least one embodiment, the job scheduler 1032 may include Spark drivers to facilitate the scheduling of workloads supported by different layers of data center 1000. The configuration manager 1034 may be able to configure different layers, such as the software layer 1030 and the framework layer 1020 (which includes Spark and a distributed file system 1038 for supporting large-scale data processing). The resource manager 1036 may be able to manage computing resources mapped to or allocated to clusters of distributed file system 1038 and job scheduler 1032 to support distributed file system 1038 and job scheduler 1032. In at least one embodiment, the clustered or grouped computing resources may include grouped computing resources 1014 in the data center infrastructure layer 1010. The resource manager 1036 may coordinate with the resource coordinator 1012 to manage these mapped or allocated computing resources.

[0213] In at least one embodiment, the software 1032 included in the software layer 1030 may include software used in at least a portion of the nodes CRs 1016(1)-1016(N), the grouped computing resources 1014, and / or the distributed file system 1038 of the framework layer 1020. One or more types of software may include, but are not limited to, internet web search software, email virus scanning software, database software, and streaming video content software.

[0214] In at least one embodiment, applications 1042 included in application layer 1040 can include one or more types of applications used by at least portions of node C.R.s 1016(1)-1016(N), grouped computing resources 1014, and / or distributed file systems 1038 of framework layer 1020. One or more types of applications can include, but are not limited to, any number of genomics applications, cognitive computing and machine learning applications including training or inference software, machine learning framework software (e.g., PyTorch, TensorFlow, Caffe, etc.), and / or other machine learning applications used in conjunction with one or more embodiments.

[0215] In at least one embodiment, any of configuration manager 1034, resource manager 1036, and resource orchestrator 1012 can implement any number and type of self-modification actions based on any quantity and type of data acquired in any technically feasible manner. Self-modification actions can free data center operators of data center 1000 from making potentially poor configuration decisions and can avoid underutilization and / or poor performance portions of a data center.

[0216] According to one or more embodiments described herein, data center 1000 can include tools, services, software, or other resources to train one or more machine learning models or use one or more machine learning models to predict or infer information. For example, machine learning model(s) can be trained by computing weight parameters according to a neural network architecture using software and / or computing resources described above with respect to data center 1000. In at least one embodiment, trained or deployed machine learning models corresponding to one or more neural networks can be used to infer or predict information using resources described above with respect to data center 1000 by using weight parameters computed through one or more training techniques such as, but not limited to, those described herein.

[0217] In at least one embodiment, data center 1000 can use CPUs, application specific integrated circuits (ASICs), GPUs, FPGAs, and / or other hardware (or virtual computing resources corresponding thereto) to perform training and / or inference using resources described above. Further, one or more software and / or hardware resources described above can be configured as a service that allows users to train or perform inference on information, such as image recognition, speech recognition, or other artificial intelligence services.

[0218] Example network environment

[0219] Network environments suitable for implementing embodiments of the present disclosure can include one or more client devices, servers, network-attached storage (NAS), other backend devices, and / or other device types. The client devices, servers, and / or other device types (e.g., each device) can be implemented on one or more instances of computing device(s) 900 - e.g., each device can include similar components, features, and / or functionality of computing device(s) 900. Further, where backend devices (e.g., servers, NAS, etc.) are implemented, the backend devices can be included as part of a data center 1000, an example of which is described in more detail herein with respect to FIG. 1, and can be implemented on one or more instances of computing device(s) 900. Figure 9 Figure 10

[0220] Components of the network environment can communicate with each other via a network, which can be wired, wireless, or both. The network can include multiple networks or one of multiple networks. For example, the network can include one or more wide area networks (WANs), one or more local area networks (LANs), one or more public networks (such as the Internet and / or the public switched telephone network (PSTN)), and / or one or more private networks. Where the network includes a wireless telecommunication network, components such as base stations, communication towers, or even access points (among other components) can provide wireless connectivity.

[0221] Compatible network environments can include one or more peer-to-peer network environments (in which case servers can not be included in the network environment) and one or more client-server network environments (in which case one or more servers can be included in the network environment). In a peer-to-peer network environment, functionality described herein for servers can be implemented on any number of client devices.

[0222] In at least one embodiment, the network environment can include one or more cloud-based network environments, distributed computing environments, combinations thereof, and the like. A cloud-based network environment can include a framework layer, a job scheduler, a resource manager, and a distributed file system implemented on one or more servers, which can include one or more core network servers and / or edge servers. The framework layer can include a framework that supports one or more applications of a software layer and / or an application layer. The software or applications can include network-based service software or applications, respectively. In embodiments, one or more client devices can use the network-based service software or applications (e.g., by accessing the service software and / or applications via one or more application programming interfaces (APIs)). The framework layer can be, without limitation, a free and open-source software web application framework as can be used for large-scale data processing (e.g., “big data”) using a distributed file system. ​​

[0223] The cloud-based network environment can provide cloud computing and / or cloud storage that performs any combination of the computing and / or data storage functions described herein (or one or more portions thereof). Any of these different functions can be distributed across multiple locations from a central or core server (e.g., can be distributed across one or more data centers in a state, region, country, globally, etc.). The core server can designate at least a portion of the functions to an edge server if the connection to the user (e.g., client device) is relatively close to the edge server. The cloud-based network environment can be private (e.g., limited to a single organization), public (e.g., available to many organizations), and / or a combination thereof (e.g., a hybrid cloud environment).

[0224] The client device(s) can include at least some of the components, features, and functionality of the example computing device 900 described herein with respect to Figure 9 As examples and not by way of limitation, a client device can be implemented as a personal computer (PC), laptop computer, mobile device, smartphone, tablet computer, smartwatch, wearable computer, personal digital assistant (PDA), MP3 player, virtual reality headset, global positioning system (GPS) or device, video player, video camera, surveillance device or system, vehicle, boat, spaceship, virtual machine, drone, robot, handheld communication device, hospital device, gaming device or system, entertainment system, vehicle computer system, embedded system controller, remote control, appliance, consumer electronic device, workstation, edge device, any combination of these depicted devices, or any other suitable device.

[0225] The present disclosure can be described in the general context of machine-usable instructions or computer code, including computer-executable instructions such as program modules, being executed by a computer or other machine, such as a personal data assistant or other handheld device. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform particular tasks or implement particular abstract data types. The present disclosure can be practiced in a variety of system configurations, including hand-held devices, consumer electronics, general- purpose computers, more specialty computing devices, etc. The present disclosure can also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network.

[0226] As used herein, the term "and / or," with respect to a listing of two or more items, means that one, including only one of the items, or combinations of the items, can be employed. For example, "A and / or B" can include A alone, B alone, or A and B. Likewise, "at least one of A or B" can include A alone, B alone, or at least one of A and at least one of B. Further, "at least one of A and B" can include at least one of A, at least one of B, or at least one of A and at least one of B.

[0227] The subject matter of the present disclosure is described with specificity herein to meet statutory requirements. However, the description itself is not intended to limit the scope of this disclosure. Rather, the present disclosure has been presented for the purpose of illustration and description so as to enable others, skilled in the art, to employ the subject matter of the present disclosure in various ways. Furthermore, although the terms "step" and / or "block" can be used herein to connote different elements of methods employed, the terms should not be interpreted as implying any particular order among or between various steps herein disclosed unless and except the order of individual steps is explicitly described.

Claims

1. A method comprising: detecting a power-off indication corresponding to a machine; based at least on detecting the power-off indication, performing a diagnosis on a computer system that performs one or more autonomous control operations of the machine; based at least on performing the diagnosis, restarting one or more portions of the computer system; storing a state of a computing environment corresponding to the computer system in computer storage as a saved state and based on at least one of the restarting; entering a low-power mode based at least on storing the state of the computing environment; detecting a power-on indication corresponding to the machine; exiting the low-power mode based at least on detecting the power-on indication; and based at least on exiting the low-power mode, restoring the state of the computing environment corresponding to the computer system to the saved state of the computer storage. The one or more portions of the computer system are restarted prior to running the diagnosis based at least on detecting the power-off indication of the machine.

2. The method of claim 1, further comprising:

3. The method of claim 1, wherein the state of the computing environment is a first state, and the method further comprises: based at least on one or more criteria being met, exiting the low-power mode prior to the power-on indication; re-running the diagnosis on the computer system; restarting the computer system to generate a second state of the computing environment; storing the second state of the computing environment in the computer storage as a second saved state; and re-entering the low-power mode, wherein restoring the state of the computing environment comprises restoring the state of the computing environment corresponding to the computer system to the second state.

4. The method of claim 1, wherein the computer storage comprises a volatile memory.

5. The method of claim 1, wherein the machine comprises an energy storage medium that provides power to the computer system, wherein the machine is at least one of: at least partially electrically powered; or at least partially internal combustion engine powered.

6. The method of claim 1, wherein the machine comprises one or more sensors, wherein the low-power mode is characterized by at least one of the one or more sensors not receiving power.

7. The method of claim 1, wherein the machine comprises a graphics processing unit that performs the autonomous control operations, wherein the low-power mode is characterized by the graphics processing unit not receiving power.

8. The method of claim 1, wherein at least one of running the diagnosis, the restarting, entering the low-power mode, exiting the low-power mode, or enabling the autonomous control operations is performed at least partially by a controller that generates control signals to implement the autonomous control operations.

9. The method of claim 8, wherein the controller comprises a circuit that remains powered during the low-power mode to wake up a system-on-a-chip from the low-power mode.

10. A processor comprising: one or more circuits to: detect a power-off indication of a vehicle; ​ ​ instructing a computer system for autonomous control of the vehicle to perform a diagnostic; triggering a reboot of the computer system to configure the computer system to a computing state and store the computing state as a saved state in computer storage; triggering a low-power mode of the computer system while the saved state is stored in the computer storage; and triggering an exit from the low-power mode based at least on an indication of a power-on of the vehicle, the triggering of the exit causing a restoration from the computer storage of the saved state based at least on the saved state to enable the autonomous control of the vehicle by the computer system.

11. The processor of claim 10, wherein the processor is included in at least one of: a control system for an autonomous or semi-autonomous machine; a perception system for an autonomous or semi-autonomous machine; a system for performing simulation operations; a system for performing deep learning operations; a system implemented using edge devices; a system implemented using robots; a system including one or more virtual machines (VMs); a system implemented at least in part in a data center; or a system implemented at least in part using cloud computing resources.

12. The processor of claim 10, wherein the one or more circuits are microcontroller units of the computer system.

13. The processor of claim 10, wherein the computing state is a first computing state, and the one or more circuits are further configured to: trigger an exit from the low-power mode prior to the power-on or start indication based at least on one or more criteria being met; instruct the computer system to re-run the diagnostic; trigger a reboot of the computer system to generate a second computing state and store the second computing state as the saved state in the computer storage; and trigger a re-entry into the low-power mode, wherein the saved state restored to enable the autonomous control is the second computing state.

14. The processor of claim 10, wherein the diagnostic includes a potential fault test of one or more components of the computer system.

15. The processor of claim 10, wherein the vehicle includes one or more sensors, wherein the low-power mode is characterized by at least one of the one or more sensors not receiving power.

16. A system comprising: one or more processing units of a vehicle; and one or more memory units storing instructions that, when executed by the one or more processing units, cause the one or more processing units to perform operations comprising: performing an in-system test of a computer system for autonomous control of the vehicle based at least on a first indication of the vehicle being turned off; based at least on completion of the in-system test, configuring the computer system to a computing state that enables the autonomous control; running in a low-power mode while the computing state is stored as a saved state in computer storage; exiting the low-power mode based at least on a second indication of the vehicle being turned on; and ​ ​ based on at least recovering the saved state from the computer storage, enabling the computer system to perform the autonomous control of the vehicle.

17. The system of claim 16, wherein the system is included in at least one of: a control system for an autonomous or semi-autonomous machine; a perception system for an autonomous or semi-autonomous machine; a system for performing simulation operations; a system for performing deep learning operations; a system implemented using edge devices; a system implemented using robots; a system including one or more virtual machines (VMs); a system implemented at least in part in a data center; or a system implemented at least in part using cloud computing resources.

18. The system of claim 16, wherein the operations further comprise: prior to performing the in-system test within the system based on at least the first indication that the vehicle is turned off, restarting one or more portions of the computer system.

19. The system of claim 16, wherein the computing state is a first computing state, and the operations further comprise: based on at least one or more criteria being met, exiting the low-power mode prior to the first indication; re-performing the in-system test of the computer system; and configuring the computer system to generate a second computing state, wherein the saved state recovered from the computer storage is the second computing state.

20. The system of claim 16, further comprising: one or more sensors, wherein the low-power mode is characterized by at least one of the one or more sensors not receiving power.

Images