State interruption to optimize the startup process of autonomous vehicles

Abstract

To provide a system that implements a state suspension for optimizing a start-up process of an autonomous vehicle.SOLUTION: Diagnostics and boot up for AV hardware and software of a computer system of an autonomous vehicle may be performed based at least on receiving a shutdown or power off indication, then a computing state of the computer system may be suspended with the computer system entering a low-power mode. The suspended computing state can be rapidly restored without requiring a reboot and diagnostics for key-on. To ensure integrity of the saved computing state, the computer system may exit the low-power mode, rerun the diagnostics, reload the programs, and then reenter the low-power mode. Restoring the suspended computing state may be triggered by a user inserting an ignition key, pressing a button to turn on the vehicle, opening a door to the vehicle, remotely unlocking the vehicle, remotely starting the vehicle, etc.SELECTED DRAWING: Figure 1

Description

[Background technology]

[0001] Designing a system for autonomously driving a vehicle without supervision at the level of safety required for actual acceptance is extremely difficult. Autonomous vehicles (AVs) should at least have the ability to function as functionally equivalent to a careful driver, using cognitive and behavioral systems with incredible capabilities to identify and react to moving and static obstacles in complex environments in order to avoid collisions with other objects or structures along their path. To meet these standards, autonomous vehicles are running increasingly complex programs. Along with this complexity, there is an increase in the boot time of the autonomous driving system, partly due to the increase in software data that must be loaded and configured. At the same time, there is an increasing requirement to reduce the startup time from when the vehicle is ignited (e.g., the time from when the vehicle is turned on so that it can be safely driven).

[0002] Before executing the autonomous driving software, the autonomous driving system may perform an initial diagnosis (e.g., latent defect inspection). This typically requires that the AV software be loaded from persistent memory into random access memory (RAM), and that AV hardware and AV software initialization follow system reboot, during which the diagnosis is performed. Conventionally, this process is triggered by turning the vehicle key on or opening the door. Due to the complexity of modern AV systems, this process may take too much time to complete (e.g., more than 20 seconds). One way to reduce startup time is to fragment the vehicle architecture, for example, using dedicated electronic control units (ECUs: Electronic Control Unit) for clusters (rear view cameras), another ECU for vehicle networking, and another ECU for ADAS, so that each of these ECUs can be dedicated to more quickly loading the corresponding functionality. However, the more fragmented the vehicle architecture, the more dedicated hardware and software are required, and the complexity of the system increases. In addition, as AV software becomes more complex, this approach may not be sufficient on its own to meet the startup time requirements.

Summary of the Invention

Problems to be Solved by the Invention

[0003] Embodiments of the present disclosure relate to an interrupted state for rapid startup of an autonomous vehicle. Systems and methods are disclosed that reduce the startup time required to bring a computer system (and, by extension, an autonomous vehicle) to full operating mode after turning the key on.

Means for Solving the Problems

[0004] In contrast to conventional systems such as those mentioned above, the diagnosis and boot-up of the AV hardware and software of an autonomous vehicle's computer system may be performed at least based on receiving a shutdown or power-off instruction, and the computing state of the computer system may then be interrupted as the computer system enters a low-power mode. The interrupted computing state can be quickly restored when the autonomous vehicle is turned on, without requiring a reboot and diagnosis for key-on.

[0005] To enter low-power mode, the computer system may perform various diagnostic functions, execute safety mechanisms, and then reload the program into a memory storage medium, such as random-access memory (RAM). While in low-power mode, certain components of the computer system may be fully powered, some may be partially powered, and some may be powered off. To ensure the integrity of the saved computational state, the computer system may exit low-power mode after a certain time interval. When the interval ends, the computer system may re-run the diagnostics, reload the program into RAM, and then re-enter low-power mode. When the driver returns to the vehicle, the computer system may exit low-power mode (for example, by receiving a key-on event signal). Since the diagnostics were performed before entering low-power mode, further diagnostics may not be necessary. Furthermore, since the program is already in RAM, the time required to prepare the program for execution can be significantly reduced. Therefore, computer systems can have faster boot times even when a more generalized system architecture is used (e.g., a single ECU), enabling simpler and more cost-effective systems.

[0006] The system and method for interrupting the computer system in autonomous vehicles are described in detail below with reference to the attached drawings. [Brief explanation of the drawing]

[0007] [Figure 1] This flowchart illustrates methods for entering and exiting an interruption state according to some embodiments of the present disclosure. [Figure 2] This is a system diagram showing the components of a computer system for an autonomous vehicle according to some embodiments of the present disclosure. [Figure 3A] This flowchart illustrates the processing of a request to enter a suspended state, as performed by a processing system and controller, according to some embodiments of the present disclosure. [Figure 3B] This flowchart illustrates how to enter an interruption state, as performed by a processing system and controller, according to some embodiments of the present disclosure. [Figure 4A] This flowchart illustrates how, according to some embodiments of the present disclosure, a processing system and controller exit an interrupted state to resume autonomous control. [Figure 4B] This flowchart illustrates how to exit a power-off state from an interrupted state, as performed by a processing system and controller according to some embodiments of the present disclosure. [Figure 5] This flowchart illustrates how to enter and exit a low-power mode to resume autonomous control, according to some embodiments of the present disclosure. [Figure 6] This flowchart illustrates a method for controlling a processing system to enter and exit a low-power mode, according to some embodiments of the present disclosure. [Figure 7] This flowchart illustrates methods for instructing the system to enter and exit low-power mode, according to some embodiments of the present disclosure. [Figure 8A] Illustrations of exemplary autonomous vehicles according to some embodiments of the present disclosure. [Figure 8B] Figure 8A shows examples of camera positions and fields of view of an exemplary autonomous vehicle according to some embodiments of the present disclosure. [Figure 8C]Figure 8A is a block diagram of an exemplary system architecture of an exemplary autonomous vehicle according to some embodiments of the present disclosure. [Figure 8D] This is a system diagram of communication between a cloud-based server and the exemplary autonomous vehicle shown in Figure 8A, according to some embodiments of the present disclosure. [Figure 9] This is a block diagram of an exemplary computing device suitable for use in implementing some embodiments of the present disclosure. [Figure 10] This is an exemplary data center block diagram suitable for use in implementing some embodiments of the present disclosure. [Modes for carrying out the invention]

[0008] Systems and methods relating to interrupted states for the rapid startup of autonomous vehicles are disclosed. This disclosure may be described in relation to exemplary autonomous vehicles 800 (or referred to herein as “Vehicle 800” or “Ego Vehicle 800,” examples thereof are described with respect to Figures 8A–8D), but this is not intended to be limiting. For example, the systems and methods described herein may be used by non-autonomous vehicles, semi-autonomous vehicles (for example, in one or more adaptive driver assistance systems (ADAS)), manned and unmanned robots or robotics platforms, warehouse vehicles, off-road vehicles, vehicles attached to one or more trailers, flying ships, boats, shuttles, emergency response vehicles, motorcycles, electric or motorized bicycles, aircraft, construction vehicles, submarines, drones, and / or other vehicle types. In addition, while this disclosure may describe booting and suspending computer systems of autonomous vehicles, this is not intended to be limiting, and the systems and methods described herein may be used in any other technological space where computer booting and suspending may be used, such as augmented reality, virtual reality, mixed reality, robotics, security and surveillance, autonomous or semi-autonomous machine applications, and / or computer booting and suspending.

[0009] In one or more embodiments, the computer system may enter a low-power mode upon receiving instructions (e.g., from the driver and / or system components) to turn off the vehicle (e.g., shutdown or power off). For example, the user may remove the ignition key, press a physical or virtual button to turn off the vehicle, or exit the vehicle. In one or more embodiments, the computer system may end a driving cycle in which the vehicle has been operating at least partially autonomously. This may include shutting down various sensors and displays. In one or more embodiments, the computer system will perform hardware-based diagnostics. In any powered component, there may be opportunities for inverted bits or other errors to accumulate during operation. For example, these errors may be at least partially caused by radiation (e.g., solar radiation, alpha particles, etc.). Diagnostics may check for the accumulation of errors in operation of various components. These diagnostics may include checking for potential defects.

[0010] In one or more embodiments, the computer system will be at least partially reinitialized or rebooted into an operating cycle prior to performing diagnostics. In one or more embodiments, the computer system will suspend its components after the diagnostics and / or reboot and save the current state to RAM. A potential defect check, which may be performed as at least part of the diagnostics, may destroy, impair, or otherwise affect the current state of the processor. System safety may be ensured by issuing a full reboot of the system after the completion of the potential defect checks and before the suspension to RAM. The computer system may then enter a low-power mode. A low-power mode may refer to any of the various modes that reduce the power consumption of the computer system, as discussed herein. Other components may be disabled or put into standby mode. In one or more embodiments, the system may continue to perform diagnostics while performing a low-power mode entry, disabling safety mechanisms only at points where the logic hosting the mechanism is disabled.

[0011] A computer system may remain in low-power mode until triggered to exit low-power mode. Examples of such triggers will be briefly discussed, but only as examples. Firstly, the computer system may detect or determine that the operator has returned to the vehicle and cause the computer system to exit low-power mode and enter normal operating mode. Normal operating mode may include autonomous controls and other functions performed at full power. Secondly, a certain time interval may elapse to allow the computer system to rerun diagnostics, reload programs, and re-enter low-power mode. Thirdly, below a certain battery level threshold (or other threshold), the computer system may exit low-power mode and completely shut down or power off (to prevent further battery drain). While batteries are described herein, the description of batteries may apply to one or more energy storage media (e.g., capacitors).

[0012] In low-power mode, the most power-consuming components may be powered off. Examples of components that may be powered off include the central processing unit (CPU) and graphics processing unit (GPU). The CPU and / or GPU may be used in part to power autonomous driving functions, such as computer vision using neural networks. Certain components may be fully powered while in low-power mode (examples include the vehicle battery, power pre-regulation circuits, power sequencers, voltage regulators for always-on segments of the processor, and voltage regulators for DRAM). The vehicle battery may power the computer system via a switch or directly from the vehicle battery. Certain components may be partially powered while in low-power mode (examples include power management integrated circuits, always-on segments of the processor, FlexRay, Ethernet®, etc.). Some of these may be powered directly from the battery and may be configured to provide wake signals to other components as needed to exit low-power mode. Furthermore, certain components may be completely powered off while in low-power mode (examples include graphics processing units, microcontroller units, vehicle harnesses, peripherals, sensors, and displays). Microcontroller units and / or other controllers may be used for safety purposes, to check other aspects of the system, and to perform power control.

[0013] As described herein, a computer system may remain in a low-power mode for a certain time interval. For example, the time interval may be 8 hours or 24 hours. The time interval may be a static number, a programmable static number (e.g., by the vehicle manufacturer or end customer), or a dynamic number. After the interval, the computer system can easily exit low-power mode, rerun diagnostics, reinitialize programs, reboot, and re-enter low-power mode. This cycle may continue until an event is detected that causes it to exit, for example, a key-on instruction is received or the battery falls below a certain level.

[0014] In one or more embodiments, entering low-power mode may be done by turning off the key (or some other instruction to terminate the operating cycle). However, entering low-power mode may be configured to occur only when various potential criteria or conditions are met. The user may also be presented with the option to completely shut off the power if necessary.

[0015] The computer system can exit low-power mode to enable normal autonomous driving. Exiting low-power mode can be triggered by an indication that returning to normal driving is imminent or necessary. For example, the user may insert an ignition key, press a button to turn on the vehicle, open a door to the vehicle, remotely unlock the vehicle, or remotely start the vehicle.

[0016] To exit the low power mode, a wake-up signal can be sent to the microcontroller unit (and / or other controller). The microcontroller unit can trigger the computer system to leave the low power mode based at least on the wake-up signal. After being triggered, the computer system can leave the low power mode. Leaving the low power mode can include, for example, the following tasks or operations (by way of example and without limitation): turning on the clock, sensors, and display. Leaving the low power mode can also include, for example, the following tasks or operations (by way of example and without limitation): authenticating, restoring the saved state, and otherwise preparing the computer system for normal operation. Since the computational state is saved in the RAM (and / or other storage medium), the computer system can transition to normal operation (for example, when an autonomous vehicle is ready for autonomous control) much more quickly than with conventional processes and solutions. For example, all application states and sensor states can already be in the RAM.

[0017] In some instances, a computer system can transition from a low-power mode to a power-off mode or state. A computer system can transition to power-off mode based on at least one or more criteria being met. For example, a computer system can transition to power-off mode when the battery level falls below a certain threshold. Another example is that a computer system can transition to power-off mode based on at least a timeout of total time in low-power mode. In these instances, a microcontroller unit can be awakened by the microcontroller power management circuit. The microcontroller unit can detect the timeout and then instruct various components in low-power mode to completely shut down or power off. The microcontroller can also put its power management circuit into standby mode and then power off itself. Some components, such as the power management circuit, may remain powered in the power-off state to allow for revival. A computer system may remain in power-off mode until a key-on or other instruction is received (e.g., the battery is fully charged, the vehicle is plugged in, etc.).

[0018] Referring to FIG. 1, FIG. 1 is a flowchart showing a method 100 for entering and exiting an interrupt state according to some embodiments of the present disclosure. Each block of method 100 (and other methods described herein) may include a computing process that can be executed using any combination of hardware, firmware, and / or software. For example, various functions may be implemented by a processor that executes instructions stored in memory. Method 100 may also be implemented as computer-usable instructions stored on a computer storage medium. Method 100 may be provided, for example, by a stand-alone application, a service or a hosted service (either stand-alone or in combination with another hosted service), or a plug-in to another product. In addition, method 100 is described, by way of example, with respect to the computer system 200 of FIG. 2. However, method 100 may be executed additionally or alternatively by any one system, or any combination of systems, including but not limited to those described herein.

[0019] Method 100 may be used to boot, interrupt, and load the state of the computer system 200 (e.g., an ECU) of FIG. 2 in order to enable autonomous control of an autonomous vehicle (e.g., vehicle 800). However, method 100 may also be used for other types of computer systems or to enable other types of functionality using a computer system. In an embodiment of the present disclosure, method 100 may be executed when the autonomous vehicle is key-off. Method 100 may be configured to execute under certain criteria or conditions in addition to or instead of key-off. Further, the user may have the option to completely power off the vehicle's computer system 200 as needed, instead of executing method 100 and putting the computer system 200 into a low power mode. For example, if the user knows that the autonomous vehicle will not be used for an extended period of time, the user may choose to completely power off the vehicle instead of enabling the autonomous vehicle to enter method 100, which may be executed by default.

[0020] Method 100 includes receiving or detecting a power-off instruction in block B102. The computer system may initiate entering a low-power mode based at least on a shutdown or power-off instruction, for example, an instruction from the driver (and / or system component) that the vehicle has stopped operating or is to stop operating (and / or that autonomous control is to be disabled). For example, the user may remove the ignition key, press a button to turn off the vehicle, or exit the vehicle. Based on the shutdown or power-off instruction, the computer system may terminate the driving cycle in which the vehicle was operating autonomously. This may include shutting down various sensors and displays of the vehicle.

[0021] Method 100 includes, in block B104, that the diagnosis is performed on a computer system, for example, an ECU. The computer system may perform any of a variety of hardware-related diagnostics. In one or more embodiments, the diagnosis may be performed using in-system testing (IST). During normal computer operation, bits within the computer system may be flipped or experience other errors or faults. These errors and faults may be caused by radiation (solar radiation, alpha particles, etc.) or other external sources, or may be induced in other ways while power is supplied. The diagnosis performed may include a potential defect check to determine and correct such errors. Although not shown in Figure 1, in one or more embodiments, the computer system may be at least partially rebooted prior to the diagnosis (for example, based on a shutdown or power-off instruction in block B102 and / or based on exiting low-power mode in block B124).

[0022] Method 100 includes a reboot of the computer system in block B106. The computer system may be at least partially rebooted back into a driving cycle after the diagnosis. The computer system may perform at least partially a reboot because the potential defect check has destroyed the current state of the processor. The reboot may produce a new computational state containing the currently loaded instances of one or more programs used for autonomous control and / or other functions of the autonomous vehicle.

[0023] Method 100 includes storing a computation state in block B108. For example, the computation state may be stored in one or more computer storage media, which may include at least one volatile memory, such as random-access memory (RAM). Storing the computation state before entering low-power mode may allow the computation state to be quickly recalled when leaving low-power mode without having to perform a full reboot.

[0024] Method 100 includes entering a low-power state in block B110. To enter a low-power state, the computer system may suspend one or more components described herein. Other components may be disabled or put into standby mode. Examples of such components are discussed further with reference to Figure 2. In some embodiments, the low-power mode is system control 7 (SC7), where system control may refer to a system power state. Other system controls may include system control 0, where the computer system is at full power, or system control 8, where the system is completely powered off (except for a wake circuit). Other levels and configurations of the low-power mode may also be used.

[0025] In one or more embodiments, after entering low-power mode in block B110, various other functions may be performed depending on that various criteria are met. For example, blocks B112 to B116, B118 to B120, B122 to B124, or B126 to B128 may occur. With respect to blocks B112 to B116, method 100 includes the occurrence of a power-on or start-up instruction in block B112. The power-on or start-up instruction may trigger exit from low-power mode and may include, but not limited to, an instruction that it is initial, imminent, or otherwise required to return to normal operating operation. The power-on or start-up instruction may be received from components outside the computer system (e.g., ECU) in the autonomous vehicle, from sensors coupled to communicate with the computer system, and / or from some other electronic device (e.g., a key fob). In one or more embodiments, a user may trigger a power-on or start command by interacting with the autonomous vehicle in some way. For example, the user may insert an ignition key, press a button to turn on the vehicle, open a door to the vehicle, remotely unlock the vehicle, remotely start the vehicle, or approach a registered vehicle with a wireless communication device.

[0026] Method 100 includes exiting low-power mode in block B114. For example, the controller may trigger the processing system and other components of the computer system to exit low-power mode. This may include the controller instructing the power manager to provide power to various components, as described herein. The controller and / or processing system may also turn on the clock, sensors, and displays. This may include authenticating, booting, and / or preparing the computer system for normal operation in other ways. Since the previous computing state has been saved to storage (for example, by block B108), the computer system can proceed to runtime without loading and initializing all AV software and hardware configurations. For example, all application states and sensor states may already be loaded in the computing state saved to storage.

[0027] Method 100 includes enabling autonomous control in block B116. For example, once the computing state is recovered from storage, the computer system may autonomously control the vehicle. The computer system may continue to autonomously control the vehicle until a shutdown or power-off instruction is received or another event occurs. For example, Method 100 may loop back to block B102 and continue another cycle.

[0028] With respect to blocks B118 to B120, method 100 includes, in block B118, the occurrence of a low power supply. For example, a low battery may be determined by the computer system, components outside the computer system, or some other device. In one or more embodiments, a lower power supply may be determined at least on the basis that the power level is below a threshold.

[0029] Method 100 includes completely cutting off power to the computer system in block B120. By completely cutting off power, power losses will be stopped or greatly reduced.

[0030] Low power can prevent an autonomous vehicle from starting or operating for a sufficiently long period of time, so continuing to operate in low-power mode when the power falls below a certain threshold can be disadvantageous. For example, in a vehicle supplied with internal combustion power, a low battery can prevent the starter from successfully starting the engine. Similarly, in a vehicle supplied with electric power, a low battery can reduce the range available to the electronic vehicle. Blocks B122 to B124 may be used to improve such situations. With respect to blocks B122 to B124, method 100 includes in block B122 that a certain time interval has elapsed (and / or other conditions have been met). The time interval may indicate that a diagnosis should be performed again to remain in low-power mode.

[0031] The computer system can remain in a low-power state for a certain time interval. For example, the time interval may be 8 hours or 24 hours. The time interval may be a predetermined number, a static number that is programmable (e.g., by the vehicle manufacturer or end customer), or a dynamic number. After the interval, the computer system can easily exit the low-power state, rerun diagnostics, reinitialize programs, and re-enter low-power mode (e.g., according to blocks B104 to B110). In one or more embodiments, the computer system may also reboot before block B104.

[0032] Therefore, method 100 includes exiting low-power mode in block B124, after which method 100 may return to block B104 (e.g., after a reboot) so that the diagnosis can be performed again. The computer system may then reboot, store the computation state, and then re-enter low-power mode (e.g., in block B110). The computer system may repeat this cycle, for example, as long as the power supply (e.g., battery) has sufficient power and no power-on or startup instruction has been received. If the power supply is charged externally, the computer system may repeat this cycle until a power-on or startup instruction is received (e.g., as discussed in block B112).

[0033] Method 100 may include, in block B126, the detection of one or more faults while the computer system is in low-power mode. For example, the controller may perform a diagnostic while the processing system and other components of the computer system are in low-power mode. This may include the controller instructing the processor, power manager, or other components to test for one or more faults periodically or on some trigger. The diagnostic may be the same as or different from the diagnostic performed in block B104.

[0034] Method 100 may include logging in block B128 after a fault has been detected in block B126. Logging a fault may create or update a record indicating what components and what type of fault was detected, which may include various sensor readings and similar information about the fault. In some embodiments of the present disclosure, following the detection and / or logging of a fault, the controller may instruct a complete power off. This complete power off may include one or more steps described in block B120 of this specification. In other embodiments of the present disclosure, following the detection and / or logging of a fault, the computer system may remain in a low-power mode for a period of time (e.g., discussed in block B122) and / or receive a power-on or start instruction (e.g., discussed in block B112). In these embodiments, the logged fault may be further evaluated and tested when the computer system returns to full-power mode (e.g., discussed in blocks B114 and / or B124). If a fault is verified after the system has returned to full power mode, the controller may instruct the computer system to completely cut off power (as discussed, for example, in block B120).

[0035] Referring to Figure 2, Figure 2 is an exemplary computer system 200 for an autonomous vehicle 800 or other machine. The computer system 200 may have one or more processing systems 202. One or more processing systems 202 may load and run autonomous control software for the autonomous vehicle and / or perform other functions. The computer system 200 may also have one or more controllers 204. The controllers may include a microcontroller unit (MCU) or other controllers. For safety reasons, the controllers 204 may be used to check, instruct, and monitor other components of the computer system 200. The controllers 204 may also control the supply of power to other components of the computer system 200. The computer system 200 may also have computer storage 206 (e.g., volatile memory storage). The computer storage 206 may include one or more computer storage media, e.g., RAM, and in one or more embodiments may include dynamic random access memory (DRAM). Computer system 200 may also have non-volatile memory (not shown) for storing various programs and configurations of processing system 202, or may have access to non-volatile memory. This memory may be used to load programs and configurations during reboots of computer system 200, as described herein.

[0036] The computer system 200 may further have a power supply 208, or may be otherwise associated with a power supply 208. Examples of power supplies 208 include vehicle batteries, e.g., batteries that provide power to electrically powered vehicles, batteries that provide electrical functionality to internally powered vehicles, dedicated batteries for electronic control units, and / or other batteries. The computer system 200 may further have an interface manager 210. The interface manager may manage one or more communication interfaces, e.g., a controller area network (CAN), an Ethernet® network, a FlexRay network, and / or those of other network or interface types. In one or more embodiments, the interface manager 210 may include a CAN controller, an Ethernet® physical layer (e.g., a chip or software that sends and receives Ethernet® frames), a FlexRay communication bus, and / or other components. The computer system 200 may also have a power manager 212. For example, a power manager may include a power manager integrated circuit (PMIC). The power manager 212 may interface with the vehicle harness 214 and the controller 204.

[0037] The computer system 200 may further have one or more pre-regulators 216. The pre-regulators 216 may be configured to reduce ripple present in the output power from the power supply 208. The pre-regulators 216 may, in addition or otherwise, reduce or minimize power losses in the voltage regulator 220. The pre-regulators 216 may interface with peripheral power 218. Peripheral power 218 may directly or indirectly supply power to any of the various peripheral components (e.g., sensors described herein).

[0038] The voltage regulator 220 may supply power to various other components of the computer system 200, including a wake module 222 on the processing system 202. The wake module 222 may include a constantly-on segment of the processing system 202 and / or a segment that can be on while the processing system 202 is in low-power mode. The voltage regulator 220 may supply power to the wake module 222 via a wake module power supply 224, which may include a voltage regulator for the constantly-on segment of the processing system 202. The voltage regulator 220 may also supply power to the computer storage 206 via a storage power supply 226. The voltage regulator may also supply power to the processing system 202 via the processing system 202's main power supply 228 (for example, while the processing system 202 is not in low-power mode and / or under normal operation).

[0039] As described herein, in low-power mode, certain components may receive full power, partial power, or no power at all. In various embodiments, components may choose full power, partial power, or no power based on their functionality and the need to be in and out of low-power mode. To enter low-power mode, the component that consumes the most power in the computer system 200 may be powered off. This may include a central processing unit (CPU) and / or GPU, which may be a component of the processing system 202 or otherwise related to the processing system 202. For example, the processing system 202 may be one or more of the SoC 804 in Figure 8C, or may include this. Furthermore, the processor 810, CPU 806, accelerator 814, and / or GPU 808 may be powered off for low-power mode. In embodiments where the processing system 202 includes the logic unit 920 in Figure 9, they may also be powered off for low-power mode.

[0040] While in a low-power mode or state, certain components may be fully powered, as shown in Figure 2 (examples include power supply 208, pre-regulator 216, wake module 222, wake module power supply 224, storage power supply 226, etc.). The vehicle battery, or another power supply 208, provides power to the electronic control unit, which may be from a switch or directly from the vehicle. While in a low-power state, certain components may be partially powered (examples include power manager 212, interface manager 210, computer storage 206, processing system 202, etc.). Some of the partially powered components may be powered by power supply 208 and configured to provide wake signals to other components. For example, power manager 212 may be configured to provide one or more wake signals to wake module 222 to wake processing system 202 from low-power mode. While in low-power mode, certain components may be completely powered off (examples include the processing system 202, controller 204, vehicle harness 214, and the most power-consuming components of peripheral devices, sensors, and / or displays that may be powered by peripheral power 218 as described herein). Various safety mechanisms may be partially or completely powered during low-power mode. Examples of these safety mechanisms may relate to monitoring and controlling power, clock, temperature, and other variables to ensure safe operation of the computer system 200 when leaving low-power mode, which may further improve startup time by reducing the amount of inspection required for awakening.

[0041] Referring here to Figures 3A and 3B, the general flow between the processing system 202 and the controller 204 is shown. As an example, the processing system 202 and the controller 204 may interact according to their general flow to implement at least some of blocks B102 through B110 of Method 100. Figure 3A depicts an exemplary flowchart 300A that includes receiving and processing a shutdown or power-off request for the computer system 200. Figure 3B depicts an exemplary flowchart 300B that includes entering a low-power mode in response to the shutdown or power-off request received and processed in flowchart 300A.

[0042] The functions that can be performed by the processing system 202 are shown on the left side of Figures 3A and 3B, and the steps that can be performed by the controller 204 are shown on the right side of Figures 3A and 3B. Figures 3A and 3B generally show the passage of time moving downward with respect to the exemplary process. It should be understood that the length of the illustrated passage of time is merely an example, and that the passage of time may vary in the amount of time in or between each block without departing the scope of this disclosure. Similarly, arrows between the two sides may indicate one or more messages or status updates sent between the processing system 202 and the controller 204, or may indicate the next step taken without any information or messages being sent between the processing system 202 and the controller 204. Specifically, the arrows and time intervals are simply intended to illustrate the relevant concepts to the reader.

[0043] In flowchart 300A, block B302 includes controller 204 requesting a shutdown or power-off of processing system 202, for example, in response to a shutdown or power-off instruction from an external vehicle component of computer system 200. Controller 204 may send this shutdown or power-off request to processing system 202. Block B304 includes controller 204 initiating a power-off sequence for processing system 202. Block B306 includes controller 204 triggering a reboot of processing system 202. Block B308 includes processing system 202 preparing for a reboot, which is triggered by controller 204. Processing system 202 may report to controller 204 that it is ready to reboot back, or controller 204 may wait for a certain time interval. Block B310 includes controller 204 asserting a reset of processing system 202. The controller 204 may also initiate a diagnosis that is to be performed by the processing system 202. Block B312 includes the processing system 202 rebooting and then performing the diagnosis as instructed by the controller 204. When the diagnosis is complete, the processing system 202 may report completion to the controller 204. The report may include an instruction that no faults were found, an instruction that any faults were found, or other information. Block B314 includes the controller 204 asserting a reset of the processing system 202 upon instruction that the diagnosis is complete. Block B316 includes the processing system 202 rebooting into a functional operating mode. The functional operating mode may include loading all or part of the programs, applications, functions, and other computer instructions related to the autonomous control of the autonomous vehicle. Block B318 includes the controller 204 detecting (or otherwise receiving instruction that) that the processing system 202 has entered a functional mode.

[0044] In flowchart 300B, block B350 includes the processing system 202 requesting to enter a low-power mode after the completion of a particular timer. The timer may allow the cancellation of the low-power mode. Block B352 includes the controller 204 triggering the low-power mode and responding to the processing system 202. Block B354 includes the processing system 202 suspending a unit and saving the computation state in computer storage 206. The suspended unit may include any of the various components of the computer system 200, and additionally or otherwise may include various other components outside the computer system 200, such as peripherals, sensors, displays, input devices, output devices, and / or other electronic devices. The computation state, generated using block B316 of flowchart 300A and detected using block B318 of flowchart 300A, may be saved in computer storage 206. This may allow the computation state to be retrieved from computer storage 206 when booted. Block B356 includes the controller reporting status to one or more components of the autonomous vehicle that are outside the computer system 200. Block B356 may be executed by the controller 204 while the processing system 202 is suspending a unit and / or saving the computation state. Block B358 includes the processing system 202 initiating a power-off sequence. Block B360 includes the processing system 202 putting at least a portion of the computer system 200 into a low-power mode.

[0045] Referring here to Figure 4A, Figure 4A is flowchart 400A showing the process of exiting a suspended state and resuming autonomous control, as performed by the processing system 202 and controller 204 in some embodiments of the present disclosure. In example, the processing system 202 and controller 204 may interact according to flowchart 400A to implement at least some of blocks B112 to B114 of Method 100.

[0046] Block B402 includes the controller 204 receiving a vehicle power-on or start trigger or instruction. The trigger may be received from the vehicle harness 214 via the power manager 212. Block B404 also includes the supply of power to the controller 204, for example, by the power manager 212. For example, power may be supplied from the power supply 208 via the power manager 212. Block B406 includes the controller 204 booting up. The controller 204 may initiate a process of initiating one of the various powers to process the power-on or start trigger and instructing other components of the computer system 200 to exit low-power mode. Block B408 includes the controller 204 responding to the power-on or start trigger. For example, the controller 204 may send a message to an external component of the computer system 200 indicating that the power-on or start trigger has been received and is being processed.

[0047] Block B410 includes the controller 204 instructing the processing system 202 to exit low-power mode and resume normal operation. The controller 204 may also instruct the pre-regulator 216 and / or voltage regulator 220 to supply power (e.g., full power) to the corresponding components of the computer system 200, as described herein. The pre-regulator 216 and / or voltage regulator 220 may also supply power to the processing system 202, peripheral devices, computer storage 206, and other components.

[0048] Block B412 includes preparing the processing system 202 to exit low-power mode. This may include restoring the computation state from computer storage 206. Block B414 includes the processing system 202 instructing the controller 204 that the processing system is ready for autonomous control. Block B416 includes the controller 204 sending a report to an external component of computer system 200 that the processing system 202 is ready. B418 includes the controller 204 instructing the processing system 202 to autonomously control the autonomous vehicle. The processing system may then autonomously control until the next shutdown or power-off instruction is received, or until any other event occurs or is detected.

[0049] Referring here to Figure 4B, Figure 4B is flowchart 400B showing the exit from a suspended state to a power-off state, as performed by the controller 204 in some embodiments of the present disclosure. As an example, the controller 204 may operate according to flowchart 400B to implement at least some of blocks B122 to B124 of Method 100.

[0050] Block B452 includes the power manager 212 triggering a timeout from low-power mode due to low battery, the total time in a low-power mode cycle, or other triggers. Block B454 includes the controller 204 being powered by the power manager 212 in response to the power manager 212 waking up. Block B456 includes the controller 204 confirming the timeout via a trigger. Block B458 includes the controller 204 instructing other components in the computer system 200 (e.g., those powered by the low-power mode rails) to be powered off. Block B460 includes the controller 204 putting the power manager 212 into standby mode and powering it off itself. During the power-off, all rails of the computer system 200 may be off, except that the power manager 212 may be in standby or sleep mode. Additionally, the interface manager 210 may be in low-power standby mode.

[0051] Figure 5 is a flowchart illustrating methods 500 for entering and exiting a low-power mode to resume autonomous control, according to some embodiments of the present disclosure. Method 500 is described, by example, with respect to the computer system 200 of Figure 2. However, method 500 may be performed additionally or alternatively by any one system or any combination of systems, including but not limited to those described herein.

[0052] Method 500 includes, in block B502, performing a diagnosis in a computer system. For example, the processing system 202 may perform a diagnosis in a computer system 200 used for the autonomous control of a machine (e.g., a vehicle 800) based at least on a machine shutdown or power-off instruction.

[0053] Method 500 includes rebooting the computer system in block B504. For example, the processing system 202 may reboot one or more parts of the computer system 200 based at least partially on diagnostics to configure the computer system 200 into a computing state.

[0054] Method 500 includes storing the calculation state in block B506. For example, the processing system 202 may store the calculation state in computer storage 206 as a saved state.

[0055] Method 500 includes entering a low-power mode in block B508. For example, the processing system 202 may enter a low-power mode while the saved state is stored in computer storage 206.

[0056] Method 500 includes exiting low-power mode in block B510. For example, the processing system 202 may exit low-power mode at least based on a machine power-on or startup instruction.

[0057] Method 500 includes enabling autonomous control in block B512. For example, the processing system may enable autonomous control of the machine by the computer system 200, at least on the basis of restoring a state saved from computer storage 206.

[0058] Referring now to Figure 6, Figure 6 is a flowchart illustrating a method for controlling a processing system to enter and exit a low-power mode, according to some embodiments of the present disclosure. Method 600 is described, by example, with respect to the computer system 200 of Figure 2. However, Method 600 may be carried out additionally or alternatively by any one system or any combination of systems, including but not limited to those described herein.

[0059] Method 600 includes initiating a power-off sequence in block B602. For example, the controller 204 may initiate a power-off sequence in response to detecting a shutdown or power-off instruction for a vehicle (e.g., vehicle 800).

[0060] Method 600 includes instructing block B604 to perform a diagnosis. For example, controller 204 may instruct computer system 200, used for autonomous control of the vehicle, to perform a diagnosis.

[0061] Method 600 includes instructing a reboot to be performed in block B606. For example, the controller 204 may instruct one or more parts of the computer system 200 (e.g., processing system 202) to reboot and configure the computer system 200 into a computing state and store the computing state as a saved state in computer storage 206.

[0062] Method 600 includes triggering a low-power mode in block B608. For example, the controller 204 may trigger a low-power mode for one or more components in the computer system 200 while the state to be saved is stored in the computer storage 206.

[0063] Method 600 includes triggering an exit from low-power mode in block B610. For example, controller 204 may trigger an exit from low-power mode at least based on a vehicle power-on or start command, and the trigger for exiting enables autonomous control of the vehicle by computer system 200 at least based on a saved state being restored from computer storage 206.

[0064] Referring now to Figure 7, Figure 7 is a flowchart illustrating a method 700 for instructing to enter and exit low-power mode, according to some embodiments of the present disclosure. The method 700 is described, by example, with respect to the computer system 200 (discussed herein) shown in Figure 2. However, the method 700 may be performed additionally or alternatively by any one system or any combination of systems, including but not limited to those described herein.

[0065] Method 700 includes performing an in-system inspection in block B702. For example, processing system 202 may perform an in-system inspection of computer system 200 used for autonomous control of the vehicle, at least based on a first instruction to turn off the vehicle's ignition.

[0066] Method 700 includes configuring a computational state in block B704. For example, the processing system 202 may configure the computer system 200 into a computational state that has the capability to perform autonomous control, at least based on the completion of an in-system check.

[0067] Method 700 includes operating in a low-power mode in block B706. For example, the processing system 202 may operate in a low-power mode while the computation state is stored in computer storage 206 as a saved state.

[0068] Method 700 includes exiting low-power mode in block B708. For example, the processing system 202 may exit low-power mode at least based on a second instruction to turn the vehicle key on.

[0069] Method 700 includes enabling autonomous control in block B710. For example, the processing system 202 may enable autonomous control of the machine by the computer system 200, at least on the basis of restoring a state saved from computer storage 206.

[0070] Exemplary autonomous vehicle Figure 8A shows an exemplary autonomous vehicle 800 according to some embodiments of the present disclosure. The autonomous vehicle 800 (or referred to herein as "vehicle 800") may include, but is not limited to, passenger vehicles such as cars, trucks, buses, first responder vehicles, shuttles, electric or motorized bicycles, motorcycles, fire engines, police vehicles, ambulances, boats, construction vehicles, submarines, drones, trailer-mounted vehicles, and / or other types of vehicles (e.g., unmanned and / or carrying one or more passengers). Autonomous vehicles are generally described in terms of automation levels as defined by the National Highway Traffic Safety Administration (NHTSA), departments of the U.S. Department of Transportation, and the Society of Automotive Engineers (SAE) "Taxonomy and Definitions for Terms Related to Driving Automation Systems for On-Road Motor Vehicle" (standard number J3016-201806, published June 15, 2018; standard number J3016-201609, published September 30, 2016; and previous and future versions of this standard). The mobile vehicle 800 may have the capability to perform functions at one or more of the autonomous driving levels from Level 3 to Level 5. For example, depending on the embodiment, the vehicle 800 may have the capability of driver assistance (Level 1), partial automation (Level 2), conditional automation (Level 3), high automation (Level 4), and / or full automation (Level 5). In this specification, the term “autonomous” may include any and / or all types of autonomy of the vehicle 800 or any other machine, for example, being fully autonomous, highly autonomous, conditionally autonomous, partially autonomous, providing auxiliary autonomy, semi-autonomous, primarily autonomous, or other designations.

[0071] The vehicle 800 may include components such as the vehicle's chassis, body, wheels (e.g., 2, 4, 6, 8, 18, etc.), tires, axles, and other components. The vehicle 800 may include a propulsion system 850, such as an internal combustion engine, a hybrid power unit, a fully electric engine, and / or another propulsion system type. The propulsion system 850 may be connected to the vehicle 800's drivetrain, which may include a transmission, to enable propulsion for the vehicle 800. The propulsion system 850 may be controlled in response to receiving signals from a throttle / accelerator 852.

[0072] A steering system 854, which may include a steering wheel, may be used to steer the vehicle 800 (for example, along a desired course or route) when the propulsion system 850 is operating (for example, when the vehicle is moving). The steering system 854 may receive signals from the steering actuator 856. The steering wheel may also be an option for fully automated (level 5) functionality.

[0073] The brake sensor system 846 may be used to operate the vehicle brakes in response to receiving signals from the brake actuator 848 and / or the brake sensor.

[0074] The controller 836, which may include one or more system-on-a-chip (SoC) 804 (Figure 8C) and / or GPUs, can provide signals (e.g., expressions of commands) to one or more components and / or systems of the vehicle 800. For example, the controller can send signals to actuate the vehicle brakes via one or more brake actuators 848, actuate the steering system 854 via one or more steering actuators 856, and actuate the propulsion system 850 via one or more throttle / accelerators 852. The controller 836 may include one or more onboard (e.g., integrated) computing devices (e.g., supercomputers) that process sensor signals and output operational commands (e.g., signals representing commands) to enable autonomous driving and / or assist the driver in driving the vehicle 800. The controller 836 may include a first controller 836 for autonomous driving functions, a second controller 836 for functional safety functions, a third controller 836 for artificial intelligence functions (e.g., computer vision), a fourth controller 836 for infotainment functions, a fifth controller 836 for redundancy in emergency situations, and / or other controllers. In some examples, a single controller 836 may handle two or more of the aforementioned functions, and two or more controllers 836 may handle a single function, and / or any combination thereof.

[0075] The controller 836 can provide signals for controlling one or more components and / or systems of the mobile vehicle 800 in response to sensor data (e.g., sensor inputs) received from one or more sensors. Sensor data may be received from, for example and without limitation, global navigation satellite system sensors 858 (e.g., global positioning system sensors), RADAR sensors 860, ultrasonic sensors 862, LIDAR sensors 864, inertial measurement unit (IMU) sensors 866 (e.g., accelerometers, gyroscopes, magnetic compasses, magnetometers, etc.), microphones 896, stereo cameras 868, wide-view cameras 870 (e.g., fisheye cameras), infrared cameras 872, surround cameras 874 (e.g., 360-degree cameras), long-range and / or medium-range cameras 898, speed sensors 844 (e.g., for measuring the speed of a moving vehicle 800), vibration sensors 842, steering sensors 840, brake sensors (e.g., as part of a brake sensor system 846), and / or other sensor types.

[0076] One or more of the controllers 836 may receive inputs (represented, for example, by input data) from the instrument cluster 848 of the mobile vehicle 800 and provide outputs (represented, for example, by output data, display data, etc.) via the human-machine interface (HMI) display 834, an audible annunciator, a loudspeaker, and / or other components of the mobile vehicle 800. The outputs may include information such as mobile vehicle velocity, speed, time, map data (e.g., HD map 822 in Figure 8C), location data (e.g., the location of the mobile vehicle 800, such as on the map), direction, the locations of other mobile vehicles (e.g., occupied grids), and information about objects and the status of objects as grasped by the controller 836. For example, the HMI display 834 may display information regarding the presence of one or more objects (e.g., road signs, warning signs, changes in traffic signals, etc.) and / or driving operations that the moving vehicle has performed, is performing, or will perform (e.g., that it is now changing lanes, or that it is about to exit at Exit 34B within 3.22 km (2 miles)).

[0077] The mobile vehicle 800 further includes a network interface 824 that can communicate over one or more networks using one or more wireless antennas 826 and / or a modem. For example, the network interface 824 may have the capability to communicate over LTE, WCDMA®, UMTS, GSM, CDMA2000, etc. The wireless antenna 826 can also enable communication between objects in the environment (e.g., mobile vehicles, mobile devices, etc.) using local area networks such as Bluetooth®, Bluetooth® LE, Z-Wave, ZigBee, and / or low-power wide-area networks (LPWANs) such as LoRaWAN, SigFox.

[0078] Figure 8B shows examples of camera positions and fields of view of the exemplary autonomous vehicle 800 of Figure 8A according to several embodiments of the present disclosure. The cameras and their respective fields of view are exemplary embodiments and are not intended to limit the scope. For example, additional and / or alternative cameras may be included, and / or cameras may be placed in different positions on the mobile vehicle 800.

[0079] The camera type may include, but is not limited to, a digital camera that can be used with components and / or systems of the mobile vehicle 800. The camera may operate at Automotive Safety Integrity Level (ASIL) B and / or other ASILs. Depending on the embodiment, the camera type may have the capability of any image capture rate, such as 60 frames per second (fps), 120 fps, 240 fps, etc. The camera may have the capability to use a roll shutter, a global shutter, another type of shutter, or a combination thereof. In some examples, the color filter array may include an RCCC (red clear clear clear) color filter array, an RCCB (red clear clear blue) color filter array, an RBGC (red blue green clear) color filter array, a Foveon X3 color filter array, a Bayer sensor (RGGB) color filter array, a monochrome sensor color filter array, and / or another type of color filter array. In some embodiments, clear pixel cameras, such as cameras having RCCC, RCCB, and / or RBGC color filter arrays, may be used in efforts to increase light sensitivity.

[0080] In some applications, one or more cameras may be used to perform advanced driver assistance system (ADAS) functions (e.g., as part of a redundant or fail-safe design). For example, a multi-function mono-camera may be installed to provide functions including lane departure warning, traffic sign assist, and intelligent headlamp control. One or more cameras (e.g., all cameras) may simultaneously record and provide image data (e.g., video).

[0081] One or more of the cameras may be mounted in custom-designed (3D-printed) mounting parts to eliminate stray light and reflections from inside the vehicle (e.g., reflections from the dashboard reflected in the windshield mirror) that may interfere with the camera's image data capture capability. Referring to side mirror mounting parts, the side mirror parts may be custom 3D-printed so that the camera mounting plate conforms to the shape of the side mirror. In some examples, the camera may be integrated within the side mirror. For side-view cameras, the camera may also be integrated within four struts located at each corner of the cabin.

[0082] A camera having a field of view that includes a portion of the environment in front of the moving vehicle 800 (e.g., a forward-facing camera) may be used for surround view to help identify the forward path and obstacles and, with the help of one or more controllers 836 and / or control SoCs, to help provide information essential for generating an occupied grid and / or determining a preferred moving vehicle path. The forward-facing camera may also be used to perform many of the same ADAS functions as LIDAR, including emergency braking, pedestrian detection, and collision avoidance. The forward-facing camera may also be used for ADAS functions and systems, including other functions such as lane departure warning (LDW), autonomous cruise control (ACC), and / or traffic sign recognition.

[0083] Various cameras may be used in forward-facing configurations, including, for example, a monocular camera platform that includes a CMOS (complementary metal oxide semiconductor) color imaging device. Another example may be a wide-view camera 870, which can be used to capture objects entering the view from the periphery (e.g., pedestrians, crossing traffic, or bicycles). Although only one wide-view camera is shown in Figure 8B, any number of wide-view cameras 870 may be present in the mobile vehicle 800. In addition, long-range cameras 898 (e.g., a long-view stereo camera pair) may be used for depth-based object detection, particularly for objects for which the neural network has not yet been trained. Long-range cameras 898 may also be used for object detection and classification, as well as basic object tracking.

[0084] One or more stereo cameras 868 may also be included in the forward-facing configuration. The stereo camera 868 may include an integrated control unit with an expandable processing unit that may provide a programmable logic (FPGA) and a multi-core microprocessor with an integrated CAN or Ethernet® interface on a single chip. Such a unit may be used to generate a 3D map of the moving vehicle's environment, including distance estimates of all points in the image. An alternative stereo camera 868 may include a compact stereo vision sensor that includes two camera lenses (one on the left and one on the right) and an image processing chip that can measure the distance from the moving vehicle to an object and activate autonomous emergency braking and lane departure warning functions using the generated information (e.g., metadata). Other types of stereo cameras 868 may be used in addition to or instead of those described herein.

[0085] Cameras having a field of view including a portion of the environment to the sides of the mobile vehicle 800 (e.g., side-view cameras) may be used for surround view, providing information used to create and update the occupancy grid and generate side impact collision warnings. For example, surround cameras 874 (e.g., four surround cameras 874 as shown in Figure 8B) may be positioned on the mobile vehicle 800. The surround cameras 874 may include wide-view cameras 870, fisheye cameras, 360-degree cameras, and / or similar. For example, four fisheye cameras may be positioned in front of, behind, and to the sides of the mobile vehicle. In an alternative configuration, the mobile vehicle may use three surround cameras 874 (e.g., left, right, and rear) and utilize one or more other cameras (e.g., forward-facing cameras) as a fourth surround view camera.

[0086] A camera having a field of view that includes a portion of the environment behind the mobile vehicle 800 (e.g., a rear-view camera) may be used for parking assistance, surround view, rear collision warning, and creation and updating of the occupancy grid. A wide variety of cameras may be used, including, but not limited to, cameras suitable as forward-facing cameras (e.g., long-range and / or medium-range cameras 898, stereo cameras 868), infrared cameras 872, etc., as described herein.

[0087] Figure 8C is a block diagram of an exemplary system architecture of the exemplary autonomous vehicle 800 of Figure 8A, according to some embodiments of the present disclosure. It should be understood that this and other arrangements described herein are merely illustrative. Other arrangements and elements (e.g., machines, interfaces, functions, sequences, groupings of functions, etc.) may be used in addition to or instead of those shown, and some elements may be omitted together. Furthermore, many of the elements described herein are functional entities that can be implemented as individual or distributed components or in combination with other components, and in any appropriate combination and location. Various functions described herein as being performed by entities may be performed by hardware, firmware, and / or software. For example, various functions may be performed by a processor that executes instructions stored in memory.

[0088] Each component, feature, and system of the mobile vehicle 800 in Figure 8C is illustrated as being connected via a bus 802. Bus 802 may include a Controller Area Network (CAN) data interface (or referred to as the "CAN bus"). CAN may also be a network within the mobile vehicle 800 used to help control various features and functions of the mobile vehicle 800, such as the operation of brakes, acceleration, steering, windshield wipers, etc. The CAN bus may be configured to have dozens or hundreds of nodes, each having its own unique identifier (e.g., CAN ID). The CAN bus may be read to find steering angle, ground speed, engine revolutions per minute (RPM), button position, and / or other mobile vehicle status indicators. The CAN bus may be ASIL B compliant.

[0089] Bus 802 is described herein as a CAN bus, but this is not intended to limit it. For example, FlexRay and / or Ethernet® may be used in addition to, or as an alternative to, a CAN bus. In addition, a single line is used to represent bus 802, but this is not intended to limit it. There may be any number of buses 802, which may include, for example, one or more CAN buses, one or more FlexRay buses, one or more Ethernet® buses, and / or one or more other types of buses using different protocols. In some examples, two or more buses 802 may be used to perform different functions and / or for redundancy. For example, a first bus 802 may be used for collision avoidance and a second bus 802 may be used for operation control. In any example, each bus 802 may communicate with any of the components of the mobile vehicle 800, and two or more buses 802 may communicate with the same component. In some examples, each SoC804, each controller836, and / or each computer within the mobile vehicle may have access to the same input data (e.g., input from sensors in the mobile vehicle 800) and may be connected to a common bus such as a CAN bus.

[0090] The mobile vehicle 800 may include one or more controllers 836, such as those described herein with respect to Figure 8A. The controllers 836 may be used for a variety of functions. The controllers 836 may be connected to any of the various other components and systems of the mobile vehicle 800 and may be used for the control of the mobile vehicle 800, the artificial intelligence of the mobile vehicle 800, infotainment for the mobile vehicle 800, and / or the like.

[0091] The mobile vehicle 800 may include a system-on-a-chip (SoC) 804. The SoC 804 may include a CPU 806, a GPU 808, a processor 810, a cache 812, an accelerator 814, a data store 816, and / or other components and features not shown. The SoC 804 may be used to control the mobile vehicle 800 in various platforms and systems. For example, the SoC 804 may be coupled in a system (e.g., the system of the mobile vehicle 800) that has an HD map 822 that can obtain map refreshes and / or updates from one or more servers (e.g., server 878 in Figure 8D) via a network interface 824.

[0092] The CPU806 may include a CPU cluster or CPU complex (also referred to as "CCPLEX"). The CPU806 may include multiple cores and / or L2 caches. For example, in some embodiments, the CPU806 may include eight cores in a coherent multiprocessor configuration. In some embodiments, the CPU806 may include four dual-core clusters, each cluster having its own dedicated L2 cache (e.g., 2MBL2 cache). The CPU806 (e.g., CCPLEX) may be configured to support concurrent cluster operation, allowing any combination of the CPU806 clusters to be active at any given time.

[0093] The CPU806 can implement power management capabilities that include one or more of the following features: individual hardware blocks may be automatically clock-gated when idle to conserve dynamic power; each core clock may be gated when a core is not actively executing instructions by executing WFI / WFE instructions; each core may be independently power-gated; each core cluster may be independently clock-gated when all cores are clock-gated or power-gated; and / or each core cluster may be independently power-gated when all cores are power-gated. The CPU806 can further implement enhanced algorithms for managing power states, where acceptable power states and expected wake-up times are specified, and the hardware / microcode determines the best power state to input to the cores, clusters, and CCPLEX. Processing cores may support a simplified power state input sequence in software where the work is offloaded to microcode.

[0094] The GPU808 may include an integrated GPU (or, as referred to herein, "iGPU"). The GPU808 may be programmable and efficient for parallel workloads. In some embodiments, the GPU808 may be able to use an enhanced tensor instruction set. The GPU808 may include one or more streaming microprocessors, each of which may include an L1 cache (e.g., an L1 cache with a storage capacity of at least 96KB), and two or more of the streaming microprocessors may share a cache (e.g., an L2 cache with a storage capacity of 512KB). In some embodiments, the GPU808 may include at least eight streaming microprocessors. The GPU808 may be able to use a Computational Application Programming Interface (API). In addition, the GPU808 may be able to use one or more parallel computing platforms and / or programming models (e.g., NVIDIA's CUDA).

[0095] The GPU808 can be power-optimized for optimal performance in automotive and embedded use cases. For example, the GPU808 can be manufactured on a FinFET (Fin field-effect transistor). However, this is not intended to be a limitation, and the GPU808 can be manufactured using other semiconductor manufacturing processes. Each streaming microprocessor can incorporate several mixed-precision processing cores divided into multiple blocks. Not limited to, for example, 64 PF32 cores and 32 PF64 cores may be divided into four processing blocks. In such an example, each processing block may be allocated 16 FP32 cores, 8 FP64 cores, 16 INT32 cores, two mixed-precision NVIDIA tensor cores for deep learning matrix operations, an L0 instruction cache, a warp scheduler, a dispatch unit, and / or a 64KB register file. In addition, the streaming microprocessor may include independent parallel integer and floating-point data paths to provide efficient execution of workloads with a mixture of computation and addressing operations. A streaming microprocessor may include independent thread scheduling capabilities to enable finer-grained synchronization and coordination between concurrent threads. A streaming microprocessor may also include a combined L1 data cache and shared memory unit to simplify programming while improving performance.

[0096] In some examples, the GPU808 may include high-bandwidth memory (HBM) and / or a 16GB HBM2 memory subsystem to provide a peak memory bandwidth of 900 GB / s. In some examples, in addition to or instead of HBM memory, synchronous graphics random-access memory (SGRAM), such as graphics double data rate type five synchronous random-access memory (GDDR5), may be used.

[0097] The GPU808 can incorporate unified memory technology, including access counters, to enable more precise movement of memory pages to the processor that most frequently accesses them, thereby improving the efficiency of shared memory ranges across processors. In some examples, address translation service (ATS) support may be used to allow the GPU808 to directly access the CPU806 page table. In such examples, when the GPU808 memory management unit (MMU) experiences a miss, an address translation request may be sent to the CPU806. In response, the CPU806 can look up its page table for virtual-to-real-address mapping and send the translation back to the GPU808. As such, unified memory technology can enable a single, unified virtual address space for both the CPU806 and GPU808 memory, thereby simplifying GPU808 programming and porting of applications to the GPU808.

[0098] In addition, the GPU808 may include access counters that can record how often the GPU808 accesses the memory of other processors. Access counters can help ensure that memory pages are moved to the physical memory of the processor that accesses that page most frequently.

[0099] The SoC804 may include any number of caches 812, including those described herein. For example, cache 812 may include an L3 cache available to both the CPU806 and the GPU808 (e.g., connected to both the CPU806 and the GPU808). Cache 812 may include a write-back cache that can record line states, for example, by using a cache coherence protocol (e.g., MEI, MESI, MSI, etc.). The L3 cache may include 4MB or more, depending on the embodiment, although a smaller cache size may be used.

[0100] The SoC804 may include an arithmetic logic unit (ALU) that can be used to perform processing for any of the various tasks or operations of the vehicle 800 (for example, a processing DNN). In addition, the SoC804 may include a floating-point unit (FPU) (or other mass coprocessor or numerical coprocessor type) for performing mathematical operations within the system. For example, the SoC104 may include one or more FPUs integrated as execution units within the CPU806 and / or GPU808.

[0101] The SoC804 may include one or more accelerators 814 (e.g., a hardware accelerator, a software accelerator, or a combination thereof). For example, the SoC804 may include a hardware acceleration cluster that may include an optimized hardware accelerator and / or a large on-chip memory. The large on-chip memory (e.g., 4 MB of SRAM) may enable the hardware acceleration cluster to accelerate neural networks and other computations. The hardware acceleration cluster may be used to complement the GPU808 and to offload some of the GPU808's tasks (e.g., to free up more cycles of the GPU808 to perform other tasks). As an example, accelerator 814 may be used for target workloads that are sufficiently stable to be suitable for acceleration (e.g., perception, convolutional neural networks (CNNs), etc.). In this specification, the term "CNN" may include all types of CNNs, including region-based or regional convolutional neural networks (RCNNs) and fast RCNNs (for example, as used for object detection).

[0102] Accelerator 814 (e.g., hardware acceleration cluster) may include a deep learning accelerator (DLA). A DLA may include one or more tensor processing units (TPUs) that can be configured to provide an additional 10 trillion operations per second for deep learning applications and inference. The TPU may also be an accelerator configured and optimized to perform image processing functions (e.g., CNN, RCNN, etc.). The DLA may further be optimized for a specific set of neural network types and floating-point operations, as well as for inference. A DLA design can provide more performance per millisecond than a general-purpose GPU and significantly exceed CPU performance. The TPU can perform several functions, including, for example, single-instance convolutional functions supporting INT8, INT16, and FP16 data types for both features and weights, as well as post-processing functions.

[0103] DLA can quickly and efficiently run neural networks, particularly CNNs, on processed or unprocessed data for any of a variety of functions, including but not limited to: CNNs for object recognition and detection using data from camera sensors; CNNs for distance estimation using data from camera sensors; CNNs for emergency vehicle detection, identification, and detection using data from microphones; CNNs for facial recognition and mobile vehicle owner identification using data from camera sensors; and / or CNNs for security and / or safety-related events.

[0104] DLA can perform any function of GPU808, and by using inference accelerators, for example, a designer can target either DLA or GPU808 for any function. For example, a designer can focus on CNN and floating-point arithmetic processing on DLA, and leave other functions to GPU808 and / or other accelerators 814.

[0105] The accelerator 814 (for example, a hardware accelerator cluster) may include a programmable vision accelerator (PVA), which may be referred to herein as a computer vision accelerator. A PVA may be designed and configured to accelerate computer vision algorithms for advanced driver assistance systems (ADAS), autonomous driving, and / or augmented reality (AR) and / or virtual reality (VR) applications. A PVA can provide a balance between performance and flexibility. For example, each PVA may, but is not limited to, any number of reduced instruction set computer (RISC) cores, direct memory access (DMA), and / or any number of vector processors.

[0106] A RISC core can interact with an image sensor (for example, the image sensor of one of the cameras described herein), an image signal processor, and / or similar devices. Each RISC core may contain any amount of memory. Depending on the embodiment, a RISC core may use one of several protocols. In some examples, a RISC core can run a real-time operating system (RTOS). A RISC core may be implemented using one or more integrated circuit devices, application-specific integrated circuits (ASICs), and / or memory devices. For example, a RISC core may include an instruction cache and / or tightly coupled RAM.

[0107] DMA can enable PVA components to access system memory independent of the CPU 806. DMA can support any number of features used to bring optimizations to the PVA, including but not limited to supporting multidimensional addressing and / or circular addressing. In some examples, DMA can support up to six or more dimensions of addressing, which may include block width, block height, block depth, horizontal block stepping, vertical block stepping, and / or depth stepping.

[0108] A vector processor may also be a programmable processor that can be designed to efficiently and flexibly execute the programming of computer vision algorithms and provide signal processing capabilities. In some examples, a PVA may include a PVA core and two vector processing subsystem partitions. The PVA core may include a processor subsystem, a DMA engine (e.g., two DMA engines), and / or other peripherals. The vector processing subsystem can act as the primary processing engine of the PVA and may include a vector processing unit (VPU), an instruction cache, and / or vector memory (e.g., VMEM). The VPU core may include a digital signal processor, such as a single-instruction, multiple-data (SIMD), or very-long instruction word (VLIW) digital signal processor. A combination of SIMD and VLIW can increase throughput and speed.

[0109] Each vector processor may include an instruction cache and be linked to dedicated memory. As a result, in some examples, each vector processor may be configured to run independently of other vector processors. In other examples, the vector processors included in a particular PVA may be configured to use data parallelism. For example, in some embodiments, multiple vector processors included in a single PVA can run the same computer vision algorithm, but on different regions of an image. In other examples, the vector processors included in a particular PVA can run different computer vision algorithms simultaneously on the same image, or even run different algorithms sequentially on the image or parts of an image. In particular, any number of PVAs may be included in a hardware acceleration cluster, and any number of vector processors may be included in each PVA. In addition, a PVA may include additional error correction code (ECC) memory to enhance overall system safety.

[0110] The accelerator 814 (for example, a hardware accelerator cluster) may include a computer vision network on-chip and SRAM to provide high-bandwidth, low-latency SRAM for the accelerator 814. In some examples, the on-chip memory may include at least 4 MB of SRAM consisting of eight field-configurable memory blocks, which may be accessible by both the PVA and DLA, for example, and not limited to. Each pair of memory blocks may include an advanced peripheral bus (APB) interface, configuration circuitry, a controller, and a multiplexer. Any type of memory may be used. The PVA and DLA can access the memory via a backbone that provides the PVA and DLA with high-speed access to the memory. The backbone may include a computer vision network on-chip that interconnects the PVA and DLA to the memory (for example, using an APB).

[0111] A computer vision network on-chip may include an interface that determines whether both the PVA and DLA are activatable and enable signals before any control signals / addresses / data are transmitted. Such an interface can provide separate phases and separate channels for transmitting control signals / addresses / data, as well as burst-type communication for continuous data transfer. This type of interface may conform to ISO 26262 or IEC 61508 standards, but other standards and protocols may be used.

[0112] In some embodiments, the SoC804 may include a real-time ray tracing hardware accelerator, as described in U.S. Patent Application No. 16 / 101,232, filed August 10, 2018. The real-time ray tracing hardware accelerator may be used to quickly and efficiently determine the location and size of objects (e.g., in a world model) to generate real-time visualization simulations for RADAR signal interpretation, acoustic propagation synthesis and / or analysis, SONAR system simulation, general wave propagation simulation, comparison to LIDAR data for localization and / or other functions, and / or other uses. In some embodiments, one or more tree traversal units (TTUs) may be used to perform one or more ray tracing-related operations.

[0113] The accelerator 814 (e.g., a hardware accelerator cluster) has diverse applications for autonomous driving. The PVA may also be a programmable vision accelerator that can be used in critical processing stages in ADAS and autonomous vehicles. The PVA's capabilities are suitable for areas of algorithms requiring predictable processing at low power and low latency. In other words, the PVA performs well in semi-high density or high density typical computations, even on small data sets, where predictable execution time is required along with low latency and low power. Therefore, because the PVA is efficient in object detection and integer computation, in relation to a platform for autonomous vehicles, the PVA is designed to run classic computer vision algorithms.

[0114] For example, according to one embodiment of this technology, PVA is used to perform computer stereo vision. While semi-global matching-based algorithms may be used in some examples, this is not intended to be a limitation. Numerous applications for Level 3-5 autonomous driving require motion estimation / stereo matching on the fly (e.g., SFM (structure from motion), pedestrian recognition, lane detection, etc.). PVA can perform computer stereo vision functions with input from two monocular cameras.

[0115] In some applications, PVA can be used to perform high-density optical flow by processing raw RADAR data (e.g., using 4D Fast Fourier Transform) to provide processed RADAR data. In other applications, PVA is used for flight depth processing, for example, by processing raw flight data to provide processed flight data.

[0116] DLA can be used to run any type of network to enhance control and driving safety, for example, a neural network that outputs a confidence value for each object detection. Such confidence values ​​can be interpreted as probabilities or as providing the relative "weight" of each detection compared to other detections. This confidence value allows the system to make further decisions about which detections should be considered true positives rather than false positives. For example, the system can set a confidence threshold and consider only detections that exceed the threshold as true positives. In an automatic emergency braking (AEB) system, a false positive detection would cause the moving vehicle to automatically apply the emergency brakes, which is obviously undesirable. Therefore, only the most confident detections should be considered as triggers for the AEB. DLA can run a neural network that devolves the confidence values. The neural network can accept at least a subset of parameters as its input, such as bounding box dimensions, ground plane estimation acquired (e.g., from another subsystem), object orientation, distance, inertial measurement unit (IMU) sensor output correlated with 3D position estimation of the moving object acquired from the neural network and / or other sensors (e.g., LIDAR sensor 864 or RADAR sensor 860), and others.

[0117] The SoC804 may include a data store 816 (for example, memory). The data store 816 may also be the on-chip memory of the SoC804 and can store neural networks that will run on the GPU and / or DLA. In some examples, the data store 816 may have a capacity large enough to store multiple instances of the neural network for redundancy and safety. The data store 812 may comprise an L2 or L3 cache 812. References to the data store 816 may include references to memory associated with the PVA, DLA, and / or other accelerators 814, as described herein.

[0118] The SoC804 may include one or more processors 810 (e.g., embedded processors). The processors 810 may include a boot and power management processor, which may be a dedicated processor and subsystem for handling boot power and management capabilities and associated security enforcement. The boot and power management processor may also be part of the SoC804 boot sequence and can provide runtime power management services. The boot power and management processor may provide clock and voltage programming, assistance with system low-power state transitions, management of SoC804 thermal and temperature sensors, and / or management of SoC804 power states. Each temperature sensor may be implemented as a ring oscillator whose output frequency is proportional to temperature, and the SoC804 may use the ring oscillators to detect the temperatures of the CPU 806, GPU 808, and / or accelerator 814. If the temperature is determined to have exceeded a threshold, the boot and power management processor may enter a temperature fault routine, placing the SoC804 into a lower power state and / or putting the mobile vehicle 800 into chauffeur safe shutdown mode (for example, safely shutting down the mobile vehicle 800).

[0119] The processor 810 may further include a set of integrated processors that can perform the functions of an audio processing engine. The audio processing engine may also be an audio subsystem enabling full hardware support for multi-channel audio through multiple interfaces and a wide and flexible range of audio I / O interfaces. In some examples, the audio processing engine is a dedicated processor core having a digital signal processor with dedicated RAM.

[0120] The processor 810 may further include an always-on processor engine that can provide the necessary hardware features to support low-power sensor management and wake use cases. The always-on processor engine may include a processor core, tightly coupled RAM, support peripherals (e.g., timer and interrupt controllers), various I / O controller peripherals, and routing logic.

[0121] The processor 810 may further include a safety cluster engine, which includes a dedicated processor subsystem for handling safety management in automotive applications. The safety cluster engine may include two or more processor cores, tightly coupled RAM, supporting peripherals (e.g., timers, interrupt controllers, etc.), and / or routing logic. In safety mode, the two or more cores may operate in lockstep mode and function as a single core with comparison logic to detect any differences between their operations.

[0122] The processor 810 may further include a real-time camera engine, which may include a dedicated processor subsystem for handling real-time camera management.

[0123] The processor 810 may further include a high dynamic range signal processor, which may include an image signal processor, a hardware engine that is part of the camera processing pipeline.

[0124] The processor 810 may include a video image synthesizer, which may also be a processing block (for example, implemented on a microprocessor) that implements post-video processing functions required by the video playback application to produce the final image for the player window. The video image synthesizer can perform lens distortion correction on the wide-view camera 870, the surround camera 874, and / or the in-cabin surveillance camera sensors. The in-cabin surveillance camera sensors are preferably monitored by a neural network running on another instance of the advanced SoC, configured to identify and appropriately respond to in-cabin events. The in-cabin system can perform lip-reading to activate cellular services and make phone calls, transcribe emails, change the vehicle's destination, activate or change the vehicle's infotainment system and settings, or provide voice-activated web surfing. Certain functions are available to the driver only when operating in autonomous mode and are otherwise disabled.

[0125] A video image synthesizer may include enhanced temporal noise reduction for both spatial and temporal noise reduction. For example, if motion occurs in the video, noise reduction reduces the weight of information provided by adjacent frames and appropriately weights the spatial information. If the image or part of the image does not contain motion, the temporal noise reduction performed by the video image synthesizer can use information from previous images to reduce noise in the current image.

[0126] The video image synthesizer can also be configured to perform stereo rectification on the input stereo lens frame. Furthermore, the video image synthesizer can be used for user interface compositing when the operating system desktop is in use, so that the GPU808 is not required to continuously render new surfaces. Even when the GPU808 is powered on and actively performing 3D rendering, the video image synthesizer can be used to offload the GPU808 to improve performance and responsiveness.

[0127] The SoC804 may further include a Mobile Industry Processor Interface (MIPI) camera serial interface, a high-speed interface, and / or a video input block that can be used for camera and associated pixel input functions to receive video and input from a camera. The SoC804 may further include an input / output controller that can be controlled by software and can be used to receive I / O signals that are not committed to a specific role.

[0128] The SoC804 may further include a wide range of peripheral interfaces to enable communication with peripheral devices, audio codecs, power management, and / or other devices. The SoC804 may be used to process data from cameras (connected, for example, via Gigabit Multimedia Serial Link and Ethernet®), sensors (e.g., LiDAR sensor 864, RADAR sensor 860, etc., which may be connected via Ethernet®), data from bus 802 (e.g., speed of vehicle 800, steering wheel position, etc.), and data from GNSS sensor 858 (connected, for example, via Ethernet® or CAN bus). The SoC804 may further include its own DMA engine and a dedicated high-performance mass storage controller which may be used to free up CPU 806 from routine data management tasks.

[0129] The SoC804 may also be an inter-terminal platform with a flexible architecture that extends to automation levels 3-5, thereby providing a comprehensive functional safety architecture that leverages and efficiently uses computer vision and ADAS techniques for diversity and redundancy, and provides a platform for a flexible, reliable driving software stack along with deep learning tools. The SoC804 can be faster, more reliable, more energy-efficient, and more space-efficient than conventional systems. For example, when the accelerator 814 is coupled with the CPU 806, the GPU 808, and the data store 816 can provide a fast and efficient platform for autonomous vehicles at levels 3-5.

[0130] Therefore, this technology brings capabilities and functionality that cannot be achieved by conventional systems. For example, computer vision algorithms can be executed on a CPU, which can be configured using high-level programming languages ​​such as the C programming language to execute a wide variety of processing algorithms across a wide variety of visual data. However, CPUs often cannot meet the performance requirements of many computer vision applications, such as those related to execution time and power consumption. Specifically, many CPUs cannot execute real-time complex object detection algorithms, which are required for in-vehicle ADAS applications and actual Level 3-5 autonomous vehicles.

[0131] In contrast to conventional systems, by providing CPU complexes, GPU complexes, and hardware acceleration clusters, the technologies described herein enable multiple neural networks to run simultaneously and / or sequentially, and the results to be combined to enable Level 3–5 autonomous driving capabilities. For example, a DLA or CNN running on a dGPU (e.g., GPU820) may include text and word recognition, enabling a supercomputer to read and understand traffic signs, including signs for which the neural network has not been specifically trained. The DLA may further include a neural network capable of identifying, interpreting, and providing a semantic understanding of signs and passing that semantic understanding to a route planning module running on the CPU complex.

[0132] As another example, multiple neural networks may run simultaneously, as required for Level 3, 4, or 5 driving. For instance, a warning sign consisting of a flashing light and the text "Caution: Flashing light indicates frozen conditions" may be interpreted independently or collectively by several neural networks. The sign itself may be identified as a traffic sign by a first deployed neural network (e.g., a trained neural network), and the text "Flashing light indicates frozen conditions" may be interpreted by a second deployed neural network, informing the vehicle's route planning software (preferably running on a CPU complex) that frozen conditions are present when flashing light is detected. The flashing light may be identified by informing the vehicle's route planning software of the presence (or absence) of the flashing light, and by operating a third deployed neural network through multiple frames. All three neural networks can run simultaneously within the DLA and / or on the GPU808, for example.

[0133] In some applications, a CNN for facial recognition and vehicle owner identification can use data from camera sensors to identify the presence of the legitimate driver and / or owner of the vehicle 800. An always-on sensor processing engine may be used to unlock the vehicle and turn on the lights when the owner approaches the driver's side door, and, in security mode, to stop the vehicle when the owner leaves it. In this way, the SoC804 provides security against theft and / or vehicle hijacking.

[0134] In another example, a CNN for emergency vehicle detection and identification can detect and identify emergency vehicle sirens using data from microphone 896. In contrast to conventional systems that use a general classifier to detect sirens and manually extract features, SoC804 uses a CNN for classifying environmental and urban sounds, as well as for classifying visual data. In a preferred embodiment, a CNN running on DLA is trained to identify the relative terminal velocity of emergency vehicles (for example, by using the Doppler effect). The CNN may also be trained to identify emergency vehicles specific to the local area in which the vehicle is operating, as identified by GNSS sensor 858. Thus, for example, when operating in Europe, the CNN would attempt to detect European sirens, and when in the United States, the CNN would attempt to identify only North American sirens. After an emergency vehicle is detected, a control program may be used, with the assistance of ultrasonic sensor 862, to perform emergency vehicle safety routines such as slowing down the vehicle, stopping it at the side of the road, parking the vehicle, and / or idling the vehicle until the emergency vehicle has passed.

[0135] The mobile vehicle may include a CPU 818 (e.g., a separate CPU, or dCPU) which can be connected to the SoC 804 via a high-speed interconnect (e.g., PCIe). The CPU 818 may include, for example, an x86 processor. The CPU 818 may be used to perform any of a variety of functions, including, for example, mediating the consequences of a potential mismatch between ADAS sensors and the SoC 804, and / or monitoring the status and condition of the controller 836 and / or the infotainment SoC 830.

[0136] The mobile vehicle 800 may include a GPU 820 (e.g., a separate GPU, or dGPU) which can be connected to the SoC 804 via a high-speed interconnect (e.g., NVIDIA NVLINK). The GPU 820 can provide additional artificial intelligence capabilities, such as by running redundant and / or different neural networks, and may be used to train and / or update neural networks based on input from the mobile vehicle 800's sensors (e.g., sensor data).

[0137] The mobile vehicle 800 may further include a network interface 824 which may include one or more wireless antennas 826 (e.g., one or more wireless antennas for different communication protocols, such as cellular antennas and Bluetooth® antennas). The network interface 824 may be used to enable wireless connectivity to a cloud over the Internet (e.g., with server 878 and / or other network devices), to other mobile vehicles, and / or to computing devices (e.g., passenger client devices). To communicate with other mobile vehicles, a direct link may be established between two mobile vehicles, and / or an indirect link may be established (e.g., over a network and over the Internet). The direct link may be provided using a mobile vehicle-to-mobile communication link. The mobile vehicle-to-mobile communication link can provide mobile vehicle 800 information about mobile vehicles in close proximity to mobile vehicle 800 (e.g., mobile vehicles in front of, beside, and / or behind mobile vehicle 800). This functionality may also be part of the mobile vehicle 800's joint adaptive cruise control functionality.

[0138] The network interface 824 may include an SoC that provides modulation and demodulation functions and enables the controller 836 to communicate over a wireless network. The network interface 824 may include a radio frequency front end for up-conversion from baseband to radio frequency and down-conversion from radio frequency to baseband. Frequency conversion can be performed through well-known processes and / or using a superheterodyne process. In some examples, the radio frequency front end functionality may be provided by a separate chip. The network interface may include wireless functionality for communication over LTE, WCDMA®, UMTS, GSM, CDMA2000, Bluetooth®, Bluetooth® LE, Wi-Fi, Z-Wave, ZigBee, LoRaWAN, and / or other wireless protocols.

[0139] The mobile unit 800 may further include a data store 828 which may include storage outside the chip (for example, outside the SoC 804). The data store 828 may include one or more storage elements, including RAM, SRAM, DRAM, VRAM, flash, hard disk, and / or other components and / or devices capable of storing at least one bit of data.

[0140] The vehicle 800 may further include GNSS sensors 858. The GNSS sensors 858 (e.g., GPS, assisted GPS sensors, differential GPS (DGPS) sensors, etc.) assist in mapping, perception, occupy grid generation, and / or route planning functions. Any number of GNSS sensors 858 may be used, including, but not limited to, GPS using a USB connector with Ethernet® to a serial (RS-232) bridge.

[0141] The mobile vehicle 800 may further include a RADAR sensor 860. The RADAR sensor 860 may be used by the mobile vehicle 800 for long-range mobile vehicle detection, even in darkness and / or severe weather conditions. The RADAR functional safety level may be ASIL B. In some examples, the RADAR sensor 860 may use CAN and / or bus 802 for control and to access object tracking data (for example, to transmit data generated by the RADAR sensor 860) using Ethernet® access for accessing raw data. A wide variety of RADAR sensor types may be used. For example, and without limitation, the RADAR sensor 860 may be suitable for front, rear, and side RADAR use. In some examples, a pulsed Doppler RADAR sensor may be used.

[0142] The RADAR sensor 860 may include different configurations, such as long-range with a narrow field of view, short-range with a wide field of view, and short-range side coverage. In some examples, the long-range RADAR may be used for adaptive cruise control functions. The long-range RADAR system can provide a wide field of view achieved by two or more independent scans, such as within a range of 250m. The RADAR sensor 860 can help distinguish between static and moving objects and may be used by ADAS systems for emergency brake assist and forward collision warning. The long-range RADAR sensor may include monostatic multimodal RADARs with multiple (e.g., six or more) fixed RADAR antennas and high-speed CAN and FlexRay interfaces. In one example with six antennas, the four central antennas may create a focused beam pattern designed to record around the moving vehicle 800 at high speed with minimal interference from traffic in adjacent lanes. The other two antennas can widen the field of view, enabling rapid detection of moving vehicles entering or leaving the lane of the moving vehicle 800.

[0143] As an example, a medium-range RADAR system may include a range of up to 860m (forward) or 80m (rear) and a field of view of up to 42 degrees (forward) or 850 degrees (rear). A short-range RADAR system may include, but is not limited to, RADAR sensors designed to be mounted on both ends of the rear bumper. When mounted on both ends of the rear bumper, such a RADAR sensor system can create two beams that constantly monitor the blind spots behind and beside the moving vehicle.

[0144] Short-range radar systems can be used in ADAS systems for blind spot detection and / or lane change assistance.

[0145] The mobile vehicle 800 may further include ultrasonic sensors 862. Positioned on the front, rear, and / or sides of the mobile vehicle 800, the ultrasonic sensors 862 may be used for parking assistance and / or for creating and updating the occupancy grid. A wide variety of ultrasonic sensors 862 may be used, and different ultrasonic sensors 862 may be used for detection at different ranges (e.g., 2.5m, 4m). The ultrasonic sensors 862 may operate at a functional safety level of ASIL B.

[0146] The mobile vehicle 800 may include a LiDAR sensor 864. The LiDAR sensor 864 may be used for object and pedestrian detection, emergency braking, collision avoidance, and / or other functions. The LiDAR sensor 864 may also have a functional safety level of ASIL B. In some examples, the mobile vehicle 800 may include multiple LiDAR sensors 864 (e.g., two, four, six, etc.) that can use Ethernet® (for example, to provide data to a Gigabit Ethernet® switch).

[0147] In some examples, the LIDAR sensor 864 may have the ability to provide a list of objects and their distances within a 360-degree field of view. A commercially available LIDAR sensor 864 may have an advertised range of approximately 800m, for example, with an accuracy of 2cm to 3cm and support for 800Mbps Ethernet® connectivity. In some examples, one or more non-protruding LIDAR sensors 864 may be used. In such examples, the LIDAR sensor 864 may be implemented as a small device that can be incorporated into the front, rear, side, and / or corners of a mobile vehicle 800. In such examples, the LIDAR sensor 864 may have a range of 200m even for low-reflection objects and can provide a field of view up to 120 degrees horizontal and 35 degrees vertical. A front-mounted LIDAR sensor 864 may be configured for a horizontal field of view between 45 and 135 degrees.

[0148] In some applications, LiDAR technologies such as 3D flash LiDAR may also be used. 3D flash LiDAR uses a laser flash as a source to illuminate the area around a moving vehicle up to approximately 200m. The flash LiDAR unit includes receptors that record the laser pulse travel time and reflected light on each pixel, sequentially corresponding to the range from the moving vehicle to the object. Flash LiDAR can enable the generation of high-precision and distortion-free images of the surroundings with every laser flash. In some applications, four flash LiDAR sensors may be deployed, one on each side of the moving vehicle. Available 3D flash LiDAR systems include solid-state 3D steering array LiDAR cameras (e.g., non-scanning LiDAR devices) that have no moving parts other than a blower. Flash LiDAR devices can use 5 nanosecond Class I (eye-safe) laser pulses per frame and can capture reflected laser light in the form of a 3D range point cloud and co-documented intensity data. By using flash LiDAR, and because flash LiDAR is a solid-state device with no moving parts, the LiDAR sensor 864 may be less susceptible to motion blur, vibration, and / or shock.

[0149] The mobile vehicle may further include an IMU sensor 866. In some examples, the IMU sensor 866 may be positioned in the center of the rear axle of the mobile vehicle 800. The IMU sensor 866 may include, but is not limited to, an accelerometer, magnetometer, gyroscope, magnetic compass, and / or other sensor types. In some examples, such as in a 6-axis application, the IMU sensor 866 may include an accelerometer and a gyroscope, while in a 9-axis application, the IMU sensor 866 may include an accelerometer, a gyroscope, and a magnetometer.

[0150] In some embodiments, the IMU sensor 866 may be implemented as a miniature, high-performance GPS-aided inertial navigation system (GPS / INS) that combines a micro-electro-mechanical system (MEMS) inertial sensor, a high-sensitivity GPS receiver, and an advanced Kalman filtering algorithm to provide estimates of position, velocity, and attitude. As such, in some examples, the IMU sensor 866 may enable the moving vehicle 800 to estimate its direction of travel without requiring input from a magnetic sensor by directly observing and correlating velocity changes from the GPS to the IMU sensor 866. In some embodiments, the IMU sensor 866 and the GNSS sensor 858 may be combined in a single integrated unit.

[0151] The mobile vehicle may include a microphone 896 placed inside and / or around the mobile vehicle 800. The microphone 896 may, among other things, be used for emergency vehicle detection and identification.

[0152] The mobile vehicle may further include any number of camera types, including a stereo camera 868, a wide-view camera 870, an infrared camera 872, a surround camera 874, a long-range and / or medium-range camera 898, and / or other camera types. The cameras may be used to capture image data around the entire exterior surface of the mobile vehicle 800. The type of camera used will depend on the embodiment and requirements of the mobile vehicle 800, and any combination of camera types may be used to achieve the required coverage around the mobile vehicle 800. In addition, the number of cameras may vary depending on the embodiment. For example, the mobile vehicle may include six cameras, seven cameras, ten cameras, twelve cameras, and / or another number of cameras. The cameras may, as an example, support Gigabit Multimedia Serial Link (GMSL) and / or Gigabit Ethernet®. Each camera is described in more detail herein in relation to Figures 8A and 8B.

[0153] The mobile vehicle 800 may further include a vibration sensor 842. The vibration sensor 842 can measure vibrations of components of the mobile vehicle, such as axles. For example, a change in vibration may indicate a change in the road surface. In another example, when two or more vibration sensors 842 are used, the difference in vibration may be used to determine friction or slippage of the road surface (for example, when the difference in vibration is between a power-driven axle and a free-rotating axle).

[0154] The mobile vehicle 800 may include an ADAS system 838. In some examples, the ADAS system 838 may include a System of Control (SoC). The ADAS system 838 may include autonomous / adaptive / automatic cruise control (ACC), cooperative adaptive cruise control (CACC), forward crash warning (FCW), automatic emergency braking (AEB), lane departure warning (LDW), lane keep assist (LKA), blind spot warning (BSW), rear cross-traffic warning (RCTW), collision warning system (CWS), lane centering (LC), and / or other features and functions.

[0155] The ACC system may utilize a radar sensor 860, a lithium-ion sensor 864, and / or cameras. The ACC system may include longitudinal ACC and / or transverse ACC. Longitudinal ACC monitors and controls the distance of vehicle 800 to the vehicle immediately in front of it and automatically adjusts the vehicle speed to maintain a safe distance from the vehicle ahead. Transverse ACC performs distance maintenance and advises vehicle 800 to change lanes when necessary. Transverse ACC is related to other ADAS applications such as LCA and CWS.

[0156] CACC uses information from other vehicles that can be received from other vehicles via a wireless link through the network interface 824 and / or wireless antenna 826, or indirectly via a network connection (e.g., via the Internet). Direct links may be provided by vehicle-to-vehicle (V2V) communication links, while indirect links may be infrastructure-to-vehicle (I2V) communication links. Generally, the V2V communication concept provides information about the vehicle immediately ahead (e.g., a vehicle in the same lane as vehicle 800, immediately in front of vehicle 800), while the I2V communication concept provides information about traffic further ahead. A CACC system may include either or both I2V and V2V information sources. Given information about vehicles ahead of vehicle 800, CACC can be more reliable, and CACC has the potential to make traffic flow smoother and reduce road congestion.

[0157] The FCW system is designed to warn the driver of hazards so that the driver can take corrective action. The FCW system uses a forward-facing camera and / or radar sensor 860, coupled to a dedicated processor, DSP, FPGA, and / or ASIC, electrically coupled to driver feedback such as a display, speaker, and / or vibration components. The FCW system can provide warnings in the form of audible, visual, vibration, and / or quick brake pulses.

[0158] An AEB system can detect an imminent forward collision with another moving vehicle or other object and automatically apply the brakes if the driver does not take corrective action within a specified time or distance parameter. The AEB system may use a forward-facing camera and / or radar sensor 860 coupled to a dedicated processor, DSP, FPGA, and / or ASIC. When the AEB system detects a hazard, it typically first warns the driver to take corrective action to avoid the collision. If the driver does not take corrective action, the AEB system may automatically apply the brakes as part of an effort to prevent, or at least mitigate, the impact of the anticipated collision. The AEB system may include techniques such as dynamic brake support and / or impending collision braking.

[0159] The LDW system warns the driver when the vehicle 800 crosses a lane marking by providing visual, audible, and / or tactile warnings, such as vibration of the steering wheel or seat. The LDW system does not activate when the driver indicates an intentional lane departure by activating the turn signal. The LDW system may use a forward-facing camera connected to a dedicated processor, DSP, FPGA, and / or ASIC, which is electrically coupled to driver feedback, such as a display, speaker, and / or vibration components.

[0160] The LKA system is a modified version of the LDW system. The LKA system provides steering input or braking to correct the mobile vehicle 800 if it begins to drift out of its lane.

[0161] The BSW system detects and warns the driver of a moving vehicle in the vehicle's blind spot. The BSW system can provide visual, audible, and / or tactile warnings to indicate that merging or changing lanes is unsafe. The system can provide additional warnings when the driver uses the turn signal. The BSW system can use a rear-facing camera and / or radar sensor 860 coupled to a dedicated processor, DSP, FPGA, and / or ASIC, electrically coupled to driver feedback, such as a display, speaker, and / or vibration component.

[0162] The RCTW system can provide visual, audible, and / or haptic notifications when an object is detected outside the range of the rear camera while the vehicle 800 is reversing. Some RCTW systems include AEB to ensure that the vehicle brakes are applied to avoid a collision. The RCTW system may use one or more rear-facing RADAR sensors 860 coupled to a dedicated processor, DSP, FPGA, and / or ASIC, which are electrically coupled to driver feedback, such as a display, speaker, and / or vibration component.

[0163] Conventional ADAS systems warn the driver and allow the driver to determine whether a safe condition truly exists and act accordingly. However, conventional ADAS systems have sometimes tended to produce misjudgments that, while not usually catastrophic, can be troubling and distracting to the driver. In the autonomous vehicle 800, however, if the results are contradictory, the vehicle 800 itself must decide whether to heed the results from the primary computer or the secondary computer (e.g., the first controller 836 or the second controller 836). For example, in some embodiments, the ADAS system 838 may also be a backup and / or secondary computer to provide perceptual information to a backup computer rationality module. The backup computer rationality monitor can run a variety of redundant software on hardware components to detect failures in perceptual and dynamic driving tasks. The output from the ADAS system 838 may be provided to the supervisory MCU. If the outputs from the primary and secondary computers are contradictory, the supervisory MCU must decide how to reconcile the contradiction to ensure safe operation.

[0164] In some implementations, a primary computer may be configured to provide a supervising MCU with a reliability score indicating the reliability of the primary computer in a selected outcome. If the reliability score exceeds a threshold, the supervising MCU may follow the primary computer's instructions, regardless of whether the secondary computer gives conflicting or inconsistent results. If the reliability score does not meet the threshold, and the primary and secondary computers produce different results (e.g., conflicting results), the supervising MCU may mediate between the computers to determine an appropriate outcome.

[0165] The supervisory MCU may be configured to run a neural network trained and configured to determine, based on the outputs from the primary and secondary computers, when a secondary computer provides a false alarm. Thus, the neural network in the supervisory MCU can learn when the output of the secondary computer is reliable and when it is not. For example, when the secondary computer is a radar-based forward crossing (FCW) system, the neural network in the supervisory MCU can learn when the FCW identifies metal objects that are not actually dangerous, such as sewer grates or manhole covers that trigger an alarm. Similarly, when the secondary computer is a camera-based lane departure warning (LDW) system, the neural network in the supervisory MCU can learn to ignore the LDW when a cyclist or pedestrian is present and lane departure is actually the safest operation. In embodiments involving a neural network running on the supervisory MCU, the supervisory MCU may include at least one DLA or GPU suitable for running a neural network with associated memory. In a preferred embodiment, the supervisory MCU may comprise and / or be included as a component of the SoC804.

[0166] In other examples, ADAS system 838 may include a secondary computer that performs ADAS functions using conventional rules of computer vision. As such, the secondary computer may use classical computer vision rules (if-then), and the presence of a neural network within the supervisory MCU can improve reliability, safety, and performance. For example, diverse implementations and intentional non-identities make the entire system more fault-tolerant, particularly to failures caused by software (or software-hardware interface) functions. For instance, if a software bug or error exists in the software running on the primary computer, and non-identical software code running on the secondary computer produces the same overall result, the supervisory MCU may have greater confidence that the overall result is correct and that the bug in the software or hardware on the primary computer did not cause a critical error.

[0167] In some examples, the output of the ADAS system 838 may be supplied to the perception block and / or the dynamic driving task block of the primary computer. For example, if the ADAS system 838 indicates a forward collision warning due to an object immediately ahead, the perception block can use this information when identifying the object. In other examples, the secondary computer may have its own neural network, which is trained as described herein and therefore reduces the risk of misjudgment.

[0168] The mobile vehicle 800 may further include an infotainment SoC 830 (for example, an in-vehicle infotainment system (IVI)). Although illustrated and described as an SoC, the infotainment system does not have to be an SoC and may include two or more separate components. The infotainment SoC 830 may include a combination of hardware and software that can be used to provide the mobile vehicle 800 with audio (e.g., music, personal digital assistant, navigation commands, news, radio, etc.), video (e.g., TV, movies, streaming, etc.), telephone (e.g., hands-free calling), network connectivity (e.g., LTE, Wi-Fi, etc.), and / or information services (e.g., navigation system, rear parking assist, radio data system, fuel level, total mileage, brake fuel level, oil level, door open / close, air filter information, and other mobile vehicle-related information). For example, the infotainment SoC 830 may also include wireless, disc player, navigation system, video player, USB and Bluetooth® connectivity, car computer, in-car entertainment, Wi-Fi, steering wheel audio control unit, hands-free voice control, heads-up display (HUD), HMI display 834, telematics device, control panel (for example, for controlling and / or interacting with various components, features, and / or systems), and / or other components. The infotainment SoC 830 may be further used to provide information (for example, visual and / or audible) to the user of the vehicle, such as information from the ADAS system 838, autonomous driving information such as planned vehicle operation, trajectory, surrounding environment information (for example, intersection information, vehicle information, road information, etc.), and / or other information.

[0169] The infotainment SoC 830 may include GPU functionality. The infotainment SoC 830 can communicate with other devices, systems, and / or components of the vehicle 800 via bus 802 (e.g., CAN bus, Ethernet®, etc.). In some examples, the infotainment SoC 830 may be coupled to a supervisory MCU so that the infotainment system's GPU can perform certain self-drive functions in the event of a primary controller 836 (e.g., the primary and / or backup computer of the vehicle 800) failure. In such examples, the infotainment SoC 830 can put the vehicle 800 into a chauffeur-safe stop mode as described herein.

[0170] The mobile vehicle 800 may further include an instrument cluster 832 (e.g., a digital dash, an electronic instrument cluster, a digital instrument panel, etc.). The instrument cluster 832 may include a controller and / or a supercomputer (e.g., a separate controller or supercomputer). The instrument cluster 832 may include a set of instruments such as a speedometer, fuel level indicator, oil pressure indicator, tachometer, odometer, turn signals, gear shift position indicator, seat belt warning light, parking brake warning light, engine fault light, airbag (SRS) system information, lighting control device, safety system control device, and navigation information. In some examples, information may be displayed and / or shared between the infotainment SoC 830 and the instrument cluster 832. In other words, the instrument cluster 832 may be included as part of the infotainment SoC 830, and vice versa.

[0171] Figure 8D is a system diagram of communication between the cloud-based server of Figure 8A and an exemplary autonomous vehicle 800, according to some embodiments of the present disclosure. System 876 may include a server 878, a network 890, and a mobile vehicle including the mobile vehicle 800. Server 878 may include a plurality of GPUs 884(A) to 884(H) (collectively referred to herein as GPU 884), PCIe switches 882(A) to 882(H) (collectively referred to herein as PCIe switch 882), and / or CPUs 880(A) to 880(B) (collectively referred to herein as CPU 880). The GPUs 884, CPUs 880, and PCIe switches may be interconnected by high-speed interconnects, such as, for example, NVLink interfaces 888 and / or PCIe connections 886 developed by NVIDIA. In some examples, the GPU884 is connected via NVLink and / or NVSwitch SoCs, and the GPU884 and PCIe switch 882 are connected via PCIe interconnects. Eight GPU884s, two CPU880s, and two PCIe switches are illustrated, but this is not intended to be an limitation. Depending on the embodiment, each server 878 may contain any number of GPU884s, CPU880s, and / or PCIe switches. For example, server 878 may contain eight, sixteen, thirty-two, and / or more GPU884s, respectively.

[0172] Server 878 can receive image data from mobile vehicles via network 890, representing images showing unexpected or altered road conditions, such as recently started road construction. Server 878 can transmit map information 894, including information about traffic and road conditions, to mobile vehicles via network 890, including information about neural networks 892, updated neural networks 892, and / or map information 894. Updates to map information 894 may include updates to HD map 822, such as information about construction sites, potholes, detours, floods, and / or other obstacles. In some instances, neural networks 892, updated neural networks 892, and / or map information 894 may have arisen from new training and / or experience represented in data received from any number of mobile vehicles in the environment, and / or based on training performed in a data center (for example, using server 878 and / or other servers).

[0173] Server 878 may be used to train a machine learning model (e.g., a neural network) based on training data. The training data may be generated by a mobile device and / or in a simulation (e.g., using a game engine). In some instances, the training data is tagged (e.g., if the neural network benefits from supervised learning) and / or otherwise pre-processed, while in other instances, the training data is not tagged and / or pre-processed (e.g., if the neural network does not require supervised learning). Training may be performed according to any one or more classes of machine learning techniques, including but not limited to the following: supervised training, semi-supervised training, unsupervised training, self-learning, reinforcement learning, associative learning, transfer learning, feature learning (including key component and cluster analysis), multilinear subspace learning, manifold learning, representation learning (including pre-dictionary learning), rule-based machine learning, anomaly detection, and variations or combinations thereof. After the machine learning model has been traced, it may be used by the mobile vehicle (for example, transmitted to the mobile vehicle via network 890), and / or the machine learning model may be used by server 878 to remotely monitor the mobile vehicle.

[0174] In some examples, Server 878 can receive data from a mobile vehicle and apply that data to a state-of-the-art real-time neural network for real-time intelligent inference. Server 878 may include deep learning supercomputers and / or dedicated AI computers powered by GPU 884, such as the DGX and DGX Station Machines developed by NVIDIA. However, in some examples, Server 878 may include deep learning infrastructure that uses only CPU-powered data centers.

[0175] The deep learning infrastructure of server 878 can have the capability for high-speed real-time inference, which can be used to evaluate and verify the condition of the processor, software, and / or associated hardware within mobile vehicle 800. For example, the deep learning infrastructure can receive periodic updates from mobile vehicle 800, such as images of a sequence and / or objects located within images of that sequence (e.g., via computer vision and / or other machine learning object classification techniques). The deep learning infrastructure can run its own neural network to identify objects and compare them with objects identified by mobile vehicle 800. If the results do not match and the infrastructure concludes that the AI ​​within mobile vehicle 800 is not functioning correctly, server 878 can send a signal to mobile vehicle 800 instructing the failsafe computer in mobile vehicle 800 to infer control, notify passengers, and complete a safe parking operation.

[0176] For inference, server 878 may include GPU 884 and one or more programmable inference accelerators (e.g., NVIDIA TensorRT). The combination of a GPU-powered server and inference accelerator can enable real-time responsiveness. In other examples, such as when high performance is not required, a server powered by a CPU, FPGA, and other processors may be used for inference.

[0177] Exemplary computing devices Figure 9 is a block diagram of an example of a computing device 900 suitable for use in implementing some embodiments of the present disclosure. The computing device 900 may include an interconnection system 902 that indirectly or directly connects the following devices: memory 904, one or more central processing units (CPUs) 906, one or more graphics processing units (GPUs) 908, a communication interface 910, input / output (I / O) ports 912, input / output components 914, a power supply device 916, one or more presentation components 918 (e.g., a display), and one or more logical units 920. In at least one embodiment, the computing device 900 may include one or more virtual machines (VMs), and / or any of its components may include virtual components (e.g., virtual hardware components). As an unrestricted example, one or more of the GPUs 908 may include one or more vGPUs, one or more of the CPUs 906 may include one or more vCPUs, and / or one or more of the logical units 920 may include one or more virtual logical units. As such, the computing device 900 may include individual components (e.g., an entire GPU dedicated to the computing device 900), virtual components (e.g., a portion of a GPU dedicated to the computing device 900), or a combination thereof.

[0178] The various blocks in Figure 9 are shown connected by lines via the interconnection system 902, but this is not intended to be restrictive and is simply for clarity. For example, in some embodiments, a presentation component 918, such as a display device, could be considered an I / O component 914 (for example, if the display is a touchscreen). In another example, the CPU 906 and / or GPU 908 may include memory (for example, memory 904 may represent a storage device in addition to the memory of the GPU 908, CPU 906, and / or other components). In other words, the computing devices in Figure 9 are merely illustrative. Categories such as “workstation,” “server,” “laptop,” “desktop,” “tablet,” “client device,” “mobile device,” “handheld device,” “game console,” “electronic control unit (ECU),” “virtual reality system,” and / or other device or system types are all intended to fall within the scope of the computing devices in Figure 9 and are therefore not distinguished.

[0179] The interconnection system 902 may represent one or more links or buses, such as an address bus, a data bus, a control bus, or a combination thereof. The interconnection system 902 may include one or more bus or link types, such as an industry standard architecture (ISA) bus, an extended industry standard architecture (EISA) bus, a VESA (video electronics standards association) bus, a peripheral component interconnect (PCI) bus, a peripheral component interconnect express (PCIe) bus, and / or other types of buses or links. In some embodiments, direct connections exist between components. For example, the CPU 906 may be directly connected to the memory 904. Furthermore, the CPU 906 may be directly connected to the GPU 908. Where direct or point-to-point connections exist between components, the interconnection system 902 may include PCIe links to implement the connections. In these embodiments, the PCI bus does not need to be included in the computing device 900.

[0180] Memory 904 may include any of various computer-readable media. The computer-readable media may be any available media accessible by the computing device 900. The computer-readable media may include both volatile and non-volatile media, and removable and non-removable media. For example, but not limited to, the computer-readable media may include computer storage media and communication media.

[0181] Computer storage media may include both volatile and non-volatile media and / or removable and non-removable media implemented in any method or technique for storing information such as computer-readable instructions, data structures, program modules, and / or other data types. For example, memory 904 may store computer-readable instructions (e.g., representing programs and / or program elements), such as an operating system. Computer storage media may include, but are not limited to, RAM, ROM, EEPROM, flash memory, or other memory technologies, CD-ROM, digital versatile disk (DVD), or other optical disk storage, magnetic cassette, magnetic tape, magnetic disk storage, or other magnetic storage devices, or any other media that can be used to store desired information and can be accessed by computing device 900. In this specification, computer storage media does not include signals themselves.

[0182] Computer storage media include any information distribution medium that can implement computer-readable instructions, data structures, program modules, and / or other data types in modulated data signals such as carrier waves or other transfer mechanisms. The term “modulated data signal” may refer to a signal that has been modified in a manner that has one or more of its characteristic sets or encodes information within the signal. For example, but not limited to, computer storage media may include wired media such as wired networks or direct wired connections, and wireless media such as acoustic, RF, infrared, and other wireless media. Any combination of the foregoing should also be included in the scope of computer-readable media.

[0183] The CPU 906 may be configured to execute at least some computer-readable instructions to control one or more components of the computing device 900 to execute one or more of the methods and / or processes described herein. The CPU 906 may include one or more cores (e.g., one, two, four, eight, twenty-eight, seventy-two, etc.) each capable of processing a large number of software threads concurrently. The CPU 906 may include any type of processor, and depending on the type of computing device 900 in which it is implemented, it may include different types of processors (e.g., a processor with fewer cores for mobile devices and a processor with more cores for servers). For example, depending on the type of computing device 900, the processor may be an Advanced RISC Machines (ARM) processor implemented using Reduced Instruction Set Computing (RISC), or an x86 processor implemented using Complex Instruction Set Computing (CISC). The computing device 900 may include one or more CPUs 906 in one or more microprocessors or auxiliary coprocessors, such as a computing coprocessor.

[0184] In addition to or instead of the CPU 906, the GPU 908 may be configured to execute at least some computer-readable instructions to control one or more components of the computing device 900 to execute one or more of the methods and / or processes described herein. One or more of the GPU 908 may be an integrated GPU (for example, one or more of the CPU 906), and / or one or more of the GPU 908 may be a discrete GPU. In embodiments, one or more of the GPU 908 may be a coprocessor of one or more of the CPU 906. The GPU 908 may be used by the computing device 900 to render graphics (for example, 3D graphics) or to perform general-purpose computing. For example, the GPU 908 may be used for general-purpose computing on a GPU (GPGPU). It may be used for a GPU. The GPU908 may include hundreds or thousands of cores capable of processing hundreds or thousands of software threads simultaneously. The GPU908 can generate pixel data for an output image in response to rendering commands (for example, rendering commands from the CPU906 received via the host interface). The GPU908 may include graphics memory, such as display memory, for storing pixel data or any other suitable data, such as GPGPU data. Display memory may be included as part of memory 904. The GPU908 may include two or more GPUs operating in parallel (for example, via a link). The link can connect directly to the GPUs (for example, using NVLINK) or via a switch (for example, using NVSwitch). When coupled together, each GPU908 can generate pixel data or GPGPU data for different parts of an output or for different outputs (for example, the first GPU for the first image and the second GPU for the second image). Each GPU may have its own memory or may share memory with other GPUs.

[0185] In addition to or instead of the CPU 906 and / or GPU 908, the logic unit 920 may be configured to execute at least some computer-readable instructions to control one or more of the computing devices 900 to execute one or more of the methods and / or processes described herein. In embodiments, the CPU 906, GPU 908, and / or logic unit 920 can execute any combination of methods, processes, and / or parts thereof discretely or congruently. One or more of the logic units 920 may be part of and / or integrated with one or more of the CPU 906 and / or GPU 908, and / or one or more of the logic units 920 may be discrete components of the CPU 906 and / or GPU 908 or otherwise external to them. In embodiments, one or more of the logic units 920 may be coprocessors of one or more of the CPU 906 and / or one or more of the GPU 908.

[0186] Examples of the logic unit 920 include one or more processing cores and / or components thereof, such as Tensor Cores (TCs), Tensor Processing Units (TPUs), Pixel Visual Cores (PVCs), Vision Processing Units (VPUs), Graphics Processing Clusters (GPCs), Texture Processing Clusters (TPCs), Streaming Multiprocessors (SMs), Tree Traversal Units (TTUs), Artificial Intelligence Accelerators (AIAs), Deep Learning Accelerators (DLAs), Logical Units (ALUs), Application-Specific Integrated Circuits (ASICs), Floating-Point Units (FPUs), Input / Output (I / O) elements, Peripheral Component Interconnect (PCI) or Peripheral Component Interconnect Express (PCIe) elements, and / or similar.

[0187] The communication interface 910 may include one or more receivers, transmitters, and / or transceivers that enable the computing device 900 to communicate with other computing devices via an electronic communication network, including wired and / or wireless communication. The communication interface 910 may include components and functions to enable communication over any of several different networks, such as wireless networks (e.g., Wi-Fi, Z-Wave, Bluetooth®, Bluetooth® LE, ZigBee, etc.), wired networks (e.g., communicating via Ethernet® or InfiniBand), low-power wide-area networks (e.g., LoRaWAN, SigFox, etc.), and / or the Internet.

[0188] The I / O port 912 can enable the computing device 900 to be logically connected to other devices, including I / O components 914, presentation components 918, and / or other components, some of which can be built into (e.g., integrated into) the computing device 900. Exemplary I / O components 914 include microphones, mice, keyboards, joysticks, gamepads, game controllers, satellite dishes, scanners, printers, wireless devices, and the like. The I / O components 914 can provide a natural user interface (NUI) that processes air gestures, voice, or other physiological inputs generated by the user. In some cases, the input may be transmitted to appropriate network elements for further processing. The NUI may implement any combination of voice recognition, stylus recognition, face recognition, biometric recognition, on-screen and adjacent-screen gesture recognition, air gestures, head and target tracking, and touch recognition related to the display of the computing device 900 (as described in more detail below). The computing device 900 may include depth cameras, such as stereoscope camera systems, infrared camera systems, RGB camera systems, touchscreen technology, and combinations thereof, for gesture detection and recognition. Additionally, the computing device 900 may include accelerometers or gyroscopes to enable motion detection (for example, as part of an inertia measurement unit (IMU)). In some examples, the output of the accelerometer or gyroscope may be used by the computing device 900 to render immersive augmented reality or virtual reality.

[0189] The power supply device 916 may include a hardwired power supply device, a battery power supply device, or a combination thereof. The power supply device 916 can supply power to the computing device 900 to enable the components of the computing device 900 to operate.

[0190] The presentation component 918 may include a display (e.g., a monitor, touch screen, television screen, head-up display device (HUD), other display types, or a combination thereof), a speaker, and / or other presentation components. The presentation component 918 can receive data from other components (e.g., GPU 908, CPU 906, etc.) and output data (e.g., as images, videos, sounds, etc.).

[0191] Exemplary data center Figure 10 shows an exemplary data center 1000 that may be used in at least one embodiment of the present disclosure. The data center 1000 may include a data center infrastructure layer 1010, a framework layer 1020, a software layer 1030, and / or an application layer 1040.

[0192] As shown in Figure 10, the data center infrastructure layer 1010 may include a resource orchestrator 1012, grouped computing resources 1014, and node computing resources ("node CRs") 1016(1) to 1016(N), where "N" represents any integer or natural number. In at least one embodiment, the node CRs 1016(1) to 1016(N) may include, but are not limited to, any number of central processing units ("CPUs") or other processors (including accelerators, field-programmable gate arrays (FPGAs), graphics processors or graphics processing units (GPUs), etc.), memory devices (e.g., dynamic read-only memory), storage devices (e.g., solid-state or disk drives), network input / output ("NW I / O") devices, network switches, virtual machines ("VMs"), power modules, and / or cooling modules. In some embodiments, one or more nodes CR1016(1) to 1016(N) may correspond to a server having one or more of the aforementioned computing resources. In addition, in some embodiments, nodes CR1016(1) to 10161(N) may include one or more virtual components, such as vGPUs, vCPUs, and / or similar, and / or one or more nodes CR1016(1) to 1016(N) may correspond to a virtual machine (VM).

[0193] In at least one embodiment, the grouped computing resources 1014 may include a separate group of nodes CR1016 housed in one or more racks (not shown), or a number of racks housed in data centers in various geographical locations (also not shown). The separate group of nodes CR1016 within the grouped computing resources 1014 may include grouped compute, network, memory, or storage resources that can be configured or allocated to support one or more workloads. In at least one embodiment, several nodes CR1016, including CPUs, GPUs, and / or other processors, may be grouped in one or more racks to provide computing resources to support one or more workloads. The one or more racks may also include any number of power modules, cooling modules, and / or network switches in any combination.

[0194] The resource orchestrator 1022 can configure or otherwise control one or more nodes CR1016(1) to 1016(N) and / or grouped computing resources 1014. In at least one embodiment, the resource orchestrator 1022 may include a software design infrastructure ("SDI") management entity for the data center 1000. The resource orchestrator 1022 may include hardware, software, or any combination thereof.

[0195] In at least one embodiment, as shown in Figure 10, the framework layer 1020 may include a job scheduler 1032, a configuration manager 1034, a resource manager 1036, and / or a distributed file system 1038. The framework layer 1020 may include a framework to support the software 1048 of the software layer 1030 and / or one or more applications 1042 of the application layer 1040. The software 1048 or application 1042 may each include web-based service software or applications, such as those provided by Amazon Web Services, Google Cloud, and Microsoft Azure. The framework layer 1020 may also be, but is not limited to, a type of free and open-source software web application framework, such as Apache Spark® ("Spark"), which may use the distributed file system 1038 for large-scale data processing (e.g., "big data"). In at least one embodiment, the job scheduler 1032 may include a Spark driver to facilitate scheduling of workloads supported by various layers of the data center 1000. The configuration manager 1034 may have the ability to configure different layers, for example, a software layer 1030 and a framework layer 1020 including Spark and a distributed file system 1038 to support large-scale data processing. The resource manager 1036 may have the ability to manage clustered or grouped computing resources mapped or allocated for support of the distributed file system 1038 and the job scheduler 1032. In at least one embodiment, the clustered or grouped computing resources may include computing resources 1014 grouped in the data center infrastructure layer 1010. The resource manager 1036 can coordinate with the resource orchestrator 1012 to manage these mapped or allocated computing resources.

[0196] In at least one embodiment, the software 1048 included in the software layer 1030 may include software used by at least a portion of nodes CR1016(1) to 1016(N), grouped computing resources 1014, and / or the distributed file system 1038 of the framework layer 1020. One or more types of software may include, but are not limited to, internet web page search software, email virus scanning software, database software, and streaming video content software.

[0197] In at least one embodiment, application 1042 included in application layer 1040 may include one or more types of applications used by at least a portion of nodes CR1016(1) to 1016(N), grouped computing resources 1014, and / or the distributed file system 1038 of framework layer 1020. One or more types of applications may include, but are not limited to, any number of genomics applications, cognitive computing, and machine learning applications, including training or inference software, machine learning framework software (e.g., PyTorch, TensorFlow, Caffe, etc.), and / or other machine learning applications used in conjunction with one or more embodiments.

[0198] In at least one embodiment, any of the configuration manager 1034, resource manager 1036, and resource orchestrator 1012 may implement any number and type of self-rewriting actions based on any amount and type of data obtained in any technically possible manner. Self-rewriting actions may free the data center operator of data center 1000 from making potentially poor configuration decisions and possibly avoiding underutilized and / or underperforming parts of the data center.

[0199] Data Center 1000 may include tools, services, software, or other resources for training one or more machine learning models or for predicting or inferring information using one or more machine learning models according to one or more embodiments described herein. For example, a machine learning model may be trained by calculating weight parameters by a neural network architecture using the software and / or computing resources described herein with respect to Data Center 1000. In at least one embodiment, a trained or deployed machine learning model corresponding to one or more neural networks may be used to infer or predict information using the resources described herein with respect to Data Center 1000 by using weight parameters calculated via one or more training techniques, not limited to those described herein.

[0200] In at least one embodiment, the data center 1000 may use a CPU, application-specific integrated circuit (ASIC), GPU, FPGA, and / or other hardware (or corresponding virtual computing resources) for performing training and / or inference using the aforementioned resources. Furthermore, one or more of the aforementioned software and / or hardware resources may be configured as services that enable users to train or perform inference of information, such as image recognition, speech recognition, or other artificial intelligence services.

[0201] Exemplary network environment A network environment suitable for use in implementing the embodiments of this disclosure may include one or more client devices, servers, network-attached storage (NAS), other backend devices, and / or other device types. Each client device, server, and / or other device type (e.g., each device) may be implemented as one or more instances of the computing device 900 in Figure 9, for example, each device may include similar components, features, and / or functionalities of the computing device 900. In addition, if backend devices (e.g., servers, NAS, etc.) are implemented, they may be included as part of the data center 1000, examples of which are further detailed herein with respect to Figure 10.

[0202] Components of a network environment may communicate with one another via the network, either wired, wirelessly, or both. A network may include multiple networks, or a network of networks. For example, a network may include one or more wide area networks (WANs), one or more local area networks (LANs), one or more public networks, such as the Internet and / or the Public Switched Telephone Network (PSTN), and / or one or more private networks. If a network includes a wireless telecommunications network, its components, such as base stations, towers, or access points (and other components), may provide wireless connectivity.

[0203] Compatible network environments may include one or more peer-to-peer network environments (in which case servers may not be included in the network environment) and one or more client-server network environments (in which case one or more servers may be included in the network environment). In a peer-to-peer network environment, the functionality described herein with respect to the server can be implemented on any number of client devices.

[0204] In at least one embodiment, the network environment may include one or more cloud-based network environments, distributed computing environments, or a combination thereof. The cloud-based network environment may include a framework layer, a job scheduler, a resource manager, and a distributed file system implemented on one or more of the servers, which may include one or more core network servers and / or edge servers. The framework layer may include a framework to support the software in the software layer and / or one or more applications in the application layer. The software or applications may each include web-based service software or applications. In the embodiment, one or more of the client devices may use the web-based service software or applications (for example, by accessing the service software and / or applications via one or more application programming interfaces (APIs)). The framework layer may be, but is not limited to, a type of free and open-source software web application framework that may use a distributed file system for, for example, large-scale data processing (e.g., “big data”).

[0205] A cloud-based network environment may provide cloud computing and / or cloud storage that implements any combination of the computing and / or data storage functions (or one or more of them) described herein. Any of these various functions may be distributed across multiple locations from a central or core server (e.g., one or more data centers that may be distributed across states, territories, countries, or the world). If the connection to the user (e.g., a client device) is relatively close to the edge server, the core server may delegate at least a portion of its functionality to the edge server. The cloud-based network environment may be private (e.g., limited to a single organization), public (e.g., available to multiple organizations), and / or a combination thereof (e.g., a hybrid cloud environment).

[0206] A client device may include at least some of the components, features, and functionalities of the exemplary computing device 900 described herein with respect to Figure 9. As an example, and not limited to, a client device may be implemented as a personal computer (PC), laptop computer, mobile device, smartphone, tablet computer, smartwatch, wearable computer, personal digital assistant (PDA), MP3 player, virtual reality headset, global positioning system (GPS) or device, video player, video camera, surveillance device or system, vehicle, boat, airship, virtual machine, drone, robot, handheld communication device, hospital device, gaming device or system, entertainment system, vehicle computer system, embedded system controller, remote control, instrument, consumer electronic device, workstation, edge device, any combination of these depicted devices, or any other suitable device.

[0207] This disclosure may be described in general terms with computer code or machine-usable instructions, including computer-executable instructions such as program modules, which are executed by computers or other machines, such as personal digital assistants or other handheld devices. Generally, a program module, including routines, programs, objects, components, and data structures, refers to code that performs a specific task or implements a specific abstract data type. This disclosure may be implemented in a variety of configurations, including handheld devices, consumer electronics, general-purpose computers, and more specialized computing devices. This disclosure may also be implemented in a distributed computing environment where tasks are performed by remote processing devices linked over a communication network.

[0208] In this specification, any “and / or” statement relating to two or more elements should be interpreted as meaning only one element or a combination of elements. For example, “element A, element B, and / or element C” may include only element A, only element B, only element C, element A and element B, element A and element C, element B and element C, or elements A, B, and C. In addition, “at least one of element A or element B” may include at least one element A, at least one element B, or at least one element A and at least one element B. Furthermore, “at least one of element A and element B” may include at least one element A, at least one element B, or at least one element A and at least one element B.

[0209] The subject matter of this disclosure is described in a manner that is specific in order to satisfy statutory requirements. However, the description itself is not intended to limit the scope of this disclosure. Rather, the inventors intend that the claimed subject matter may be carried out in other ways, including different steps or combinations of steps similar to those described herein, in conjunction with other current or future technologies. Furthermore, the terms “step” and / or “block” may be used herein to imply different elements of the way in which they are used, but these terms should not be construed as implying any particular order among the various steps disclosed herein unless the order of the individual steps is expressly stated and, when so, expressed.

Claims

1. A method performed by a processor and a controller, A step to detect a power-off instruction corresponding to the machine, A step of performing a diagnosis in a computer system used to perform one or more autonomous control operations of the machine, based at least on the step of detecting the power off instruction, A step of rebooting one or more parts of the computer system based at least on the step of performing the diagnosis, A step of storing the state of the computing environment corresponding to the computer system in computer storage as a state to be saved, based at least on the reboot step, The steps include, at least based on the step of storing the state of the computing environment, putting the computer system into a low-power mode, A step of detecting a power-on instruction corresponding to the machine, Steps to exit the low-power mode of the computer system, at least based on the step of detecting the power-on instruction, A step of restoring the state of the computing environment corresponding to the computer system to the saved state from the computer storage, based at least on the step of exiting the low-power mode. Methods that include...

2. The method according to claim 1, further comprising the step of rebooting one or more parts of the computer system prior to the step of performing the diagnosis, based at least on the step of detecting the power-off instruction of the machine.

3. The state of the aforementioned computing environment is the first state, and the method further Steps to exit the low-power mode prior to the power-on instruction, based at least on the fulfillment of one or more criteria, The steps include: re-executing the diagnosis using the aforementioned computer system; The steps include rebooting the computer system to generate a second state of the computing environment, The steps include storing the second state of the computing environment in the computer storage as a second state to be saved, The steps of putting the computer system back into the low-power mode and The method according to claim 1, wherein the step of restoring the state of the computing environment includes the step of restoring the state of the computing environment corresponding to the computer system to the second state.

4. The method according to claim 1, wherein the computer storage includes volatile memory.

5. The machine includes an energy storage medium for supplying electrical power to the computer system, and the machine, A machine that is at least partially powered by electricity, or A machine that is supplied with at least partially internal combustion power. The method according to claim 1, wherein at least one of the following:

6. The machine includes one or more sensors, The method according to claim 1, characterized in that the low-power mode is such that at least one of the one or more sensors does not receive power.

7. The machine includes a graphics processing unit used to perform the autonomous control operation, The method according to claim 1, wherein the low-power mode is characterized in that the graphics processing unit does not receive power.

8. The method according to claim 1, wherein the controller generates a control signal used to perform the autonomous control operation, and at least one of the steps of performing the diagnosis, rebooting, entering the low-power mode, exiting the low-power mode, or enabling the autonomous control operation is performed at least partially by the controller.

9. The method according to claim 8, wherein the controller includes a circuit that remains powered during the low-power mode in order to bring the computer system out of the low-power mode.

10. The vehicle's power-off command is detected, The computer system used for the autonomous control of the vehicle is instructed to perform a diagnosis. The computer system is rebooted and configured to a computing state, and the computing state is stored in computer storage as a state to be saved. While the saved state is stored in the computer storage, the computer system is put into low-power mode. Based at least on a power-on instruction for the vehicle, the computer system is brought out of the low-power mode, and the autonomous control of the vehicle by the computer system is enabled based at least on the fact that the saved state is restored from the computer storage by bringing it out of the low-power mode. A processor comprising one or more circuits.

11. The aforementioned processor, Control systems for autonomous or semi-autonomous machines Cognitive systems for autonomous or semi-autonomous machines A system for performing simulation operations. A system for performing deep learning operations. Systems implemented using edge devices, Systems implemented using robots, A system that incorporates one or more virtual machines (VMs). A system that is at least partially implemented in a data center, or A system that is at least partially implemented using cloud computing resources. The processor according to claim 10, which is included in at least one of the following.

12. The processor according to claim 10, wherein one or more of the circuits are part of the microcontroller unit of the computer system.

13. The calculation state is the first calculation state, and the one or more circuits further Prior to the power-on instruction, the computer system is brought out of the low-power mode based on at least one or more criteria being met. The computer system is instructed to rerun the diagnosis, Reboot to generate a second calculation state, and store the second calculation state in the computer storage as the saved state. The computer system is then put back into the low-power mode. The processor according to claim 10, configured such that the saved state to be restored in order to enable the autonomous control is the second computation state.

14. The processor according to claim 10, wherein the diagnosis includes checking for potential defects in one or more components of the computer system.

15. The vehicle is equipped with one or more sensors, The processor according to claim 10, wherein the low-power mode is characterized in that at least one of the one or more sensors does not receive power.

16. One or more processing units of a vehicle, One or more memory units that store instructions and A system comprising, where the instruction is executed by one or more processing units, the one or more processing units, A step of performing an in-system test of a computer system used for autonomous control of the vehicle, based at least on a first instruction to turn off the vehicle's key, The steps include configuring the computer system into a computational state that enables the autonomous control to be performed, at least based on the completion of the in-system inspection, The steps include operating the computer system in low-power mode while the calculation state is stored in computer storage as a saved state, Steps include, at least based on a second instruction to turn the vehicle's key on, exiting the computer system from the low-power mode, The steps of enabling the autonomous control of the vehicle by the computer system, at least based on restoring the saved state from the computer storage, and A system that performs actions including those mentioned above.

17. The aforementioned system Control systems for autonomous or semi-autonomous machines Cognitive systems for autonomous or semi-autonomous machines A system for performing simulation operations. A system for performing deep learning operations. Systems implemented using edge devices, Systems implemented using robots, A system that incorporates one or more virtual machines (VMs). A system that is at least partially implemented in a data center, or A system that is at least partially implemented using cloud computing resources. The system according to claim 16, which is included in at least one of the following.

18. The system according to claim 16, wherein the operation further includes a step of rebooting one or more parts of the computer system prior to the step of performing the in-system inspection based at least on the first instruction to turn the vehicle's key off.

19. The calculation state is the first calculation state, and the operation further Steps include: exiting the computer system from the low-power mode prior to the second instruction, based at least on the fulfillment of one or more criteria; The steps include: re-executing the in-system check of the computer system; A step of configuring the computer system to generate a second computation state, wherein the saved state restored from the computer storage is the second computation state. The system according to claim 16, including the system described in claim 16.

20. One or more sensors It further includes, The system according to claim 16, wherein the low-power mode is characterized in that at least one of the one or more sensors does not receive power.

Images