Detecting faults in sensor device settings

Abstract

The invention discloses detecting faults in sensor device settings. In various examples, a technique for validating sensor settings is disclosed that includes receiving sensor output data and at least one sensor output setting associated with the sensor output data from a sensor. The technique further includes determining whether the queue of sensor input settings to be processed includes matching sensor input settings that match the sensor output settings. The technique further includes, in response to determining that the queue of sensor input settings to be processed includes the matching sensor input settings, identifying, in the queue of sensor input settings to be processed, sensor input settings to be processed in the queue that precede the matching sensor input settings. The technique further includes determining whether a sensor setting failure has occurred based on the pending request count associated with the pending input setting. The technique further includes generating an error indication based on the count of requests to be processed.

Description

BACKGROUND OF THE INVENTION

[0001] Autonomous or semi-autonomous vehicles are typically equipped with cameras, depth sensors, and / or other sensors for detecting and understanding the environment around the vehicle. Data collected by the sensors is used by downstream components to make important decisions related to stopping the vehicle, starting the vehicle, controlling the vehicle speed, determining the path for the vehicle to take, disengaging the vehicle's autonomous driving mode, and / or otherwise operating the vehicle. The accuracy of the sensors under specific conditions may depend on the correct configuration of the sensors. The sensors can accept one or more settings, which can be specific values that configure the operation of the sensors. The term "sensor settings" herein can refer to a set of one or more setting values that can be sent to a set of sensors individually and / or as a group (e.g., in a single configuration request). The setting values can be values of configuration parameters of the sensors. For example, a camera has an exposure parameter that controls the amount of time the camera sensor is exposed to light, and a gain parameter that controls the amount of amplification applied to the pixels captured by the camera sensor. The appropriate values of the exposure parameter and the gain parameter depend on various factors, including the characteristics of the scene captured by the camera. The sensor settings of the camera (also referred to herein as camera settings) can be specific values of the exposure parameter and the gain parameter, e.g., such as 25 milliseconds and 10x.

[0002] The sensor settings can be sent from other system components to the sensors. For example, a sensor controller can determine the appropriate sensor settings and send the sensor settings to the sensors. For example, a sensor can have a configuration interface that receives specific settings, and the sensor controller can send the settings to the configuration interface to cause the sensor to use the sensor settings. After receiving a specified set of settings, the sensor configures the corresponding parameters to have the specified set of sensor settings. The operation of sending the sensor settings to the sensors and / or configuring the sensors to use the sensor settings is referred to herein as applying the sensor settings to the sensors. After applying the sensor settings, the sensors operate according to the settings. Then, the sensors receive information about the environment and generate detection information according to the sensor settings.

[0003] As an example, a camera controller determines the appropriate values of the camera exposure and gain settings based on the scene statistics of the current image most recently captured by the camera and the previous camera settings sent to the camera before capturing the current image. The camera controller can perform continuous auto exposure (CAE), which involves determining the camera exposure and gain settings to achieve optimized brightness in the image captured by the camera. The camera controller applies the determined camera settings to the camera by sending the settings to the camera. If the camera settings are successfully applied to the camera, the camera uses the specified settings when capturing an image.

[0004] The failure of a sensor settings application may be due to unexpected circumstances or behaviors, such as hardware or software not operating correctly (e.g., according to specifications). The failure of the settings application may result in incorrect results, such as inaccurate images captured by a camera. For example, a failure of system components involved in sending and / or configuring the sensor to use the settings may cause the sensor settings application to fail. System components that may cause the sensor settings application to fail if not operating correctly include a camera controller, a communication bus, or a sensor. Sensor failures (e.g., the sensor not accepting the settings) may occur without generating an error notification or other specific indication that a failure has occurred. The failure of the sensor settings application may also include applying sensor settings different from the settings assigned to the sensor, in which case the sensor may produce an incorrect output.

[0005] Applications such as automotive systems have a need for accurate and timely sensor output. Therefore, accurately and efficiently verifying the applied sensor settings is crucial for reliable autonomous or semi-autonomous operation and navigation. For example, failure to correctly apply an exposure time setting to a camera in an automotive system may result in an underexposed or overexposed captured image, which may not clearly depict the objects present in the environment. Such inaccurate or otherwise incorrect sensor data may cause downstream components to misidentify or fail to identify objects in the vehicle's surrounding environment and lead to incorrect or inappropriate driving decisions.

[0006] One method that has been implemented to verify that sensor settings have been correctly applied to a sensor involves storing a copy of the "input" settings sent to the sensor. When a sensor output is received, the "output" settings provided as part of the sensor output are compared to the stored copy of the input settings. If the input settings do not match the output settings and the error tolerance is within a threshold level, it indicates that the input settings were not applied to the sensor, or if the input settings were applied, they were not applied correctly. The threshold level of the error tolerance is based on the expected amount of quantization error resulting from converting the input settings to quantized values in the sensor. Due to the limited number of bits available to represent such settings (or the number of bits available to represent values that affect such settings), the sensor may round the input settings (e.g., exposure and / or gain) to different "quantized" values. The sensor uses the quantized values for the settings, and thus the sensor provides the quantized values as the output settings, which represent the settings used to generate the sensor output. Since the output settings may differ from the input settings due to quantization error, the quantization error is used as the error tolerance (e.g., the acceptable amount of difference) between the output settings and the stored copy of the input settings when verifying whether the input settings were correctly applied to the camera.

[0007] However, when using many different sets of settings, this existing method is not effective because maintaining copies of each set of settings uses additional memory storage and computational time, and thus cannot scale to a larger number of settings. For example, a continuous auto-exposure algorithm uses a larger number of settings, which dynamically generates different sensor settings based on changes in the input image. Therefore, as the number of different settings increases beyond a fixed number, the effectiveness of the existing method decreases.

[0008] In addition, the existing method uses a fixed error tolerance (e.g., a fixed percentage threshold) to determine whether the output settings match the input settings. In the existing method, the stored copy of the input settings is associated with or modified based on a specific sensor quantization error corresponding to the input settings. However, when using many different sets of settings, the error tolerance (e.g., percentage) for each different set of sensor settings is different because different settings may have different amounts of quantization error introduced by the sensor. For example, continuously generated sensor settings (such as those used in continuous auto-exposure) are variably quantized over a range of sensor gains and exposures. Adjusting the error tolerance for each different set of sensor settings to account for different sensor quantization errors is impractical. For example, a fixed set of settings can contain five settings, and a fixed error tolerance (e.g., 5%) can be associated with these five settings. However, for continuously generated sensor settings, the percentage error tolerance varies with the sensor settings. For example, lower gain and exposure settings can have a lower error tolerance, but higher gain and exposure settings can have a larger error tolerance. Therefore, a fixed tolerance is ineffective for the range of lower to higher settings. Thus, when the settings change over time, as occurs when using continuous auto-exposure to adjust sensor settings, the fixed tolerance method is ineffective. Therefore, the existing method is not sufficient to verify whether many sensor settings (such as those used for continuous auto-exposure) have been correctly applied within an acceptable amount of time.

[0009] Accordingly, more effective techniques are needed to verify whether sensor settings have been correctly applied in, for example, autonomous or semi-autonomous systems. SUMMARY OF THE INVENTION

[0010] Embodiments of the present disclosure relate to a technique for verifying sensor settings. The technique includes: receiving sensor output data and at least one sensor output setting associated with the sensor output data from the sensor. The technique further includes: determining whether a queue of sensor input settings to be processed includes a matching sensor input setting that matches the sensor output setting. The technique further includes, in response to determining that the queue of sensor input settings to be processed includes a matching sensor input setting, identifying, in the queue of sensor input settings to be processed, the sensor input settings to be processed that are before the matching sensor input setting in the queue. The technique further includes: determining whether a sensor setting failure has occurred based on a count of pending requests associated with the sensor input settings to be processed. The technique further includes: generating an error indication based on the count of pending requests.

[0011] One technical advantage of the disclosed technique over prior solutions is the ability to detect failures in applying sensor settings when applying multiple different settings, such as what occurs when using continuous autoexposure. Another technical advantage of the disclosed technique is the ability to identify and report failure modes. Failure modes that can be detected include data corruption, loss of settings sent by the sensor but not used, or loss of output frames not received from the sensor. The identified failure modes can be used to find the cause of the failure and take corrective measures. These technical advantages represent one or more technical improvements over existing methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The present system and method for securely detecting data changes in autonomous or semi-autonomous systems and applications are described in detail below with reference to the drawings, wherein:

[0013] Figure 1 A computing device configured to implement one or more aspects of various embodiments is shown;

[0014] Figure 2 is a more detailed illustration of a Figure 1 sensor fault detector according to various embodiments;

[0015] Figure 3A A flowchart of a method for processing a setting frame containing initial data from a sensor in an initial stage according to various embodiments is shown;

[0016] Figure 3B A flowchart of a method for processing a setting frame containing sensor detection data in a runtime stage according to various embodiments is shown;

[0017] Figure 4 A flowchart of a method for queuing sensor settings into a settings history queue according to various embodiments is shown;

[0018] Figure 5A and Figure 5BShows an example execution of the initial stage of a sensor fault detector during the fault - free operation of a sensor system according to some embodiments of the present disclosure;

[0019] Figure 5C Shows an example execution of the initial stage of a sensor fault detector when a sensor setting damage occurs according to some embodiments of the present disclosure;

[0020] Figure 6A Shows an example execution of the runtime stage of a sensor fault detector when a sensor setting damage occurs according to some embodiments of the present disclosure;

[0021] Figure 6B Shows an example execution of the runtime stage of a sensor fault detector when a sensor setting delay occurs according to some embodiments of the present disclosure;

[0022] Figure 6C Shows an example execution of the runtime stage of a sensor fault detector when a sensor output frame loss occurs according to some embodiments of the present disclosure;

[0023] Figure 6D Shows an example execution of the runtime stage of a sensor fault detector when a sensor setting loss occurs according to some embodiments of the present disclosure;

[0024] Figure 7A Is an illustration of an example autonomous vehicle according to some embodiments of the present disclosure;

[0025] Figure 7B Is according to some embodiments of the present disclosure Figure 7A An example of the camera position and field of view of an example autonomous vehicle;

[0026] Figure 7C Is according to some embodiments of the present disclosure Figure 7A A block diagram of an example system architecture of an example autonomous vehicle;

[0027] Figure 7D Is a system schematic diagram of the communication between a cloud - based server and Figure 7A An example autonomous vehicle according to some embodiments of the present disclosure;

[0028] Figure 8 Is a block diagram of an example computing device suitable for implementing some embodiments of the present disclosure; and

[0029] Figure 9 Is a block diagram of an example data center suitable for implementing some embodiments of the present disclosure. Detailed Description

[0030] Systems and methods for detecting sensor setup faults in autonomous or semi-autonomous vehicles or machines are disclosed. Although the present disclosure may be described with respect to an example autonomous or semi-autonomous vehicle or machine 700 (alternatively referred to herein as "vehicle 700" or "self machine 700"), which example is described with respect to Figures 7A to 7D it is not limiting. For example, the systems and methods described herein may be used without limitation by non-autonomous vehicles or machines, semi-autonomous vehicles or machines (e.g., in one or more adaptive driver assistance systems (ADAS)), autonomous vehicles or machines, manned and unmanned robots or robotic platforms, warehouse vehicles, off-road vehicles, vehicles coupled to one or more trailers, airships, vessels, shuttles, emergency response vehicles, motorcycles, electric or motorized bicycles, airplanes, construction vehicles, underwater vehicles, drones, and / or other vehicle types. Additionally, although the present disclosure may be described with respect to sensor performance monitoring in autonomous and / or semi-autonomous vehicles, it is not intended to be limiting, and the systems and methods described herein may be used in augmented reality, virtual reality, mixed reality, robotics, security and surveillance, automated or semi-automated machine applications, and / or any other technical field where sensor monitoring may be used.

[0031] Figure 1 A computing device 100 configured to implement one or more aspects of various embodiments is shown. In at least one embodiment, the computing device 100 includes a desktop computer, a laptop computer, a smartphone, a personal digital assistant (PDA), a tablet, a server, one or more virtual machines, an embedded system, an embedded hardware module including a system-on-chip, a system-on-chip, a computing system of an autonomous, semi-autonomous, or non-autonomous machine, and / or any other type of computing device configured to receive input, process data, and optionally display images, and suitable for practicing one or more embodiments. The computing device 100 includes a memory 116, one or more processors 102, an interconnect 112, a storage 114, an input / output (I / O) device interface 104, and a network interface 106. The computing device 100 also includes one or more I / O devices 108 and / or communicates with one or more I / O devices 108. The I / O devices 108 communicate with the interconnect 112 via the I / O device interface 104. The memory 116 may be volatile random access memory or other suitable type of memory.

[0032] In a particular embodiment, the sensor fault detector 122 is stored in the memory 116. The sensor fault detector 122 detects and reports faults when applying sensor settings to a sensor (e.g., a camera, a Light Detection and Ranging (LIDAR) sensor, a Radio Detection and Ranging (RADAR) sensor, or other types of sensors configurable by applying settings). To configure a sensor with input settings, a system (e.g., a control system or a perception system for an autonomous or semi-autonomous machine) generates input settings based on environmental characteristics (e.g., light level) and requests that the input settings be applied to the sensor so that the sensor operates according to the input settings. For example, the control system can send a sensor configuration request including the input settings to the sensor. If no fault occurs, the input settings are applied to the sensor, and the sensor operates according to one or more setting values included in the input settings. For example, applying the input settings may involve configuring the sensor to use the input settings. The sensor then generates one or more sensor output frames. Each sensor output frame includes captured data that is generated by the sensor based on stimuli received by the sensor. For example, if the sensor is a camera, the captured data can include an image of the environment captured by the sensor. In other examples, if the sensor is a RADAR or LIDAR sensor, the captured data can include representations of detections, such as a point cloud or distances and directions to detected objects. Each sensor output frame also includes output settings that specify the settings used by the sensor to capture the captured data. The output settings can be different from the input settings because the sensor can adjust the input settings before capturing the data. For example, due to limited bits available to represent the setting values, the sensor can round the setting values in the input settings to different "quantized" values. The sensor uses the quantized values for the setting values, so the sensor provides the quantized values in the output settings, which represent the settings used when generating the sensor output.

[0033] The sensor fault detector 122 monitors the input settings sent to the sensor by a sensor controller or other system components. For example, the sensor fault detector 122 can receive each input setting sent to the sensor by a sensor controller or other system component that configures the sensor. The sensor fault detector 122 also monitors the output settings, which are received together with each output data frame generated by the sensor and indicate the settings used by the sensor to generate the output data frame. For example, the sensor fault detector 122 can receive each output data frame generated by the sensor and extract the output settings from each output data frame.

[0034] The sensor fault detector 122 stores each received input setting in the sensor settings queue 124. The sensor settings queue 124 is stored in the memory 116 and can be a first-in-first-out data structure containing elements. For example, each element can be a setting or a data structure including an input setting and a count. Each element can be implemented as a reference (e.g., a pointer) to an input setting or a data structure containing an input setting and a count. The input setting that has been in the queue the longest is called the "head" of the queue. The input setting that has been in the queue the shortest is called the "tail" of the queue. The sensor fault detector 122 uses the sensor settings queue 124 to detect faults that occur between sending an input setting to the sensor and receiving an output setting from the sensor. The sensor fault detector 122 identifies a specific fault mode (e.g., a fault type) based on the content of the sensor settings queue 124. Fault modes that can be identified include the sensor setting being lost or corrupted in the communication path between the transmitter (e.g., the sensor controller) and the sensor, the sensor setting being corrupted in the sensor, the frame captured by the sensor being lost during transmission to the sensor fault detector 122, the sensor data in the captured frame being corrupted during transmission to the receiver (e.g., the sensor controller or the sensor fault detector 122), and / or a delay in the sensor setting taking effect in the sensor.

[0035] After receiving an input setting, the sensor fault detector 122 stores the input setting in the settings history queue 124, for example, by adding the input setting to the end of the queue and setting the count value associated with the input setting to a value indicating the number of consecutive times the same setting has been sent to the sensor. The sensor fault detector 122 updates the count when the input setting is sent and the output setting is received so that the count represents the difference between the number of consecutive times an input setting has been sent to the sensor and the number of times an output setting that matches the input setting has been received. For example, when the setting is first sent to the sensor, the count is set to 1 (or some other value indicating that the setting has been sent once). Each time the same input setting is sent to the sensor without sending a different intermediate input setting, the count is incremented by 1 (or some other value indicating an increment for sending the same setting once).

[0036] After receiving an output frame that includes output settings, the sensor fault detector 122 searches the settings history queue for the first (e.g., oldest) input setting that matches the received output setting within a threshold tolerance. The threshold tolerance accounts for quantization errors introduced by the sensor and can be based on the value of the settings being compared. A set of setting value ranges can be used to determine the threshold tolerance, with each setting value range associated with a different threshold tolerance. The specific threshold tolerance used to determine whether a particular setting value (e.g., gain) in the input settings matches the corresponding setting value in the output settings can be identified by looking up the range that includes the particular setting value. The threshold tolerance associated with the identified range is the specific threshold tolerance for the particular setting value. As an example, five ranges can be used for the gain setting, with values that can range from 1 to 10. Tolerance T1 is associated with the range between 1 and 2, represented by the interval [1,2). Tolerance T2 is associated with the range from 2 to 4, represented by the interval [2,4). Tolerance T3 is associated with the range from 4 to 6, represented by the interval [4,6). Tolerance T4 is associated with the range from 6 to 8, represented by the interval [6,8). Tolerance T5 is associated with the range from 8 to 10, represented by the interval [8,10). In this example, an input setting with a gain of 3.5 is within the range associated with tolerance T2. The tolerance can be expressed as a tolerance value, in which case the input setting value matches the output setting value if the absolute value of the difference between the input setting value and the output setting value is less than the tolerance value. In another example, the tolerance can be expressed as a percentage. If tolerance T2 is 10%, then an input setting with a gain of 3.5 matches a received output setting with a gain of 3.4 because 3.4 is within 10% of 3.5. However, an input setting with a gain of 3.5 does not match a received output setting with a gain of 3.1 because 3.1 is not within 10% of 3.5. Although tolerance determination is described using specific determination techniques in the examples herein, any suitable determination technique can be used to determine the tolerance based on the input settings and / or output settings.

[0037] If the sensor fault detector 122 does not find a matching input setting in the sensor settings queue 124, a settings corruption fault has occurred. If a matching input setting is found, the sensor fault detector 122 decrements the counter for the matching input setting. If the matching input setting is at the head of the queue, no fault involving the input setting has occurred. Otherwise, the matching input setting is not at the head of the queue, and each non-matching setting before the matching setting is removed from the queue.

[0038] Compare the count associated with each non - matching setting with 0 (or other value corresponding to 0) to determine if a failure has occurred. If the count associated with the non - matching setting is 0, no failure has occurred. If the count of the non - matching setting is less than 0, a failure has occurred, and the specific type of failure is identified based on the counter of the next setting in the queue. If the counter of the next setting is zero, a setting update delay has occurred. If the counter of the next setting is zero, a sensor setting loss has occurred. Additionally, if the count of the non - matching setting is greater than zero, a failure has occurred, and the specific type of failure is identified based on the counter of the previous setting in the queue. If the counter of the previous setting is zero, a frame loss has occurred. If the counter of the previous setting is less than zero, a sensor setting loss has occurred.

[0039] An error message describing the failure mode and the associated input settings and / or sensor output frames can then be reported to notify the application or other system components of a sensor data validation error. For example, the error message can indicate that incorrect sensor data may have been received, or that expected sensor data may not have been received. In various embodiments, an initial verification phase is used to consume and verify an initial sequence of settings generated by the sensor during sensor initialization before performing the above operations.

[0040] It should be noted that the computing devices described herein are illustrative, and any other technically feasible configuration falls within the scope of the present disclosure. For example, multiple instances of the sensor fault detector 122 can be executed on a set of nodes in a distributed and / or cloud computing system to implement the functionality of the computing device 100. Alternatively, the computing device 100 can be implemented similar to the computing device of the exemplary autonomous or semi - autonomous machine 700 described at least with respect to Figures 7A to 7D the described example.

[0041] In at least one embodiment, computing device 100 includes, but is not limited to, an interconnect (bus) 112 connecting one or more processors 102, an input / output (I / O) device interface 104 coupled to one or more input / output (I / O) devices 108, a memory 116, a storage device 114, and / or a network interface 106. The processor 102 may include any suitable processor implemented as a central processing unit (CPU), a graphics processing unit (GPU), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), an artificial intelligence (AI) accelerator, a deep learning accelerator (DLA), a parallel processing unit (PPU), a data processing unit (DPU), a vector or vision processing unit (VPU), a programmable vision accelerator (PVA), any other type of processing unit, or a combination of different processing units, such as a CPU configured to operate in cooperation with a GPU. Generally, the processor 102 may include any technically feasible hardware unit capable of processing data and / or executing software applications. Additionally, in the context of the present disclosure, the computing elements shown in the computing device 100 may correspond to a physical computing system (e.g., a system in a data center or a machine) and / or may correspond to a virtual computing instance executing in a computing cloud.

[0042] In at least one embodiment, the I / O device 108 includes devices capable of receiving input, such as a keyboard, a mouse, a touchpad, a VR / MR / AR headset, a gesture recognition system, a steering wheel, mechanical, digital, or touch-sensitive buttons or input components, and / or a microphone, and devices capable of providing output, such as a display device, a haptic device, and / or a speaker. Additionally, the I / O device 108 may include devices capable of receiving input and providing output, such as a touchscreen, a universal serial bus (USB) port, etc. The I / O device 108 may be configured to receive various types of input from an end user (e.g., a designer) of the computing device 100 and also provide various types of output to the end user of the computing device 100, such as a displayed digital image or digital video or text. In some embodiments, one or more I / O devices 108 are configured to couple the computing device 100 to a network 110.

[0043] In at least one embodiment, the network 110 is any technically feasible communication network that allows the exchange of data between the computing device 100 and an internal, local, remote, or external entity or device (e.g., a web server or another networked computing device). For example, the network 110 may include a wide area network (WAN), a local area network (LAN), a wireless (e.g., WiFi) network, and / or the Internet, etc.

[0044] In at least one embodiment, the storage device 114 includes non-volatile memory for applications and data and may include fixed or removable disk drives, flash devices, and CD-ROM, DVD-ROM, Blu-ray, HD-DVD, or other magnetic, optical, or solid-state storage devices. The processing engine 122 and / or the analysis engine 124 may be stored in the storage device 114 and loaded into the memory 116 when executed.

[0045] In one embodiment, the memory 116 includes random access memory (RAM) modules, flash cells, and / or any other type of memory cells or combinations thereof. The processor 102, the I / O device interface 104, and the network interface 106 are configurable to read data from and write data to the memory 116. The memory 116 may include various software programs or more generally software code executable by the processor 102 and application data associated with the software programs including the processing engine 122 and / or the analysis engine 124.

[0046] Figure 2 is a more detailed illustration of Figure 1 the sensor fault detector 122 according to various embodiments. As shown, the sensor fault detector 122 is included in the sensor interface 202. The sensor interface 202 is used by the sensor settings generator 204 to send the input settings 260 to the sensor 212 and by the sensor frame consumer 236 to receive the sensor output frame 216. For ease of explanation, the term "input setting" (singular) is used herein to refer to a single one or more setting values that may be sent to the sensor 212 as a group, e.g., in a single configuration request. The term "input settings" (plural) refers herein to two or more groups of setting values. For example, two input settings 260 may be sent to the sensor 212 at different times. The sensor settings generator 204 and the sensor frame consumer 236 may be components of a sensor controller, an application, or other system components (not shown).

[0047] The sensor interface 202 includes an input settings processor 206 that processes each input setting 260 by adding the input setting 260 to the sensor settings queue 124. The input settings processor 206 includes an input settings receiver 208 that receives the input setting 260 from the sensor settings generator 204. The input settings processor 206 also includes a settings enqueuer 210 that enqueues the received input setting 260 into the sensor settings queue 124.

[0048] The input settings processor 206 also sends the input settings 260 to the sensor 212 to request configuring the sensor 212 according to the input settings 260. The sensor 212 receives the input settings 260 and applies the input settings 260 to the configuration of the sensor 212. The result of applying the input settings 260 to the sensor configuration is shown as the applied settings 214. The applied settings 214 can be different from the input settings 260, for example, if the sensor rounds the set value to a larger or smaller value compatible with the sensor, or if a failure (e.g., data corruption) occurs during the configuration of the sensor.

[0049] Upon receiving the input settings 260, the input settings processor 206 stores the input settings 260 in the settings history queue 124, for example, by adding the input settings to the tail of the queue and setting the count value associated with the input settings to a value indicating the consecutive number of times the same settings have been sent to the sensor. For example, the settings enqueuer 210 can add the input settings 260 as the tail element 226N to the sensor settings queue 124. For example, the tail element 226N can be the input settings 260 or a data structure containing the input settings 260. The sensor settings queue 124 also includes a head element 226A and the next element 226B located between the head element 226A and the tail element 226N. Each element 226 in the sensor settings queue 124 can be a different sensor setting.

[0050] Since the count value associated with each input setting in the sensor settings queue 124 indicates the number of consecutive times the same setting has been sent to the sensor 212, separate input settings with the same setting value are not stored consecutively in the sensor settings queue 124. When an input setting 260 is sent to the sensor 212 and an output setting 222 is received from the sensor 212, the sensor fault detector 122 updates the count associated with each input setting 260 such that the count associated with each input setting 260 in the sensor settings queue 124 represents the difference between the number of consecutive times the input setting 260 has been sent to the sensor 212 and the number of times an output setting 222 matching the input setting 260 has been received from the sensor 212. For example, when the input setting 260 is first sent to the sensor 212, the count associated with the input setting 260 is set to 1 (or some other value indicating that the setting has been sent once). Each time the same input setting 260 is sent to the sensor without an intervening different input setting 260 being sent, the count is incremented by 1 (or some other value indicating an increment for sending the same setting once). Alternatively, in other embodiments, each input setting 260 may be stored in the sensor settings queue 124 even if the input setting 260 has the same setting value as another adjacent input setting 260 in the queue, in which case the count value does not need to be maintained because the number of consecutive occurrences of the input setting among the queue input settings 260 can represent the count value.

[0051] The sensor 212 generates a series of sensor output frames 216 in response to an external stimulus (such as light or other electromagnetic waves). Each sensor output frame 216 may include captured data 220, which is generated by the sensor 212 based on the external stimulus according to the applied settings 214. The captured data 220 may be, for example, a camera image, a point cloud, or some other suitable representation of the data detected by the sensor 212. The sensor output frame 216 also includes output settings 222, which include one or more setting values used by the sensor 212 to capture the captured data 220. The output settings 222 may be different from the applied settings 214. For example, due to the limited number of bits available in the sensor 212 to represent setting values, the sensor 212 may round one or more of the setting values in the applied settings 214 to corresponding quantized values. The sensor uses the quantized values for the settings when capturing the captured data 220, and thus the sensor provides the quantized values as the output settings 222, which represent the settings used to generate the captured data 220.

[0052] The capture frame processor 228 detects and reports faults that occur when sending or applying the input settings 260 to the sensor 212 or when receiving the output settings 222 from the sensor 212. The capture frame processor 228 includes a sensor frame receiver 230 that receives each sensor output frame 216 from the sensor 212 and provides the sensor output frame 216 to an output settings analyzer 234. After receiving the sensor output frame 216, the output settings analyzer 234 searches the sensor settings queue 124 for the first (e.g., oldest) queue element 226 that contains an input setting that matches the output setting 222 included in or associated with the sensor output frame 216.

[0053] In some embodiments, an input setting in the sensor settings queue 124 can match the output setting 222 if each setting value in the input setting is equal to the corresponding setting value of the same parameter in the output setting 222. For example, if the input setting includes an exposure parameter value of 25 milliseconds and a gain parameter value of 10, then the input setting matches the output setting 222 if the output setting also includes an exposure parameter value of 25 milliseconds and a gain parameter value of 10. In other embodiments, an input setting matches the output setting 222 if each setting value in the input setting is within a threshold tolerance of the corresponding setting value of the same parameter in the output setting 222. For example, an input setting that includes an exposure parameter value of 25 milliseconds and a gain parameter value of 10 matches an output setting 222 that includes an exposure parameter value of 26 milliseconds and a gain parameter value of 11 within a 10% tolerance threshold. The threshold tolerance accounts for quantization errors introduced by the sensor and can be based on the values of the settings being compared.

[0054] If the output settings analyzer 234 does not find a matching input setting, then 234 determines that a settings corruption fault has occurred and sends a fault mode indicator 242 to an error generator 244. The error generator 244 sends error data 246 to an error handler 248 that identifies the fault mode 242 and further identifies the sensor output frame 216 for which no match was found. The error handler 248 can provide the error data 246 to system components such as a sensor controller or an application. The system components can then process the error based on the error data 246. For example, the error handler 248 determines that capture data 220 for which a settings corruption fault has occurred will be discarded without further processing. The system components then discard the capture data 220 associated with the sensor output frame 216.

[0055] If the output setting analyzer 234 finds a matching input setting in the sensor setting queue 124 that matches the output setting 222, the sensor fault detector 122 decrements the counter associated with the matching input setting by 1. If the matching input setting is at the head of the queue, no fault involving the input setting has occurred, and the output setting analyzer 234 continues to search the sensor setting queue 124 for another input setting that matches the output setting 222. Otherwise, the matching input setting is not at the head of the queue, and the setting dequeuer 240 removes (e.g., dequeues) each non-matching input setting before the matching setting from the sensor setting queue 124.

[0056] The output setting analyzer 234 compares the count associated with each dequeued non-matching setting with 0 (or some other value corresponding to 0) to determine if a fault has occurred. If the count associated with the dequeued non-matching setting is 0, no fault has occurred, and the output setting analyzer 234 continues to compare the next dequeued non-matching setting with 0. If the count associated with each dequeued non-matching setting is 0, the output setting analyzer 234 determines that no error has occurred involving the dequeued non-matching settings.

[0057] If the output setting analyzer 234 determines that the count associated with the dequeued non-matching setting is less than 0, a fault has occurred, and the output setting analyzer 234 identifies a specific type of fault based on the counter of the next setting in the queue. For example, the next setting can be the setting adjacent to and after the dequeued non-matching setting in the queue. If the counter of the next setting is zero, a setting update delay has occurred. If the counter of the next setting is zero, a sensor setting has been lost.

[0058] In addition, if the output setting analyzer 234 determines that the count associated with the dequeued non-matching setting is greater than 0, a fault has occurred, and the output setting analyzer 234 identifies a specific type of fault based on the counter of the previous setting in the queue. For example, the previous setting can be the setting adjacent to and before the dequeued non-matching setting in the queue. If the counter of the previous setting is zero, a frame has been lost. If the counter of the previous setting is less than zero, a sensor setting has been lost. The output setting analyzer 234 sends a fault mode 242 indicating the identified fault mode (e.g., setting update delay, sensor setting loss, or frame loss) to the error generator 244, and the error generator 244 sends error data 246 specifying the identified fault mode 242 to the error handler 248.

[0059] Figure 3AFIG. 300 shows a flow chart of a method 300 for processing a setup frame containing initial data from a sensor during an initial stage. Each block of the method 300 described herein includes a computational process that can be executed using any combination of hardware, firmware, and / or software. For example, various functions can be executed by a processor that executes instructions stored in a memory. The method can also be embodied as computer-usable instructions stored on a computer storage medium. The method can be provided by a stand-alone application, a service, or a hosted service (stand-alone or in combination with another hosted service), or a plug-in of another product, to name a few. Additionally, method 300 is described by way of example with respect to Figures 1 to 2 the system. However, the method can alternatively or additionally be executed by any one system or any combination of systems, including but not limited to the systems described herein. Additionally, the operations in method 300 can be omitted, repeated, and / or executed in any order without departing from the scope of the present disclosure.

[0060] As shown in FIG. 3, method 300 begins at operation 302, where a sensor fault detector 122 receives an output setup in a sensor output frame. At operation 304, the sensor fault detector 122 determines whether the current execution stage of the sensor fault detector 122 is an initial execution stage, where each sensor output frame 216 contains an output setup 222 having an initial setup value (also referred to herein as “Si”). Each output setup 222 during the initialization stage can have the same initial setup value. If the sensor fault detector 122 is not in the initial stage, then at operation 306, the sensor fault detector 122 executes the operations of the runtime stage by executing Figure 3B flow chart 350, and method 300 ends. If the sensor fault detector 122 is in the initial stage, the sensor fault detector 122 executes operation 308. At operation 308, the sensor fault detector 122 determines whether the current frame number (“f”) is less than or equal to the expected number of initial sensor output frames. The current frame number corresponds to the sensor output frame received at operation 302, and can be 1 for the first frame received, 2 for the second frame received, and so on. Each initial sensor output frame is a sensor output frame 216 whose output setup 222 is a predetermined initial output setup rather than an output setup 222 determined based on an input setup 260. The initial output setup can be a set of predetermined setup values, such as a predetermined exposure and a predetermined gain. The initial output setup is referred to herein as “Sinitial”. If at operation 308 the sensor fault detector 122 determines that the current frame number f is less than or equal to the expected number M of initial frames, then method 300 ends. If another output setup is received in the sensor output frame, then method 300 is called again subsequently.

[0061] If in operation 308 the sensor fault detector 122 determines that the current frame number f is not less than or equal to the number M of initial frames, then in operation 310 the sensor fault detector 122 determines whether the current frame number f is equal to the number of initial frames plus 1 ("M + 1"). If so, the sensor fault detector 122 performs operation 312.

[0062] In operation 312, the sensor fault detector 122 determines whether the output setting 222 of the current sensor output frame 216 is the initial setting Sinitial. If so, the sensor fault detector 122 performs operation 318. In operation 318, the sensor fault detector 122 decrements the counter associated with the queue element having the initial setting (e.g., the queue element equal to the initial setting, or the queue element containing the initial setting). The queue element having the initial setting can be, for example, the first (oldest) element having the setting Sinitial found in the sensor setting queue 124. If in operation 312 the sensor fault detector 122 determines that the current sensor output frame 216 does not have the initial setting, then in operation 316 the sensor fault detector 122 reports an error.

[0063] If in operation 310 the sensor fault detector 122 determines that the current frame number is not equal to the number of initial frames M plus 1, then the sensor fault detector 122 performs operation 314. In operation 314, the sensor fault detector 122 determines whether the counter of the queue element having the initial setting (Sinitial) is greater than or equal to 0. If not, then in operation 316, the sensor fault detector 122 reports an error.

[0064] If in operation 314 the sensor fault detector 122 determines that the counter of the next queue element having the initial setting is greater than or equal to 0, then in operation 320, the sensor fault detector 122 determines whether the current frame from the sensor 212 has the initial setting. If so, the sensor fault detector 122 performs operation 318, which decrements the counter of the queue element having the initial setting (Sinitial). Otherwise, operation 320 determines that the current frame does not have the initial setting, but rather the current frame instead has an output setting 222 determined at runtime. The output setting 222 is determined by the sensor 212 based on the input setting 260. In operation 322, the sensor fault detector 122 removes each queue element having the initial setting from the sensor setting queue 124. In operation 324, the sensor fault detector 122 sets the current execution phase of the sensor fault detector 122 to the runtime execution phase to indicate that the frames having the initial setting have been consumed and now frames having the output setting 222 determined at runtime can be processed. In operation 326, the sensor fault detector 122 performs Figure 3BExecute the runtime phase according to the flowchart 350 of

[0065] Figure 3B FIG. shows a flowchart of a method 350 for processing a setup frame containing sensor detection data in a runtime phase according to various embodiments. Each block of the method 350 described herein includes a computational process that can be performed using any combination of hardware, firmware, and / or software. For example, various functions can be performed by executing instructions stored in a memory by a processor. The method can also be embodied as computer-usable instructions stored on a computer storage medium. The method can be provided by a stand-alone application, service, or hosted service (stand-alone or in combination with another hosted service) or a plug-in of another product, to name a few. Additionally, method 350 is described by way of example with respect to Figures 1 to 2 the system of. However, the method can alternatively or additionally be performed by any one system or any combination of systems, including but not limited to the systems described herein. Further, the operations in method 350 can be omitted, repeated, and / or performed in any order without departing from the scope of the present disclosure.

[0066] As Figure 3B shown, method 350 begins at operation 352, where the sensor fault detector 122 determines whether the sensor setup queue 124 contains an element that matches the output setup received in operation 302 of Figure 3A method 300. Alternatively, in operation 352, the sensor fault detector 122 can determine whether the sensor setup queue 124 contains an element that matches the output setup in a sensor output frame received from any suitable source. If in operation 350 the sensor fault detector 122 determines that the sensor setup queue 124 does not contain an element that matches the output setup, then in operation 354 the sensor fault detector 122 generates an error indicating frame or setup corruption. Otherwise, the sensor setup queue 124 contains an element that matches the output setup, and the sensor fault detector 122 performs operation 356.

[0067] In operation 356, the sensor fault detector 122 decrements a counter for the matching queue element, e.g., by subtracting 1 from the counter. In operation 358, the sensor fault detector 122 determines whether the matching queue element is at the head of the sensor setup queue 124. If so, method 350 ends. Otherwise, the matching queue element is not at the head of the sensor setup queue 124, and the sensor fault detector 122 performs operation 360. In operation 360, the sensor fault detector 122 dequeues each queue element that is ahead of the matching queue element in the queue (e.g., queue elements added to the queue before the matching element was added). The sensor fault detector 122 performs operation 362 on each dequeued queue element dequeued in operation 360.

[0068] In operation 362, the sensor fault detector 122 determines whether the counter associated with the dequeued queue element is less than 0. If so, then in operation 364, the sensor fault detector 122 determines whether the value of the next queue element (e.g., the next element in the sensor settings queue 124) is 0, in which case the sensor fault detector 122 performs operation 366, or greater than 0, in which case the sensor fault detector 122 performs operation 370. In operation 366, the sensor fault detector 122 generates an error indicating that a sensor settings update delay has occurred. In operation 370, the sensor fault detector 122 generates an error indicating that a sensor settings loss has occurred.

[0069] If in operation 362 the sensor fault detector 122 determines that the counter associated with the dequeued queue element is less than 0, then the sensor fault detector 122 performs operation 372. In operation 372, the sensor fault detector 122 determines whether the counter associated with the dequeued queue element is greater than 0. If so, then the sensor fault detector 122 performs operation 376. In operation 376, the sensor fault detector 122 determines whether the value of the counter associated with the previous queue element (e.g., the previous element in the sensor settings queue 124) is less than 0, in which case the sensor fault detector 122 performs operation 370, or equal to 0, in which case the sensor fault detector 122 performs operation 378. In operation 378, the sensor fault detector 122 generates an error indicating that a sensor output frame loss has occurred. The errors generated in operations 366, 370, and / or 376 can be, for example, error data 246 that includes a fault mode indicator 242 indicating the type of error. After operation 366, 370, or 376, the sensor fault detector 122 performs operation 368, which determines whether there is another queue element before the matching queue element in the sensor settings queue 124. If so, then the sensor fault detector 122 performs operation 360 and the subsequent operations again. If not, then method 350 ends. Additionally, if in operation 372 the sensor fault detector 122 determines that there is another queue element before the matching queue element in the sensor settings queue 124, then the sensor fault detector 122 performs operation 360 and the subsequent operations again.

[0070] Figure 4A flowchart of a method 400 for enqueuing sensors into a settings history queue according to various embodiments is shown. Each block of the method 400 described herein includes a computing process that can be performed using any combination of hardware, firmware, and / or software. For example, various functions can be implemented by a processor executing instructions stored in a memory. The method can also be embodied as computer-usable instructions stored on a computer storage medium. The method can be provided by a stand-alone application, a service, or a hosted service (stand-alone or in combination with another hosted service), or a plug-in for another product, to name a few. Additionally, the method 400 is described by way of example with respect to Figures 1 to 2 the system. However, the method can alternatively or additionally be performed by any one system or any combination of systems, including but not limited to the systems described herein. Further, the operations in the method 400 can be omitted, repeated, and / or performed in any order without departing from the scope of the present disclosure.

[0071] As Figure 4 shown, the method 400 begins at operation 402, where a sensor fault detector 122 receives an input setting 260, e.g., from a sensor settings generator 204. At operation 404, the sensor fault detector 122 determines whether the input setting matches (e.g., is equal to) a previous input setting in the sensor fault detector 122. The previous input setting can be the most recently added setting prior to receiving the input setting at operation 402. If operation 404 determines that the input setting matches the previous input setting, the sensor fault detector 122 performs operation 406. At operation 406, the sensor fault detector 122 adds (e.g., enqueues) a queue element at the tail of the sensor settings queue 124. The added queue element includes the input setting and is associated with a counter value of 1. If operation 404 determines that the input setting does not match the previous setting, at operation 408, the sensor fault detector 122 increments the counter of the queue element at the tail of the sensor settings queue 124.

[0072] Figure 5A and Figure 5BShows an example execution 500 of a sensor fault detector in the initial stage of the fault - free operation of a sensor system. In the example fault - free execution 500, the input setting sequence 502 generated by the sensor setting generator 204 is provided to the sensor fault detector 122. The input setting sequence 502 is also provided to the sensor 212, for example, by the sensor fault detector 122 or provided by the sensor fault detector 122. Each element of the input setting sequence 502 corresponds to the input setting 260. The input setting sequence 502 includes elements Si, Si, Si, Si, Si, S1, S2, S2, S3, and each element represents the input setting 260. Si (also referred to herein as Sinitial) represents a predetermined initial setting generated by the sensor setting generator 204 during the initial stage, and S1, S2, S3 represent three different settings. In this example, the initial setting Si is sent to the sensor fault detector 122 (and also sent to the sensor 212) M = 5 times. For example, the initial setting can be sent to prepare the sensor for operation. The sensor fault detector 122 receives and processes each element of the input setting sequence 502 (e.g., each input setting 260) within the corresponding time period corresponding to the frame, and updates the sensor setting queue 124 accordingly, as Figure 5A and Figure 5B shown. As described with respect to Figure 3A the first M (e.g., 5) output frames received from the sensor 212 are expected to have the initial setting Si and can be discarded.

[0073] The sensor 212 generates an output setting sequence 504 in response to receiving the input setting sequence 502. Each element of the output setting sequence 504 corresponds to the output setting 222 of the sensor output frame 216. The output setting sequence 502 includes elements Sr, Sr, Si, Si, Si, Si, S1, S2, S2, S3. The setting Sr corresponds to random or unknown data generated by the sensor setting generator 204 before initialization. Thus, the output setting sequence 502 includes two occurrences of the random setting Sr, three occurrences of the initial setting Si, one occurrence of the setting S1, two occurrences of the setting S2, and one occurrence of the setting S3. The sensor fault detector 122 receives and processes each element of the output setting sequence 504 (e.g., each output setting 222) within the corresponding time period corresponding to the frame, and updates the sensor setting queue 124 accordingly, as Figure 5A and Figure 5B shown.

[0074] The input setting sequence 502 is processed by the input setting enqueuer 210. Figure 4 Shows an example method for processing input settings. The output setting sequence 504 is processed by the output setting analyzer 234. Figure 3A andFigure 3B An example method for processing output settings is shown. The input setting sequence 502 can be processed in parallel with the output setting sequence 504.

[0075] At frame "1" 520-1, the sensor fault detector 122 receives the first of five Si input settings. If the input setting is different from the previous input setting, the sensor fault detector 122 enqueues the input setting into the sensor setting queue 522. Since there is no previous setting at frame 1, the input setting Si is enqueued, resulting in a setting queue 522-1 that contains the setting Si. The sensor fault detector 122 also sets the count associated with the setting Si to 1 because the input setting is different from the previous setting (as referenced Figure 4 as described). Thus, the queue 522-1 contains a single element Si with a count = 1. The count can be stored in the queue 522-1 (e.g., in the same queue element as the setting Si), or stored independently of the queue. Additionally, at frame 1 520-1, the sensor fault detector 122 receives the first of two Sr output settings from the sensor 212. Since the frame number is 1, which is less than or equal to the expected initial frame number M = 5, the sensor fault detector 122 does not perform any further actions on frame 520-1.

[0076] At frame "2" 520-2, the sensor fault detector 122 receives the second of five Si input settings. Since Si is equal to the previous setting (the first Si, from frame 1), the sensor fault detector 122 increments the count of the previously enqueued Si (at the tail of the queue). Thus, the setting queue 522-2 contains the setting Si with an associated count = 2. Additionally, at frame 2 520-2, the sensor fault detector 122 receives the second of two Sr output settings. Since the frame number is 2, which is less than or equal to the expected initial frame number M = 5, the sensor fault detector 122 does not perform any further actions on frame 2.

[0077] At frame "3" 520-3, the sensor fault detector 122 receives the third of five Si input settings. Since Si is equal to the previous setting (the second Si, from frame 2), the sensor fault detector 122 increments the count of the previously enqueued Si. Thus, the setting queue 522-3 contains the setting Si with an associated count = 3. Additionally, at frame 3 520-3, the sensor fault detector 122 receives the first of five Si output settings. Since the frame number is 3, which is less than or equal to the expected initial frame number M = 5, the sensor fault detector 122 does not perform any further actions on frame 3.

[0078] At frame "4" 520-4, the sensor fault detector 122 receives the fourth of the five Si input settings. Since Si is equal to the previous setting (the third Si, from frame 3), the sensor fault detector 122 increments the count of the previously enqueued Si. Thus, the settings queue 522-4 contains the setting Si with an associated count = 4. Additionally, at frame 4 520-4, the sensor fault detector 122 receives the second of the five Si output settings. Since the frame number is 4, which is less than or equal to the expected initial frame number M = 5, the sensor fault detector 122 does not perform any further actions on frame 4.

[0079] At frame "5" 520-5, the sensor fault detector 122 receives the fifth of the five Si input settings. Since Si is equal to the previous setting (the fourth Si, from frame 4), the sensor fault detector 122 increments the count of the previously enqueued Si. Thus, the settings queue 522-5 contains the setting Si with an associated count = 5. Additionally, at frame 5 520-5, the sensor fault detector 122 receives the third of the five Si output settings. Since the frame number is 5, which is less than or equal to the expected initial frame number M = 5, the sensor fault detector 122 does not perform any further actions on frame 5.

[0080] At frame "6" 520-6, the sensor fault detector 122 receives the S1 input setting. Since S1 is not equal to the previous setting (the fifth Si, from frame 5), the sensor fault detector 122 adds the input setting S1 to the settings queue 522-6 with a count = 1. Thus, the settings queue 522-6 contains the setting Si with an associated count = 5 as the first element (at the head of the queue) and the setting S1 with an associated count = 1 as the second element (at the tail of the queue). Additionally, at frame 6 520-6, the sensor fault detector 122 receives the fourth of the five Si output settings. Since the frame number is 6, which is equal to the expected initial frame number M = 5 plus 1 (see Figure 3A operation 310), and the current output frame is set to Si (see operation 312), the sensor fault detector 122 performs the action of decrementing the counter of the queue element with the setting = Si (see operation 318). After performing this action, the settings queue 524-6 contains the setting Si with a count = 4 at the head of the queue. Since the currently received output frame is set to Si (see operation 312), no error occurs at frame 6 (status: Good (OK)).

[0081] At frame “7” 520-7, the sensor fault detector 122 receives the first of two S2 input settings. Since S2 is not equal to the previous setting (S1), the sensor fault detector 122 adds the input setting S2 to the setting queue 522-7 with count = 1. As a result, the setting queue 522-7 contains the setting Si with count = 4 as the first element (at the head of the queue), the setting S1 with count = 1 as the second element, and the setting S2 with count = 1 as the third element (at the tail of the queue). Additionally, at frame 7 520-7, the sensor fault detector 122 receives the fifth of five Si output settings. Since the frame number is 7, which is greater than or equal to the expected initial frame number M = 5 plus 2 (in Figure 3A operation 310), the counter of the queue element with setting Si is greater than or equal to 0 (see operation 314), and the current input frame is set to Si (see operation 320), the sensor fault detector 122 performs the action of removing each queue element with setting = Si from the queue (see operation 320). After performing this operation, the setting queue 524-7 contains the setting Si with associated count = 3 at the head of the queue. Since the currently received output frame is set to Si (see operation 312), no error occurred at frame 6 (status: OK).

[0082] At frame “8” 520-8, the sensor fault detector 122 receives the second of two S2 input settings. Since S2 is equal to the previous setting (S2), the sensor fault detector 122 increments the count associated with S2 in the setting queue 522-8. Thus, the setting queue 522-8 contains the setting Si with count = 3 as the first element (at the head of the queue), the setting S1 with count = 1 as the second element, and the setting S2 with count = 2 as the third element (at the tail of the queue). Additionally, at frame 8 520-8, the sensor fault detector 122 receives the S1 output setting. Since the frame number is 8, which is greater than or equal to the expected initial frame number M = 5 plus 2 (in Figure 3A operation 310), the counter of the queue element with setting Si is greater than or equal to 0 (see operation 314), and the current input frame setting (S1) is not equal to Si (see operation 320), the sensor fault detector 122 performs the action of removing the queue element with setting = Si from the queue (see operation 322) and decrements the counter associated with the current input frame setting (S1). The sensor fault detector 122 also sets the current phase indicator to “runtime” (operation 324) so that subsequent input settings and output settings are processed by the Figure 3B runtime phase method. After performing this action, the setting queue 524-8 contains the setting S1 with count = 0 at the head of the queue and further contains the setting S2 with count = 2 at the tail of the queue. No error occurred at frame 6 (status: OK).

[0083] At frame "9" 520-9, the sensor fault detector 122 receives the S3 input setting. Since S3 is not equal to the previous setting (S2), the sensor fault detector 122 adds S3 to the setting queue 522-9. As a result, the setting queue 522-9 contains the setting S1 with count = 0 as the first element (at the head of the queue), the setting S2 with count = 2 as the second element, and the setting S3 with count = 1 as the third element. Additionally, at frame 9 520-9, the sensor fault detector 122 executes Figure 3B the runtime phase method 350 instead of Figure 3A the initial phase method 300 because the current phase indicator is set to "runtime" in frame 8. Thus, the sensor fault detector 122 receives the output setting S2 and determines whether the setting queue 522-9 contains an element with a setting that matches the received output setting S2 (see operation 352). Since the setting queue 522-9 contains S2, the sensor fault detector 122 decrements the counter of S2 (see operation 356). The sensor fault detector 122 determines whether the matching queue element (S2) is at the head of the queue (see operation 358). Since S2 is not at the head of the queue, the sensor fault detector 122 dequeues each element that is in front of the matching queue element (see operation 360). The sensor fault detector 122 compares the counter of the dequeued element S2 with 0 (see operations 362 and 372). Since the counter is zero, frame 9 does not generate any error (status: OK). After performing this action, the setting queue 524-8 contains the setting S2 with count = 1 at the head of the queue and also contains the setting S3 with count = 1 at the tail of the queue.

[0084] Figure 5C FIG. 501 shows an example execution of the initial phase of the sensor fault detector when a sensor setting corruption occurs, according to some embodiments of the present disclosure. In the example execution 501, an input setting sequence 506 is provided to the sensor fault detector 122 and the sensor 212. The input setting sequence 506 is the same as Figure 5A the input sequence 502 and includes the elements Si, Si, Si, Si, Si, S1, S2, S2, S3. The output setting sequence 508 includes the elements Sr, Sr, Si, Si, Si, S100, S1, S2, S2, S3. Frames "1" 530-1 to "5" 530-5 are the same as Figure 5A and Figure 5BFrames "1" 520-1 to "5" 520-5 are the same. At frame "6" 530-6, the sensor fault detector 122 receives the input setting S1. The sensor fault detector 122 enqueues the input setting S1 into the sensor setting queue 532-6 because S1 is different from the previous input setting (Si). The sensor fault detector 122 also sets the count associated with the setting S1 to 1. Thus, the queue 532-6 contains the element Si with count = 5 (at the head of the queue) and the element S1 with count = 1. Additionally, at frame 6 530-6, the sensor fault detector 122 receives the S100 output setting from the sensor 212. The sensor fault detector 122 searches for the received setting S100 in the setting queue 532-6 (see Figure 3B operation 352). Since the setting queue 532-6 does not contain the setting S100, the sensor fault detector 122 generates a setting corrupted error (see operation 354).

[0085] Figure 6A FIG. 600 shows an example execution of the runtime phase of the sensor fault detector when a sensor setting corruption occurs, according to some embodiments of the present disclosure. In the example execution 600, the input setting sequence 602 includes the elements S-1, S0, S1, S2, S3, S4 (where S-1 represents S-1). The output setting sequence 604 includes the elements S-1, S0, S100, S2, S3, S4.

[0086] At frame N 620-1, the sensor fault detector 122 receives the S1 input setting. Since S1 is not equal to the previous setting, the sensor fault detector 122 adds the input setting S1 to the setting queue 622-1 with count = 1. As a result, the setting queue 522-6 contains the setting S-1 with count = 1 (at the head of the queue), the setting S0 with count = 1, and the setting S1 with count = 1. Additionally, at frame N 620-1, the sensor fault detector 122 receives the output setting S-1. The sensor fault detector 122 is in the runtime phase. The sensor fault detector 122 finds the queue element that matches the output setting S-1 (see operation 352) and decrements the counter of the matching queue element (see operation 356). Since the matching queue element is at the head of the queue, no dequeue is performed and no error occurs at frame N (status: OK). After performing this action, the setting queue 624-1 contains the setting S-1 with count = 0 at the head of the queue, the setting S0 with count = 1 in the second queue element, and the setting S1 with count = 1 in the third queue element.

[0087] At frame N+1 620-2, the sensor fault detector 122 receives the S2 input setting. Since S2 is not equal to the previous setting, the sensor fault detector 122 adds the input setting S2 to the setting queue 622-2 with count = 1. Thus, the setting queue 622-2 contains setting S-1 with count = 0 (at the head of the queue), setting S0 with count = 1, setting S1 with count = 1, and setting S2 with count = 1. Additionally, at frame N+1 620-2, the sensor fault detector 122 receives the output setting S0. The sensor fault detector 122 is operating in the runtime phase. The sensor fault detector 122 finds the queue element that matches the output setting S0 (see operation 352) and decrements the counter of the matching queue element (see operation 356). Since the matching queue element is not at the head of the queue, each element in front of the matching element S0 is dequeued (see operation 360), and the counter of each dequeued queue element is compared to 0. Since setting S-1 is in front of S0 in the queue, setting S-1 has a counter of 0 for the dequeued queue element S-1, so no error occurred at frame N (status: OK). After performing this action, the setting queue 624-2 contains setting S0 with count = 1 in the head element, setting S1 with count = 1 in the second element, and setting S2 with count 1 in the third element. Since the matching queue element S0 is at the head of the queue, no dequeue is performed, and no error occurred at frame N (status: OK).

[0088] At frame N+2 620-3, the sensor fault detector 122 receives the S3 input setting. Since S3 is not equal to the previous setting, the sensor fault detector 122 adds the input setting S3 to the setting queue 622-3 with count = 1. Thus, the setting queue 622-3 contains setting S0 with count = 1 (at the head of the queue), setting S1 with count = 1, setting S2 with count = 1, and setting S3 with count = 1. Additionally, at frame N+2 620-3, the sensor fault detector 122 receives the output setting S100. The sensor fault detector 122 searches the queue 622-3 for a queue element that matches the output setting S100, but since the setting S100 does not exist in the queue, the setting S100 is not found. Since no matching queue element is found, the sensor fault detector 122 generates a setting corrupted error (see operation 354).

[0089] Figure 6BShows an example execution 605 of the runtime phase of a sensor fault detector when a sensor setup delay occurs, according to some embodiments of the present disclosure. In the example execution 605, the input setup sequence 606 includes elements S-1, S0, S1, S2, S3, S4, S5 (where S-1 represents S-1). The output setup sequence 608 includes elements S-1, S0, S1, S1, S2, S3. Setup S2 has been delayed, so S1 appears twice in the received setup sequence 608. Figure 6B frames N 630-1 and N+1 630-2 of Figure 6A are the same as frames N 620-1 and N+1 620-2 of

[0090] At frame N+2 630-3, the sensor fault detector 122 receives the S3 input setup. Since S3 is not equal to the previous setup (S2), the sensor fault detector 122 adds the input setup S3 to the setup queue 632-3 with count = 1. Thus, the setup queue 632-3 contains setup S0 with count = 0 (at the head of the queue), followed by setups S1, S2, and S3, each with count = 1. Additionally, at frame N+2 630-3, the sensor fault detector 122 receives the first of two S1 output setups. The sensor fault detector 122 finds the queue element that matches the output setup S1 and decrements the counter of the matching queue element S1. Since the matching queue element is not at the head of the queue, the sensor fault detector 122 dequeues each element (S0) that is in front of the matching queue element. The sensor fault detector 122 compares the counter of the dequeued element S0 with 0. Since the counter is zero, no error is generated for frame N+1 630-3 (status: OK).

[0091] At frame N+3 630-4, the sensor fault detector 122 receives the S4 input setup. Since S4 is not equal to the previous setup (S3), the sensor fault detector 122 adds the input setup S4 to the setup queue 632-4 with count = 1. Additionally, at frame N+3 630-3, the sensor fault detector 122 receives the second S1 output setup. The sensor fault detector 122 finds the queue element that matches the output setup S1 and decrements the counter of the matching queue element S1. Due to this decrement, the count associated with S1 becomes -1. Since the matching queue element is at the head of the queue, the sensor fault detector 122 does not dequeue any elements at frame N+3 630-4 and no error is generated at frame N+3 630-4.

[0092] At frame N+4 630-5, the sensor fault detector 122 receives the S5 input setting. Since S5 is not equal to the previous setting (S4), the sensor fault detector 122 adds the input setting S5 to the setting queue 632-5 with count = 1. Additionally, at frame N+4 630-3, the sensor fault detector 122 receives the second S2 output setting. The sensor fault detector 122 finds the queue element that matches the output setting S2 and decrements the counter of the matching queue element S2. Due to this decrement, the count associated with S2 becomes 0. Since the matching queue element S2 is not at the head of the queue, the sensor fault detector 122 dequeues each element (S1) that is in front of the matching queue element S2. The sensor fault detector 122 compares the counter of the dequeued element S1 with 0. Since the counter of S1 is less than 0 and the counter of the next queue element S2 is 0, the sensor fault detector 122 determines that a setting update delay has occurred (see Figure 3B operation 366).

[0093] Figure 6C Illustrates an example execution 607 of the runtime phase of the sensor fault detector when a sensor output frame loss occurs, according to some embodiments of the present disclosure. In the example execution 607, the input setting sequence 610 includes the elements S-1, S0, S1, S1, S2, S3, S4 (where S-1 represents S-1). The output setting sequence 612 includes the elements S-1, S0, S1, S2, S3, S4. The second S1 does not exist in the output setting sequence 612, so an output setting loss error has occurred. Figure 6B The frame N 640-1 of Figure 6B is the same as the frame N 630-1 of

[0094] At frame N+1 640-2, the sensor fault detector 122 receives the S1 input setting. Since S1 is equal to the previous setting (S1), the sensor fault detector 122 increments the count of S1 in the setting queue 642-2, resulting in count = 2. Additionally, at frame N+1 630-2, the sensor fault detector 122 receives the S0 output setting. The sensor fault detector 122 finds the queue element that matches the output setting S0 and reduces the counter of the matching queue element S0. Due to the reduction, the count associated with S0 becomes 0. Since the matching queue element is not at the head of the queue, the sensor fault detector 122 dequeues each element (S-1) that is in front of the matching queue element (S0). The sensor fault detector 122 compares the counter of the dequeued element S-1 with 0. Since the counter is zero, no error is generated for frame N+1 640-2 (status: OK).

[0095] At frame N+2 640-3, the sensor fault detector 122 receives the S2 input setting. Since S2 is not equal to the previous setting (S1), the sensor fault detector 122 adds the input setting S2 to the setting queue 642-3 with a count = 1. Thus, the setting queue 642-3 contains setting S0 with count = 0 (at the head of the queue), followed by setting S1 with count = 2 and setting S2 with count = 1. Additionally, at frame N+2 640-3, the sensor fault detector 122 receives the S1 output setting. The sensor fault detector 122 finds the queue element that matches the output setting S1 and decrements the counter of the matching queue element S1. Since the matching queue element is not at the head of the queue, the sensor fault detector 122 dequeues each element (S0) that is located in front of the matching queue element. The sensor fault detector 122 compares the counter of the dequeued element S0 with 0. Since the counter is zero, no error is generated for frame N+2 640-3 (status: OK).

[0096] At frame N+3 640-4, the sensor fault detector 122 receives the S3 input setting. Since S3 is not equal to the previous setting (S2), the sensor fault detector 122 adds the input setting S3 to the setting queue 642-4 with a count = 1. Additionally, at frame N+3 640-4, the sensor fault detector 122 receives the S2 output setting. The sensor fault detector 122 finds the queue element that matches the output setting S2 and decrements the counter of the matching queue element S2. Due to the decrement, the counter associated with S2 becomes 0.

[0097] Since the matching queue element is not at the head of the queue, the sensor fault detector 122 dequeues each element (S1) that is located in front of the matching queue element (S2). The sensor fault detector 122 compares the counter of the dequeued element S1 with 0. Since the counter is 1 (greater than 0) and the counter of the previous dequeued element (0 for element S2), an output frame loss error is generated for frame N+3 (see Figure 3B operation 378).

[0098] Figure 6D An example execution 609 of the runtime phase of the sensor fault detector when a sensor setting is lost is shown in accordance with some embodiments of the present disclosure. In the example execution 609, the input setting sequence 614 includes elements S-1, S0, S1, S2, S3, S4 (where S-1 represents S-1). The output setting sequence 616 includes elements S-1, S0, S1, S1, S2, S3. The input setting S2 has been discarded, so S1 appears twice in the received setting sequence 616. Figure 6D Frames N 650-1 to N+3 650-4 of Figure 6Bis the same for frames N 630-1 to N+3 630-4.

[0099] At frame N+4 650-5, the sensor fault detector 122 receives the S5 input setting. Since S5 is not equal to the previous setting (S4), the sensor fault detector 122 adds the input setting S5 to the setting queue 652-5 with count = 1. As a result, the setting queue 652-5 contains the setting S1 with count = -1 (at the head of the queue), followed by the settings S2, S3, S4, and S5 with count = 1. Additionally, at frame N+4 650-5, the sensor fault detector 122 receives the S3 output setting. The sensor fault detector 122 finds the queue element that matches the output setting S3 and decrements the counter of the matching queue element S3. Since the matching queue element S3 is not at the head of the queue, the sensor fault detector 122 dequeues each element (S1 and S2) that is before the matching queue element. The sensor fault detector 122 compares the counter of the dequeued element S1 with 0. Since the counter of S1 is less than 0 and the counter of the next queue element S2 is greater than 0, an input setting loss error is generated for frame N+4 650-5 (see operation 370).

[0100] It should be understood that this and other arrangements described herein are presented only as examples. Other arrangements and elements (e.g., machines, interfaces, functions, sequences, function groupings, etc.) may be used in addition to the arrangements and elements shown, and some elements may be omitted entirely. Moreover, many of the elements described herein are functional entities that may be implemented as discrete or distributed components or in combination with other components, and may be implemented in any suitable combination and location. The various functions performed by the entities described herein may be executed by hardware, firmware, and / or software. For example, the various functions may be executed by a processor that executes instructions stored in a memory. In some embodiments, the systems, methods, and processes described herein may use components, features, and / or functions similar to those of Figures 7A to 7D the example autonomous vehicle 700, Figure 8 the example computing device 800, and / or Figure 9 the example data center 900.

[0101] In summary, the disclosed technology uses a settings verification system to detect and report faults that occur when sending or applying input sensor settings to a sensor or when receiving output sensor settings from a sensor. The settings verification system monitors the "input" settings sent by a sensor controller or other system component to the sensor. The settings verification system also monitors the "output" settings that are received with each output data frame generated by the sensor and that indicate the settings used by the sensor to generate the output data frame. When a setting is sent to the sensor, the settings verification system stores each input setting in a settings history queue and associates a count value with each input setting in the queue. When an input setting is sent to the sensor and an output setting is received from the sensor, the count associated with each input setting in the queue is updated, and represents the difference between the number of consecutive times the input setting was sent to the sensor and the number of times an output setting matching the input setting was received.

[0102] The settings verification system uses the queue to detect settings faults. For example, settings faults may include failure to receive an expected input setting at the sensor ("setting lost"), failure to receive a sensor output data frame containing an output setting that matches the corresponding input setting ("frame lost"), a delay in updating the sensor to use the input setting ("setting delay"), or corruption of an input setting or output setting ("setting corruption"). If no fault occurs, the count associated with a particular input setting in the queue goes to zero before an output data frame containing a different output setting that matches a different input setting sent to the sensor after the particular input setting is received. A setting corruption fault can be detected when an output setting is received. If no input setting matching the output setting is found in the queue, a setting corruption fault has occurred. Other types of faults, such as setting lost, frame lost, or setting delay, can be detected when an output setting is received and the output setting matches an input setting that is in the queue but not at the head of the queue. In this case, there is at least one non-matching input setting in front of the matching output setting in the queue. If the count associated with the non-matching input setting is non-zero, a fault has occurred. Fault identification is based on whether the count associated with the matching output setting is less than zero (e.g., setting update delay or sensor setting lost) or greater than zero (e.g., frame lost or sensor setting lost). A particular fault can be identified based on the count associated with the next setting in the queue or the previous setting in the queue. If a fault is identified, the settings verification system generates an error to notify other system components (e.g., a sensor controller or an application) of the fault type and the output setting and / or input setting associated with the fault.

[0103] One technical advantage of the disclosed technology, relative to the prior art, is the ability to detect faults in the application sensor settings when applying a variety of different settings, such as what occurs when using continuous auto-exposure. Another technical advantage of the disclosed technology is the ability to identify and report fault modes. Fault modes that can be detected include data corruption, loss of settings sent but not used by the sensor, or loss of output frames not received from the sensor. The identified fault modes can be used to find the cause of the fault and take corrective action. These technical advantages represent one or more technical improvements over prior art methods.

[0104] The systems and methods described herein can be used by and / or in conjunction with non-autonomous vehicles, semi-autonomous vehicles (e.g., in one or more adaptive driving assistance systems (ADAS)), manned and unmanned robots or robotic platforms, warehouse vehicles, off-road vehicles, vehicles coupled to one or more trailers, aircraft, boats, shuttles, emergency response vehicles, motorcycles, electric or motorized bicycles, airplanes, engineering vehicles, underwater vehicles, drones, and / or other vehicle types, but are not limited thereto. Additionally, the systems and methods described herein can be used for a variety of purposes, such as but not limited to, for machine control, machine motion, machine driving, synthetic data generation, model training, perception, augmented reality, virtual reality, mixed reality, robotics, security and surveillance, simulation and digital twins, autonomous or semi-autonomous machine applications, deep learning, environment simulation, object or participant simulation and / or digital twins, data center processing, conversational AI (artificial intelligence), optical transmission simulation (e.g., ray tracing, path tracing, etc.), collaborative content creation of 3D assets, cloud computing, and / or any other suitable application.

[0105] The disclosed embodiments can be incorporated into a variety of different systems, such as automotive systems (e.g., control systems of autonomous or semi-autonomous machines, perception systems of autonomous or semi-autonomous machines), systems implemented using robots, aviation systems, medical systems, navigation systems, intelligent area monitoring systems, systems performing deep learning operations, systems performing simulation operations, systems for performing digital twin operations, systems implemented using edge devices, systems including one or more virtual machines (VMs), systems for performing synthetic data generation operations, systems implemented at least partially in a data center, systems performing conversational AI operations, systems performing optical transmission simulation, systems performing collaborative content creation of 3D assets, systems implemented at least partially using cloud computing resources, and / or other types of systems.

[0106] Example autonomous vehicle

[0107] Figure 7AIllustrated is an example autonomous vehicle 700 according to some embodiments of the present disclosure. The autonomous vehicle 700 (alternatively, referred to herein as "vehicle 700") can include, but is not limited to, passenger vehicles such as cars, trucks, buses, emergency vehicles, shuttle vehicles, electric or motorized bicycles, motorcycles, fire trucks, police cars, ambulances, boats, construction vehicles, underwater vessels, robotic vehicles, drones, airplanes, vehicles coupled to trailers (e.g., semi-trailer trucks for hauling cargo) and / or other types of vehicles (e.g., driverless and / or accommodating one or more passengers). Autonomous vehicles are generally described according to the levels of automation defined by the National Highway Traffic Safety Administration (NHTSA), a division of the United States Department of Transportation, and the Society of Automotive Engineers (SAE) in "Taxonomy and Definitions for Terms Related to Driving Automation Systems for On-Road Motor Vehicles" (Standard No. J3016 - 201806, issued June 15, 2018, Standard No. J3016 - 201609, issued September 30, 2016, and prior and future versions of the standard). The vehicle 700 is capable of performing functions corresponding to one or more of Levels 3 - 5 of the autonomous driving levels. The vehicle 700 is capable of performing functions corresponding to one or more of Levels 1 - 5 of the autonomous driving levels. For example, depending on the embodiment, the vehicle 700 is capable of capabilities such as driver assistance (Level 1), partial automation (Level 2), conditional automation (Level 3), high automation (Level 4), and / or full automation (Level 5). The term "autonomous" as used herein can include any and / or all types of autonomy of 700 or other machines, such as full autonomy, high autonomy, conditional autonomy, partial autonomy, assistive autonomy, semi-autonomy, primary autonomy, or other designations.

[0108] The vehicle 700 can include components such as a chassis, a body, wheels (e.g., 2, 4, 6, 8, 18, etc.), tires, axles, and other components of the vehicle. The vehicle 700 can include a propulsion system 750, such as an internal combustion engine, a hybrid power plant, a fully electric motor, and / or another type of propulsion system. The propulsion system 750 can be connected to the driveline of the vehicle 700, which can include a transmission, to effect the propulsion of the vehicle 700. The propulsion system 750 can be controlled in response to signals received from the throttle / accelerator 752.

[0109] A steering system 754 that may include a steering wheel can be used to steer a vehicle 700 (e.g., along a desired path or route) while the propulsion system 750 is operating (e.g., while the vehicle is in motion). The steering system 754 can receive a signal from a steering actuator 756. For fully autonomous (Level 7) functionality, the steering wheel can be optional.

[0110] A brake sensor system 746 can be used to operate vehicle brakes in response to receiving a signal from a brake actuator 748 and / or a brake sensor.

[0111] One or more controllers 736 that may include one or more system-on-chips (SoCs) 704 ( Figure 7C ) and / or one or more GPUs can provide signals (e.g., representing commands) to one or more components and / or systems of the vehicle 700. For example, one or more controllers can send signals to operate vehicle brakes via one or more brake actuators 748, operate the steering system 754 via one or more steering actuators 756, and operate the propulsion system 750 via one or more throttles / accelerators 752. One or more controllers 736 can include one or more on-board (e.g., integrated) computing devices (e.g., supercomputers) that process sensor signals and output operation commands (e.g., signals representing commands) to enable autonomous driving and / or assist a human driver in driving the vehicle 700. One or more controllers 736 can include a first controller 736 for autonomous driving functionality, a second controller 736 for functional safety functionality, a third controller 736 for artificial intelligence functionality (e.g., computer vision), a fourth controller 736 for infotainment functionality, a fifth controller 736 for redundancy in emergency situations, and / or other controllers. In some examples, a single controller 736 can handle two or more of the above functions, two or more controllers 736 can handle a single function, and / or any combination thereof.

[0112] One or more controllers 736 may provide signals for controlling one or more components and / or systems of vehicle 700 in response to sensor data (e.g., sensor inputs) received from one or more sensors. The sensor data may be received from, for example and without limitation, a Global Navigation Satellite System (“GNSS”) sensor 758 (e.g., a Global Positioning System sensor), a RADAR sensor 760, an ultrasonic sensor 762, a LiDAR sensor 764, an Inertial Measurement Unit (IMU) sensor 766 (e.g., an accelerometer, a gyroscope, a magnetic compass, a magnetometer, etc.), a microphone 796, a stereo camera 768, a wide-angle camera 770 (e.g., a fish-eye camera), an infrared camera 772, a surround camera 774 (e.g., a 360-degree camera), a long-range and / or mid-range camera 798, a speed sensor 744 (e.g., for measuring the speed of vehicle 700), a vibration sensor 742, a steering sensor 740, a brake sensor (e.g., as part of a brake sensor system 746), and / or other sensor types. In some embodiments, the controller 736 may include one or more instances of a sensor fault detector 122 for monitoring sensor performance based on the respective sensor data.

[0113] One or more of the controllers 736 may receive inputs (e.g., represented by input data) from the instrument cluster 732 of the vehicle 700 and provide outputs (e.g., represented by output data, display data, etc.) via a Human Machine Interface (HMI) display 734, an auditory annunciator, a speaker, and / or via other components of the vehicle 700. These outputs may include information such as vehicle speed, rate, time, map data (e.g., Figure 7C a high-definition (HD) map 722), location data (e.g., the location of the vehicle 700 on a map, for example), direction, the location of other vehicles (e.g., occupancy grid), information about objects and object states as perceived by the controller 736, and so on. For example, the HMI display 734 may display information about the presence of one or more objects (e.g., street signs, warning signs, traffic light changes, etc.) and / or information about driving maneuvers that the vehicle has made, is making, or will make (e.g., changing lanes now, exiting 34B in two miles, etc.).

[0114] Vehicle 700 further includes a network interface 724 that may communicate over one or more networks using one or more wireless antennas 726 and / or a modem. For example, network interface 724 may be capable of communicating via Long Term Evolution (LTE), Wideband Code Division Multiple Access (WCDMA), Universal Mobile Telecommunications System (UMTS), Global System for Mobile Communications (GSM), IMT-CDMA Multi-Carrier (CDMA2000), etc. One or more wireless antennas 726 may also be used to enable communication between objects (such as vehicles, mobile devices, etc.) in an environment using one or more local area networks such as Bluetooth, Bluetooth Low Energy (LE), Z-Wave, ZigBee, etc. and / or one or more low power wide area networks (LPWANs) such as LoRaWAN, SigFox, etc.

[0115] Figure 7B For an example autonomous vehicle 700 in accordance with some embodiments of the present disclosure for Figure 7A example camera positions and fields of view. The cameras and respective fields of view are one example embodiment and are not intended to be limiting. For example, additional and / or alternative cameras may be included, and / or these cameras may be located at different positions on vehicle 700.

[0116] The type of camera used for the cameras may include, but is not limited to, digital cameras that may be adapted to work with components and / or systems of vehicle 700. The cameras may operate under an Automotive Safety Integrity Level (ASIL) B and / or under another ASIL. The camera type may have any image capture rate, such as 60 frames per second (fps), 120 fps, 240 fps, etc., depending on the embodiment. The cameras may be capable of using a rolling shutter, a global shutter, another type of shutter, or a combination thereof. In some examples, the color filter array may include a Red Clear Clear Clear (RCCC) color filter array, a Red Clear Clear Blue (RCCB) color filter array, a Red Blue Green Clear (RBGC) color filter array, a Foveon X3 color filter array, a Bayer sensor (RGGB) color filter array, a monochrome sensor color filter array, and / or another type of color filter array. In some embodiments, clear pixel cameras such as cameras having RCCC, RCCB, and / or RBGC color filter arrays may be used in an effort to increase light sensitivity.

[0117] In some examples, one or more of the cameras may be used to perform Advanced Driver Assistance System (ADAS) functions (such as as part of a redundant or fail-safe design). For example, a multi-functional monocular camera may be installed to provide functions including lane departure warning, traffic sign assistance, and smart headlight control. One or more of the cameras (such as all of the cameras) may record and provide image data (such as video) simultaneously.

[0118] One or more of the cameras may be mounted in a mounting assembly such as a custom-designed (three-dimensional (3D) printed) component to cut off stray light and reflections from within the vehicle (such as reflections from the instrument panel reflected in the windshield mirror) that may interfere with the image data capture capabilities of the cameras. With respect to the wing mirror mounting assembly, the wing mirror assembly may be custom 3D printed such that the camera mounting plate matches the shape of the wing mirror. In some examples, one or more cameras may be integrated into the wing mirror. For side view cameras, one or more cameras may also be integrated into the four pillars at each corner of the cab.

[0119] A camera having a field of view that includes an environmental portion in front of the vehicle 700 (e.g., a front camera) may be used for surround view to help identify forward paths and obstacles and to assist in providing information critical for generating an occupancy grid and / or determining a preferred vehicle path with the help of one or more controllers 736 and / or a control SoC. The front camera may be used to perform many of the same ADAS functions as LiDAR, including emergency braking, pedestrian detection, and collision avoidance. The front camera may also be used for ADAS functions and systems, including lane departure warning (LDW), adaptive cruise control (ACC), and / or other functions such as traffic sign recognition.

[0120] A variety of cameras may be used in a front-facing configuration, including, for example, a monocular camera platform that includes a complementary metal oxide semiconductor (CMOS) color imager. Another example may be a wide-angle camera 770, which may be used to sense objects (such as pedestrians, intersection traffic, or bicycles) entering the field of view from the periphery. Although Figure 7B only one wide-angle camera is illustrated, any number (including zero) of wide-angle cameras 770 may be present on the vehicle 700. Additionally, any number of long-range cameras 798 (such as a long-view stereo camera pair) may be used for depth-based object detection, particularly for objects for which a neural network has not been trained. The long-range cameras 798 may also be used for object detection and classification and basic object tracking.

[0121] Any number of stereo cameras 768 may also be included in a front-facing configuration. In at least one embodiment, one or more stereo cameras 768 may include an integrated control unit that includes a scalable processing unit that may provide a multi-core microprocessor and programmable logic (FPGA) with an integrated controller area network (CAN) or Ethernet interface on a single chip. Such a unit may be used to generate a 3D map of the vehicle environment, including distance estimates for all points in the image. Alternative stereo cameras 768 may include a compact stereo vision sensor that may include two camera lenses (one left and one right) and an image processing chip that may measure the distance from the vehicle to a target object and activate autonomous emergency braking and lane departure warning functions using the generated information (e.g., metadata). In addition to or alternatively to those described herein, other types of stereo cameras 768 may be used.

[0122] Cameras having a field of view of an environmental portion including the side of the vehicle 700 (e.g., side-view cameras) may be used for surround view, providing information used to create and update an occupancy grid and generate side-impact collision warnings. For example, surround cameras 774 (e.g., four surround cameras 774 as shown in Figure 7B may be disposed on the vehicle 700. The surround cameras 774 may include wide-angle cameras 770, fish-eye cameras, 360-degree cameras, and / or the like. By way of example, four fish-eye cameras may be disposed on the front, rear, and sides of the vehicle. In an alternative arrangement, the vehicle may use three surround cameras 774 (e.g., left, right, and rear) and may utilize one or more other cameras (e.g., a forward camera) as a fourth surround camera.

[0123] Cameras having a field of view of an environmental portion including the rear of the vehicle 700 (e.g., rear-view cameras) may be used for assisting in parking, surround view, rear collision warnings, and creating and updating an occupancy grid. A variety of cameras may be used, including but not limited to cameras that are also suitable as front-facing cameras as described herein (e.g., long-range and / or mid-range cameras 798, stereo cameras 768, infrared cameras 772, etc.).

[0124] Figure 7C For use in accordance with some embodiments of the present disclosure Figure 7ABlock diagram of an example system architecture of an example autonomous vehicle 700. It should be understood that this and other arrangements described herein are presented only as examples. Other arrangements and elements (e.g., machines, interfaces, functions, orders, function groupings, etc.) may be used in addition to or instead of those shown, and some elements may be omitted entirely. Further, many of the elements described herein are functional entities that may be implemented as discrete or distributed components or in combination with other components, and in any suitable combination and location. The various functions described herein as being performed by entities may be implemented by hardware, firmware, and / or software. For example, the various functions may be implemented by a processor executing instructions stored in memory.

[0125] Figure 7C Each of the components, features, and systems in vehicle 700 is illustrated as being connected via a bus 702. The bus 702 may include a Controller Area Network (CAN) data interface (alternatively referred to herein as the "CAN bus"). The CAN may be a network within vehicle 700 that is used to assist in controlling various features and functions of vehicle 700, such as the actuation of brakes, acceleration, braking, steering, windshield wipers, etc. The CAN bus may be configured to have dozens or even hundreds of nodes, each with its own unique identifier (e.g., CAN ID). The CAN bus may be read to find the steering wheel angle, ground speed, engine revolutions per minute (RPM), button positions, and / or other vehicle status indicators. The CAN bus may be ASIL B compliant.

[0126] Although the bus 702 is described herein as a CAN bus, this is not intended to be limiting. For example, in addition to or instead of the CAN bus, FlexRay and / or Ethernet may be used. Further, although the bus 702 is shown as a single line, this is not intended to be limiting. For example, any number of buses 702 may be present, which may include one or more CAN buses, one or more FlexRay buses, one or more Ethernet buses, and / or one or more other types of buses using different protocols. In some examples, two or more buses 702 may be used to perform different functions, and / or may be used for redundancy. For example, a first bus 702 may be used for collision avoidance functions, and a second bus 702 may be used for drive control. In any example, each bus 702 may communicate with any component of vehicle 700, and two or more buses 702 may communicate with the same component. In some examples, each SoC 704, each controller 736, and / or each computer within the vehicle may have access to the same input data (e.g., input from sensors of vehicle 700) and may be connected to a common bus such as the CAN bus.

[0127] Vehicle 700 may include one or more controllers 736, such as those described herein with respect to Figure 7A the controllers described. The controller 736 may be used for a variety of functions. The controller 736 may be coupled to any other different components and systems of the vehicle 700 and may be used for the control of the vehicle 700, the artificial intelligence of the vehicle 700, the infotainment for the vehicle 700, and / or the like.

[0128] Vehicle 700 may include one or more system-on-chips (SoCs) 704. The SoC 704 may include a CPU 706, a GPU 708, a processor 710, a cache 712, an accelerator 714, a data store 716, and / or other components and features not shown. In a variety of platforms and systems, the SoC 704 may be used to control the vehicle 700. For example, one or more SoCs 704 may be combined with an HD map 722 in a system (such as the system of the vehicle 700), and the HD map may obtain map refreshes and / or updates from one or more servers (such as Figure 7D one or more servers 778) via the network interface 724.

[0129] The CPU 706 may include a CPU cluster or a CPU complex (alternatively, referred to herein as "CCPLEX"). The CPU 706 may include multiple cores and / or an L2 cache. For example, in some embodiments, the CPU 706 may include eight cores in a coherent multi-processor configuration. In some embodiments, the CPU 706 may include four dual-core clusters, each of which has a dedicated L2 cache (such as a 2MB L2 cache). The CPU 706 (such as CCPLEX) may be configured to support simultaneous cluster operation such that any combination of the clusters of the CPU 706 can be active at any given time.

[0130] The CPU 706 may implement power management capabilities including one or more of the following features: each hardware block may automatically perform clock gating when idle to save dynamic power; each core clock may be gated when the core is not actively executing instructions due to the execution of WFI / WFE instructions; each core may perform power gating independently; when all cores perform clock gating or power gating, each core cluster may be clock-gated independently; and / or when all cores perform power gating, each core cluster may be power-gated independently. The CPU 706 may further implement an enhanced algorithm for managing power states, where the allowed power states and the desired wake-up times are specified, and the hardware / microcode determines the best power state for the cores, clusters, and CCPLEX to enter. The processing cores may support a simplified power state entry sequence in software, and this work is offloaded to the microcode.

[0131] The GPU 708 may include an integrated GPU (alternatively referred to herein as an "iGPU"). The GPU 708 may be programmable and efficient for parallel workloads. In some examples, the GPU 708 may use an enhanced tensor instruction set. The GPU 708 may include one or more streaming microprocessors, where each streaming microprocessor may include an L1 cache (e.g., an L1 cache with at least 96 KB of storage capacity), and two or more of these streaming microprocessors may share an L2 cache (e.g., an L2 cache with 512 KB of storage capacity). In some embodiments, the GPU 708 may include at least eight streaming microprocessors. The GPU 708 may use a compute application programming interface (API). Additionally, the GPU 708 may use one or more parallel computing platforms and / or programming models (e.g., NVIDIA's CUDA).

[0132] In the case of automotive and embedded use, the GPU 708 can be power optimized for optimal performance. For example, the GPU 708 can be fabricated on fin field-effect transistors (FinFETs). However, this is not intended to be restrictive, and the GPU 708 can be fabricated using other semiconductor manufacturing processes. Each streaming microprocessor can incorporate several mixed-precision processing cores divided into multiple blocks. For example, and without limitation, 64 PF32 cores and 32 PF64 cores can be divided into four processing blocks. In such an example, each processing block can be allocated 16 FP32 cores, 8 FP64 cores, 16 INT32 cores, two mixed-precision NVIDIA tensor cores for deep learning matrix arithmetic, an L0 instruction cache, a warp scheduler, a dispatch unit, and / or a 64 KB register file. Additionally, the streaming microprocessor can include independent parallel integer and floating-point data paths to provide efficient execution of workloads by leveraging a mix of computational and addressing computations. The streaming microprocessor can include independent thread scheduling capabilities to allow for finer-grained synchronization and collaboration between parallel threads. The streaming microprocessor can include a combined L1 data cache and shared memory unit to improve performance while simplifying programming.

[0133] The GPU 708 may include a high-bandwidth memory (HBM) and / or a 16 GB HBM2 memory subsystem that provides a peak memory bandwidth of approximately 900 GB / s in some examples. In some examples, in addition to or alternatively to HBM memory, synchronous graphics random access memory (SGRAM), such as fifth-generation graphics double data rate synchronous random access memory (GDDR5), may be used.

[0134] The GPU 708 may include unified memory technology that includes access counters to allow memory pages to be migrated more precisely to the processors that most frequently access them, thereby increasing the efficiency of the memory ranges shared among the processors. In some examples, address translation service (ATS) support may be used to allow the GPU 708 to directly access the CPU 706 page tables. In such examples, when the GPU 708 memory management unit (MMU) experiences a miss, an address translation request may be transmitted to the CPU 706. In response, the CPU 706 may look up the virtual-physical mapping for the address in its page table and transmit the translation back to the GPU 708. In this way, the unified memory technology may allow a single unified virtual address space for the memory of both the CPU 706 and the GPU 708, thus simplifying GPU 708 programming and porting applications to the GPU 708.

[0135] In addition, the GPU 708 may include access counters that may track how frequently the GPU 708 accesses the memory of other processors. The access counters may help ensure that memory pages are moved to the physical memory of the processors that most frequently access those pages.

[0136] The SoC 704 may include any number of caches 712, including those described herein. For example, the cache 712 may include an L3 cache available to both the CPU 706 and the GPU 708 (e.g., which is connected to both the CPU 706 and the GPU 708). The cache 712 may include a write-back cache that may track the state of lines, for example, by using a cache coherence protocol (such as MEI, MESI, MSI, etc.). Depending on the embodiment, the L3 cache may include 4MB or more, but smaller cache sizes may also be used.

[0137] The SoC 704 may include one or more arithmetic logic units (ALUs) that may be used to perform processing for any of the various tasks or operations regarding the vehicle 700, such as processing DNNs. In addition, the SoC 704 may include a floating-point unit (FPU) or other math co-processor or digital co-processor type for performing math operations within the system. For example, the SoC 704 may include one or more FPUs integrated as execution units within the CPU 706 and / or the GPU 708.

[0138] The SoC 704 may include one or more accelerators 714 (e.g., hardware accelerators, software accelerators, or a combination thereof). For example, the SoC 704 may include a hardware acceleration cluster, which may include optimized hardware accelerators and / or large on-chip memories. The large on-chip memory (e.g., 4MB SRAM) may enable the hardware acceleration cluster to accelerate neural networks and other computations. The hardware acceleration cluster may be used to supplement the GPU 708 and offload some of the tasks of the GPU 708 (e.g., freeing up more cycles of the GPU 708 for performing other tasks). As an example, the accelerator 714 may be used for targeted workloads that are stable enough to be easily accelerated (e.g., perception, convolutional neural network (CNN), etc.). As used herein, the term "CNN" may include all types of CNNs, including region-based or region convolutional neural networks (RCNNs) and fast RCNNs (e.g., for object detection).

[0139] The accelerator 714 (e.g., the hardware acceleration cluster) may include a deep learning accelerator (DLA). The DLA may include one or more tensor processing units (TPUs) that may be configured to provide an additional one trillion operations per second for deep learning applications and inference. The TPU may be an accelerator configured to perform image processing functions (e.g., for CNNs, RCNNs, etc.) and optimized for performing image processing functions. The DLA may be further optimized for a specific set of neural network types and floating-point operations and inference. The design of the DLA may provide higher performance per millimeter than a general-purpose GPU and far exceed the performance of a CPU. The TPU may perform several functions, including single-instance convolution functions, supporting INT8, INT16, and FP16 data types for both features and weights, and post-processor functions.

[0140] The DLA may execute neural networks, especially CNNs, quickly and efficiently for any of a variety of functions on processed or unprocessed data, such as, but not limited to: CNNs for object recognition and detection using data from a camera sensor; CNNs for distance estimation using data from a camera sensor; CNNs for emergency vehicle detection and identification and detection using data from a microphone; CNNs for face recognition and vehicle owner recognition using data from a camera sensor; and / or CNNs for security and / or safety-related events.

[0141] The DLA may perform any function of the GPU 708, and by using inference accelerators, for example, the designer may target the DLA or the GPU 708 for any function. For example, the designer may focus the processing and floating-point operations of a CNN on the DLA and leave other functions to the GPU 708 and / or other accelerators 714.

[0142] The accelerator 714 (e.g., a hardware acceleration cluster) can include a programmable vision accelerator (PVA), which may alternatively be referred to herein as a computer vision accelerator. The PVA can be designed and configured to accelerate computer vision algorithms for advanced driver assistance systems (ADAS), autonomous driving, and / or augmented reality (AR) and / or virtual reality (VR) applications. The PVA can provide a balance between performance and flexibility. For example, each PVA can include, for example and without limitation, any number of reduced instruction set computer (RISC) cores, direct memory access (DMA), and / or any number of vector processors.

[0143] The RISC cores can interact with an image sensor (e.g., the image sensor of any of the cameras described herein), an image signal processor, and / or the like. Each of these RISC cores can include any number of memories. Depending on the embodiment, the RISC cores can use any of several protocols. In some examples, the RISC cores can execute a real-time operating system (RTOS). The RISC cores can be implemented using one or more integrated circuit devices, application-specific integrated circuits (ASICs), and / or storage devices. For example, the RISC cores can include an instruction cache and / or tightly coupled RAM.

[0144] The DMA can enable the components of the PVA to access system memory independently of the CPU 706. The DMA can support any number of features used to provide optimizations to the PVA, including but not limited to supporting multi-dimensional addressing and / or circular addressing. In some examples, the DMA can support addressing up to six or more dimensions, which can include block width, block height, block depth, horizontal block stride, vertical block stride, and / or depth stride.

[0145] The vector processor can be a programmable processor that can be designed to efficiently and flexibly execute programming for computer vision algorithms and provide signal processing capabilities. In some examples, the PVA can include a PVA core and two vector processing subsystem partitions. The PVA core can include a processor subsystem, one or more DMA engines (e.g., two DMA engines), and / or other peripherals. The vector processing subsystem can operate as the main processing engine of the PVA and can include a vector processing unit (VPU), an instruction cache, and / or vector memory (e.g., VMEM). The VPU core can include a digital signal processor, such as, for example, a single instruction multiple data (SIMD), very long instruction word (VLIW) digital signal processor. The combination of SIMD and VLIW can enhance throughput and rate.

[0146] Each of the vector processors may include an instruction cache and may be coupled to dedicated memory. As a result, in some examples, each of the vector processors may be configured to execute independently of the other vector processors. In other examples, the vector processors included in a particular PVA may be configured to employ data parallelization. For example, in some embodiments, multiple vector processors included in a single PVA may execute the same computer vision algorithm, but on different regions of an image. In other examples, the vector processors included in a particular PVA may execute different computer vision algorithms simultaneously on the same image, or even on sequential images or portions of an image. Among other things, any number of PVAs may be included in a hardware acceleration cluster, and any number of vector processors may be included in each of these PVAs. Additionally, a PVA may include additional error correction code (ECC) memory to enhance overall system security.

[0147] The accelerator 714 (e.g., a hardware acceleration cluster) may include an on-chip computer vision network and SRAM to provide high-bandwidth, low-latency SRAM for the accelerator 714. In some examples, the on-chip memory may include at least 4MB SRAM consisting of, for example and without limitation, eight field-configurable memory blocks, which may be accessed by both the PVA and the DLA. Each pair of memory blocks may include an advanced peripheral bus (APB) interface, configuration circuitry, a controller, and a multiplexer. Any type of memory may be used. The PVA and the DLA may access the memory via a backbone that provides high-speed memory access to the PVA and the DLA. The backbone may include, for example using APB, an on-chip computer vision network that interconnects the PVA and the DLA to the memory.

[0148] The on-chip computer vision network may include an interface that determines that both the PVA and the DLA provide ready and valid signals before transmitting any control signals / address / data. Such an interface may provide separate phases and separate channels for transmitting control signals / address / data, as well as burst communication for continuous data transfer. This type of interface may conform to the ISO 26262 or IEC 61508 standards, but other standards and protocols may also be used.

[0149] In some examples, SoC 704 may include, for example, a real-time ray tracing hardware accelerator as described in U.S. Patent Application No. 16 / 101,232, filed on August 10, 2018. The real-time ray tracing hardware accelerator can be used to quickly and efficiently determine the position and extent of objects (e.g., within a world model) in order to generate a real-time visualization simulation for RADAR signal interpretation, for sound propagation synthesis and / or analysis, for SONAR system simulation, for general wave propagation simulation, for comparison with LiDAR data for positioning and / or other functional purposes, and / or for other uses. In some embodiments, one or more tree traversal units (TTUs) may be used to perform one or more ray tracing-related operations.

[0150] Accelerator 714 (e.g., a hardware accelerator cluster) has a wide range of autonomous driving uses. The PVA can be a programmable vision accelerator that can be used in key processing stages in ADAS and autonomous vehicles. The capabilities of the PVA are a good match for algorithm domains that require predictable processing, low power, and low latency. In other words, the PVA performs well on semi-dense or dense regular computations, even on small data sets that require predictable runtimes with low latency and low power. Thus, in the context of a platform for autonomous vehicles, the PVA is designed to run classical computer vision algorithms because they are effective in object detection and integer math operations.

[0151] For example, according to one embodiment of the technology, the PVA is used to perform computer stereo vision. In some examples, an algorithm based on semi-global matching may be used, but this is not intended to be limiting. Many applications for level 3 - 5 autonomous driving require instant motion estimation / stereo matching (e.g., structure from motion, pedestrian recognition, lane detection, etc.). The PVA can perform computer stereo vision functions on inputs from two monocular cameras.

[0152] In some examples, the PVA can be used to perform dense optical flow. Process raw RADAR data (e.g., using a 4D fast Fourier transform) to provide processed RADAR. In other examples, the PVA is used for time-of-flight depth processing, which, for example, processes raw time-of-flight data to provide processed time-of-flight data.

[0153] DLA can be used to run any type of network to enhance control and driving safety, including, for example, a neural network that outputs a confidence metric for each object detection. Such confidence values can be interpreted as probabilities or as providing a relative "weight" of each detection compared to other detections. The confidence value enables the system to make further decisions about which detections should be considered true positive detections rather than false positive detections. For example, the system can set a threshold for the confidence and consider only detections that exceed the threshold as true positive detections. In an automatic emergency braking (AEB) system, false positive detections can cause the vehicle to automatically perform emergency braking, which is clearly undesirable. Therefore, only the most confident detections should be considered as triggers for AEB. DLA can run a neural network for regressing confidence values. The neural network can take as its input at least some subset of parameters, such as bounding box dimensions, a ground plane estimate obtained (e.g., from another subsystem), the output of an inertial measurement unit (IMU) sensor 766 related to the vehicle 700 orientation, distance, a 3D position estimate of an object obtained from a neural network and / or other sensors (such as a LiDAR sensor 764 or a RADAR sensor 760), etc.

[0154] The SoC 704 can include one or more data stores 716 (e.g., memory). The data store 716 can be on-chip memory of the SoC 704, which can store neural networks to be executed on the GPU and / or DLA. In some examples, for redundancy and safety, the data store 716 can be large enough in capacity to store multiple instances of the neural network. The data store 716 can include an L2 or L3 cache 712. References to the data store 716 can include references to memory associated with PVAs, DLAs, and / or other accelerators 714 as described herein.

[0155] The SoC 704 may include one or more processors 710 (e.g., embedded processors). The processor 710 may include a boot and power management processor, which may be a dedicated processor and subsystem for handling boot power and management functions as well as security implementation related. The boot and power management processor may be part of the SoC 704 boot sequence and may provide runtime power management services. The boot power and management processor may provide clock and voltage programming, auxiliary system low-power state transitions, SoC 704 heat and temperature sensor management, and / or SoC 704 power state management. Each temperature sensor may be implemented as a ring oscillator whose output frequency is proportional to temperature, and the SoC 704 may use the ring oscillator to detect the temperature of the CPU 706, GPU 708, and / or accelerator 714. If it is determined that the temperature exceeds a threshold, then the boot and power management processor may enter a temperature fault routine and place the SoC 704 in a lower power state and / or place the vehicle 700 in a driver safety stop mode (e.g., safely stop the vehicle 700).

[0156] The processor 710 may further include a set of embedded processors that may serve as an audio processing engine. The audio processing engine may be an audio subsystem that allows for full hardware support for multi-channel audio over multiple interfaces and a wide and flexible range of audio I / O interfaces. In some examples, the audio processing engine is a dedicated processor core with a digital signal processor with dedicated RAM.

[0157] The processor 710 may further include an always-on processor engine, which may provide the necessary hardware features to support low-power sensor management and wake-up use cases. The always-on processor engine may include a processor core, tightly coupled RAM, support peripherals (e.g., timers and interrupt controllers), various I / O controller peripherals, and routing logic.

[0158] The processor 710 may further include a security cluster engine, which includes a dedicated processor subsystem for handling security management of automotive applications. The security cluster engine may include two or more processor cores, tightly coupled RAM, support peripherals (e.g., timers, interrupt controllers, etc.), and / or routing logic. In the security mode, the two or more cores may operate in a lockstep mode and act as a single core with comparison logic for detecting any differences between their operations.

[0159] The processor 710 may further include a real-time camera engine, which may include a dedicated processor subsystem for handling real-time camera management.

[0160] The processor 710 may further include a high dynamic range signal processor, which may include an image signal processor, which is a hardware engine that is part of the camera processing pipeline.

[0161] The processor 710 may include a video image compositor that may be a processing block (e.g., implemented on a microprocessor) that implements the video post-processing functions required for a video playback application to generate the final image for the player window. The video image compositor may perform lens distortion correction on the wide-angle camera 770, the surround camera 774, and / or the in-cab monitoring camera sensor. The in-cab monitoring camera sensor is preferably monitored by a neural network running on another instance of the advanced SoC, configured to identify in-cab events and respond accordingly. The in-cab system may perform lip reading to activate mobile phone services and make calls, dictate emails, change the vehicle destination, activate or change the vehicle's infotainment system and settings, or provide voice-activated web surfing. Certain functions are only available to the driver when the vehicle is operating in autonomous mode and are disabled otherwise.

[0162] The video image compositor may include enhanced temporal noise reduction for spatial and temporal noise reduction. For example, in the case where there is motion in the video, the noise reduction appropriately weights the spatial information, reducing the weight of the information provided by neighboring frames. In the case where the image or a portion of the image does not include motion, the temporal noise reduction performed by the video image compositor may use information from a previous image to reduce the noise in the current image.

[0163] The video image compositor may also be configured to perform stereo correction on input stereo lens frames. When the operating system desktop is in use and the GPU 708 does not need to continuously render new surfaces, the video image compositor may further be used for user interface composition. Even when the GPU 708 is powered on and active for 3D rendering, the video image compositor may be used to relieve the burden on the GPU 708 to improve performance and responsiveness.

[0164] The SoC 704 may further include a Mobile Industry Processor Interface (MIPI) camera serial interface, a high-speed interface, and / or a video input block for receiving video and inputs from cameras and may be used for camera and related pixel input functions. The SoC 704 may further include an input / output controller that may be software-controlled and may be used to receive I / O signals not committed to a specific role.

[0165] The SoC 704 may further include a wide range of peripheral device interfaces to enable communication with peripheral devices, audio codecs, power management, and / or other devices. The SoC 704 may be used to process data from cameras (connected via Gigabit Multimedia Serial Link and Ethernet), sensors (such as LiDAR sensor 764, RADAR sensor 760, etc. that may be connected via Ethernet), data from bus 702 (such as the speed of vehicle 700, steering wheel position, etc.), and data from GNSS sensor 758 (connected via Ethernet or CAN bus). The SoC 704 may further include dedicated high-performance large-capacity storage controllers, which may include their own DMA engines and which may be used to free the CPU 706 from routine data management tasks.

[0166] The SoC 704 may be an end-to-end platform with a flexible architecture that spans automation levels 3 - 5, thus providing an integrated functional safety architecture for a platform that leverages and efficiently uses computer vision and ADAS technologies to achieve diversity and redundancy, along with deep learning tools to provide a flexible and reliable driving software stack. The SoC 704 may be faster, more reliable, and even more energy-efficient and space-efficient than conventional systems. For example, when combined with the CPU 706, GPU 708, and data storage 716, the accelerator 714 may provide a fast and efficient platform for level 3 - 5 autonomous vehicles.

[0167] Thus, this technology provides capabilities and functions that cannot be achieved by conventional systems. For example, computer vision algorithms may be executed on CPUs that may be configured using high-level programming languages such as the C programming language to perform various processing algorithms across a variety of visual data. However, CPUs often cannot meet the performance requirements of many computer vision applications, such as those related to, for example, execution time and power consumption. In particular, many CPUs cannot execute complex object detection algorithms in real time, which is a requirement for in-vehicle ADAS applications and for practical level 3 - 5 autonomous vehicles.

[0168] In contrast to conventional systems, the techniques described herein allow multiple neural networks to be executed simultaneously and / or sequentially by providing a CPU complex, a GPU complex, and a hardware acceleration cluster, and combining the results to achieve level 3-5 autonomous driving capabilities. For example, a CNN executed on a DLA or a dGPU (e.g., GPU 720) can include text and word recognition, allowing a supercomputer to read and understand traffic signs, including signs for which the neural network has not been specifically trained. The DLA can further include a neural network capable of recognizing, interpreting, and providing semantic understanding of the sign, and passing that semantic understanding to a path planning module running on the CPU complex. The DLA can also utilize metrics associated with sensor performance as inputs to one or more neural networks.

[0169] As another example, as required for level 3, 4, or 5 driving, multiple neural networks can run simultaneously. For example, a warning sign consisting of "Caution: Flashing lights indicate icy conditions" along with the lights can be interpreted independently or jointly by several neural networks. The sign itself can be recognized as a traffic sign by a first neural network deployed (e.g., a trained neural network), the text "Flashing lights indicate icy conditions" can be interpreted by a second neural network deployed, which informs the vehicle's path planning software (preferably executed on the CPU complex) that when the flashing lights are detected, there are icy conditions. The flashing lights can be recognized by operating a third neural network deployed over multiple frames, which informs the vehicle's path planning software of the presence (or absence) of the flashing lights. All three neural networks can run simultaneously, for example, within the DLA and / or on the GPU 708.

[0170] In some examples, a CNN for face recognition and owner recognition can use data from a camera sensor to recognize the presence of an authorized driver and / or owner of the vehicle 700. A processing engine always on the sensor can be used to unlock the vehicle and turn on the lights when the owner approaches the driver's door, and in a secure mode, to disable the vehicle when the owner leaves the vehicle. In this way, the SoC 704 provides security against theft and / or carjacking.

[0171] In another example, the CNN for emergency vehicle detection and recognition can use data from microphone 796 to detect and recognize emergency vehicle sirens. In contrast to conventional systems that use a general classifier to detect sirens and manually extract features, SoC 704 uses a CNN to classify environmental and city sounds as well as visual data. In a preferred embodiment, the CNN running on the DLA is trained to recognize the relative closing rate of an emergency vehicle (e.g., by using the Doppler effect). The CNN can also be trained to recognize emergency vehicles specific to the local area in which the vehicle operates as identified by the GNSS sensor 758. Thus, for example, when operating in Europe, the CNN will seek to detect European sirens, and when in the United States, the CNN will seek to recognize only North American sirens. Once an emergency vehicle is detected, with the assistance of ultrasonic sensor 762, a control program can be used to execute emergency vehicle safety routines to slow the vehicle, pull over to the side of the road, stop the vehicle, and / or idle the vehicle until the emergency vehicle passes.

[0172] The vehicle can include a CPU 718 (e.g., a discrete CPU or dCPU) that can be coupled to the SoC 704 via a high-speed interconnect (e.g., PCIe). The CPU 718 can include, for example, an X86 processor. The CPU 718 can be used to perform any of a variety of functions, including, for example, arbitrating potentially inconsistent results between the ADAS sensors and the SoC 704, and / or monitoring the status and health of the controller 736 and / or the infotainment SoC 730.

[0173] The vehicle 700 can include a GPU 720 (e.g., a discrete GPU or dGPU) that can be coupled to the SoC 704 via a high-speed interconnect (e.g., NVIDIA's NVLINK). The GPU 720 can provide additional artificial intelligence capabilities, for example, by executing redundant and / or different neural networks, and can be used to train and / or update neural networks based on inputs from the sensors of the vehicle 700 (e.g., sensor data).

[0174] Vehicle 700 may further include a network interface 724, which may include one or more wireless antennas 726 (e.g., one or more wireless antennas for different communication protocols, such as cellular antennas, Bluetooth antennas, etc.). The network interface 724 can be used to enable wireless connections via the Internet to the cloud (e.g., to server 778 and / or other network devices), to other vehicles, and / or to computing devices (e.g., the passenger's client device). To communicate with other vehicles, a direct link can be established between the two vehicles, and / or an indirect link can be established (e.g., across a network and via the Internet). The direct link can be provided using a vehicle-to-vehicle communication link. The vehicle-to-vehicle communication link can provide vehicle 700 with information about vehicles approaching vehicle 700 (e.g., vehicles in front of, to the side of, and / or behind vehicle 700). This functionality can be part of the cooperative adaptive cruise control function of vehicle 700.

[0175] The network interface 724 may include a SoC that provides modulation and demodulation functions and enables the controller 736 to communicate via a wireless network. The network interface 724 may include a radio frequency front end for upconverting from baseband to radio frequency and downconverting from radio frequency to baseband. The frequency conversion can be performed by a known process and / or can be performed using a super-heterodyne process. In some examples, the radio frequency front end functionality can be provided by a separate chip. The network interface may include wireless functionality for communicating via LTE, WCDMA, UMTS, GSM, CDMA2000, Bluetooth, Bluetooth LE, Wi-Fi, Z-Wave, ZigBee, LoRaWAN, and / or other wireless protocols.

[0176] Vehicle 700 may further include a data store 728 that may include off-chip (e.g., outside of SoC 704) storage. The data store 728 may include one or more storage elements, including RAM, SRAM, DRAM, VRAM, flash memory, hard drives, and / or other components and / or devices that can store at least one bit of data.

[0177] Vehicle 700 may further include a GNSS sensor 758. The GNSS sensor 758 (e.g., GPS, assisted GPS sensor, differential GPS (DGPS) sensor, etc.) is used to assist mapping, perception, occupancy grid generation, and / or path planning functions. Any number of GNSS sensors 758 can be used, including, for example and without limitation, a GPS using a USB connector with an Ethernet-to-serial (RS-232) bridge.

[0178] Vehicle 700 may further include a RADAR sensor 760. The RADAR sensor 760 can be used by vehicle 700 for remote vehicle detection even in dark and / or adverse weather conditions. The RADAR functional safety level can be ASIL B. The RADAR sensor 760 can use CAN and / or bus 702 (e.g., to transmit data generated by the RADAR sensor 760) for control as well as access to object tracking data, and in some examples access Ethernet to access raw data. A variety of RADAR sensor types can be used. For example and without limitation, the RADAR sensor 760 can be suitable for front, rear, and side RADAR use. In some examples, a pulsed Doppler RADAR sensor is used.

[0179] The RADAR sensor 760 can include different configurations, such as long-range with a narrow field of view, short-range with a wide field of view, short-range side coverage, etc. In some examples, long-range RADAR can be used for adaptive cruise control functions. The long-range RADAR system can provide a wide field of view (e.g., within 250m) achieved through two or more independent scans. The RADAR sensor 760 can help distinguish between static and moving objects and can be used by the ADAS system for emergency braking assistance and forward collision warning. The long-range RADAR sensor can include a single-station multi-mode RADAR with multiple (e.g., six or more) fixed RADAR antennas and high-speed CAN and FlexRay interfaces. In an example with six antennas, the central four antennas can create a focused beam pattern designed to record the surroundings of vehicle 700 with minimal traffic interference from adjacent lanes at a higher rate. The other two antennas can expand the field of view, making it possible to quickly detect vehicles entering or leaving the lane of vehicle 700.

[0180] As an example, a mid-range RADAR system can include a range of up to 760m (front) or 80m (rear) and a field of view of up to 42 degrees (front) or 750 degrees (rear). The short-range RADAR system can include, but is not limited to, RADAR sensors designed to be mounted at both ends of the rear bumper. When mounted at both ends of the rear bumper, such a RADAR sensor system can create two beams that continuously monitor the rear and the blind spots beside the vehicle.

[0181] The short-range RADAR system can be used in the ADAS system for blind spot detection and / or lane change assistance.

[0182] Vehicle 700 may further include an ultrasonic sensor 762. The ultrasonic sensor 762, which may be placed in front of, behind, and / or on the sides of the vehicle 700, may be used for parking assistance and / or creating and updating an occupancy grid. A variety of ultrasonic sensors 762 may be used, and different ultrasonic sensors 762 may be used for different detection ranges (e.g., 2.5 m, 4 m). The ultrasonic sensor 762 may operate at ASIL B of the functional safety level.

[0183] Vehicle 700 may include a LiDAR sensor 764. The LiDAR sensor 764 may be used for object and pedestrian detection, emergency braking, collision avoidance, and / or other functions. The LiDAR sensor 764 may be at ASIL B of the functional safety level. In some examples, vehicle 700 may include multiple LiDAR sensors 764 (e.g., two, four, six, etc.) that may use Ethernet (e.g., to provide data to a gigabit Ethernet switch).

[0184] In some examples, the LiDAR sensor 764 may be capable of providing a list of objects and their distances for a 360-degree field of view. Commercially available LiDAR sensors 764 may have, for example, an advertised range of approximately 700 m, an accuracy of 2 cm - 3 cm, and support for a 700 Mbps Ethernet connection. In some examples, one or more non-protruding LiDAR sensors 764 may be used. In such examples, the LiDAR sensor 764 may be implemented as a small device that may be embedded in the front, behind, on the sides, and / or corners of the vehicle 700. In such examples, the LiDAR sensor 764 may provide a field of view of up to 120 degrees horizontally and 35 degrees vertically even for low-reflectivity objects, with a range of 200 m. The front-mounted LiDAR sensor 764 may be configured for a horizontal field of view between 45 degrees and 135 degrees.

[0185] In some examples, LiDAR technologies such as 3D flash LiDAR can also be used. 3D flash LiDAR uses the flash of a laser as the emission source to illuminate the vehicle's surrounding environment up to about 200 m. The flash LiDAR unit includes a receiver that records the laser pulse transit time and the reflected light on each pixel, which in turn corresponds to the range from the vehicle to the object. Flash LiDAR can allow for the generation of highly accurate and distortion-free images of the surrounding environment using each laser flash. In some examples, four flash LiDAR sensors can be deployed, one on each side of the vehicle 700. Available 3D flash LiDAR systems include solid-state 3D staring array LiDAR cameras (e.g., non-scanning LiDAR devices) that have no moving parts other than a fan. The flash LiDAR device can use class I (eye-safe) laser pulses of 7 nanoseconds per frame and can capture the reflected laser in the form of 3D range point clouds and co-registered intensity data. By using flash LiDAR and because flash LiDAR is a solid-state device with no moving parts, the LiDAR sensor 764 can be less susceptible to motion blur, vibration, and / or shock.

[0186] The vehicle can further include an IMU sensor 766. In some examples, the IMU sensor 766 can be located at the center of the rear axle of the vehicle 700. The IMU sensor 766 can include, for example and without limitation, an accelerometer, a magnetometer, a gyroscope, a magnetic compass, and / or other sensor types. In some examples, such as in a six-axis application, the IMU sensor 766 can include an accelerometer and a gyroscope, while in a nine-axis application, the IMU sensor 766 can include an accelerometer, a gyroscope, and a magnetometer.

[0187] In some embodiments, the IMU sensor 766 can be implemented as a miniature high-performance GPS-aided inertial navigation system (GPS / INS) that combines microelectromechanical systems (MEMS) inertial sensors, a high-sensitivity GPS receiver, and an advanced Kalman filtering algorithm to provide estimates of position, velocity, and attitude. Thus, in some examples, the IMU sensor 766 can enable the vehicle 700 to estimate the heading without input from a magnetic sensor by directly observing the change in velocity from the GPS to the IMU sensor 766 and correlating it. In some examples, the IMU sensor 766 and the GNSS sensor 758 can be integrated into a single unit.

[0188] The vehicle can include a microphone 796 disposed in and / or around the vehicle 700. Among other things, the microphone 796 can be used for emergency vehicle detection and identification.

[0189] The vehicle can further include any number of camera types, including a stereo camera 768, a wide-angle camera 770, an infrared camera 772, a surround camera 774, a long-range and / or mid-range camera 798, and / or other camera types. These cameras can be used to capture image data around the entire periphery of the vehicle 700. The camera types used depend on the embodiment and the requirements of the vehicle 700, and any combination of camera types can be used to provide the necessary coverage around the vehicle 700. Additionally, the number of cameras can vary according to the embodiment. For example, the vehicle can include six cameras, seven cameras, ten cameras, twelve cameras, and / or another number of cameras. As an example and without limitation, these cameras can support Gigabit Multimedia Serial Link (GMSL) and / or Gigabit Ethernet. Each of the cameras is described in more detail herein with respect to Figure 7A and Figure 7B is described in more detail.

[0190] The vehicle 700 can further include a vibration sensor 742. The vibration sensor 742 can measure the vibration of components of the vehicle such as an axle. For example, a change in vibration can indicate a change in the road surface. In another example, when two or more vibration sensors 742 are used, the difference between the vibrations can be used to determine the friction or slip of the road surface (e.g., when there is a vibration difference between a powered drive axle and a free-rotating axle).

[0191] The vehicle 700 can include an ADAS system 738. In some examples, the ADAS system 738 can include a SoC. The ADAS system 738 can include autonomous / adaptive / automated cruise control (ACC), cooperative adaptive cruise control (CACC), forward collision warning (FCW), automatic emergency braking (AEB), lane departure warning (LDW), lane keeping assist (LKA), blind spot warning (BSW), rear cross traffic warning (RCTW), collision warning system (CWS), lane centering (LC), and / or other features and functions.

[0192] The ACC system can use RADAR sensors 760, LiDAR sensors 764, and / or cameras. The ACC system can include longitudinal ACC and / or lateral ACC. Longitudinal ACC monitors and controls the distance to the vehicle immediately in front of the vehicle 700 and automatically adjusts the vehicle speed to maintain a safe distance from the vehicle ahead. Lateral ACC performs distance keeping and, when necessary, advises the vehicle 700 to change lanes. Lateral ACC is related to other ADAS applications such as LCA and CWS.

[0193] The CACC uses information from other vehicles, which can be received indirectly from other vehicles via the network interface 724 and / or the wireless antenna 726 via a wireless link or through a network connection (e.g., via the Internet). The direct link can be provided by a vehicle-to-vehicle (V2V) communication link, while the indirect link can be an infrastructure-to-vehicle (I2V) communication link. Generally, the V2V communication concept provides information about the immediately preceding vehicle (e.g., a vehicle directly in front of vehicle 700 and in the same lane), while the I2V communication concept provides information about traffic further ahead. The CACC system can include either or both of the I2V and V2V information sources. Given information about the vehicle in front of vehicle 700, the CACC can be more reliable, and it has the potential to improve the smoothness of traffic flow and reduce road congestion.

[0194] The FCW system is designed to alert the driver of a hazard so that the driver can take corrective action. The FCW system uses a front camera and / or RADAR sensor 760 coupled to a dedicated processor, DSP, FPGA, and / or ASIC, which is electrically coupled to driver feedback such as a display, speaker, and / or vibration component. The FCW system can provide warnings in the form of, for example, sound, visual warnings, vibration, and / or a rapid braking pulse.

[0195] The AEB system detects an impending front collision with another vehicle or other object and can automatically apply the brakes if the driver does not take corrective action within a specified time or distance parameter. The AEB system can use a front camera and / or RADAR sensor 760 coupled to a dedicated processor, DSP, FPGA, and / or ASIC. When the AEB system detects a hazard, it typically first alerts the driver to take corrective action to avoid the collision, and if the driver does not take corrective action, then the AEB system can automatically apply the brakes in an effort to prevent or at least mitigate the impact of the predicted collision. The AEB system can include technologies such as dynamic brake support and / or collision imminent braking.

[0196] The LDW system provides visual, auditory, and / or tactile warnings such as steering wheel or seat vibration to alert the driver when vehicle 700 crosses a lane marking. The LDW system is not activated when the driver indicates an intentional lane departure by activating the turn signal. The LDW system can use a front-side facing camera coupled to a dedicated processor, DSP, FPGA, and / or ASIC, which is electrically coupled to driver feedback such as a display, speaker, and / or vibration component.

[0197] The LKA system is a variant of the LDW system. If the vehicle 700 starts to leave the lane, then the LKA system provides a steering input or braking to correct the vehicle 700.

[0198] The BSW system detects and warns the driver of vehicles in the blind spot of the vehicle. The BSW system can provide visual, audible, and / or tactile alerts to indicate that merging or changing lanes is unsafe. The system can provide additional warnings when the driver uses the turn signal. The BSW system can use a rear-facing camera and / or RADAR sensor 760 coupled to a dedicated processor, DSP, FPGA, and / or ASIC, which is electrically coupled to driver feedback such as a display, speaker, and / or vibrating component.

[0199] The RCTW system can provide visual, audible, and / or tactile notifications when an object is detected outside the rear camera range while the vehicle 700 is in reverse. Some RCTW systems include AEB to ensure that the vehicle brakes are applied to avoid a crash. The RCTW system can use one or more rear RADAR sensors 760 coupled to a dedicated processor, DSP, FPGA, and / or ASIC, which is electrically coupled to driver feedback such as a display, speaker, and / or vibrating component.

[0200] Conventional ADAS systems may be prone to false positive results, which can be annoying and distracting to the driver, but are typically not catastrophic because the ADAS system alerts the driver and allows the driver to decide whether a safe condition truly exists and act accordingly. However, in an autonomous vehicle 700, in the case of conflicting results, the vehicle 700 itself must decide whether to heed the results from the main computer or an auxiliary computer (e.g., the first controller 736 or the second controller 736). For example, in some embodiments, the ADAS system 738 can be a standby and / or auxiliary computer for providing perception information to a standby computer rationality module. The standby computer rationality monitor can run redundant and diverse software on hardware components to detect faults in perception and dynamic driving tasks. The output from the ADAS system 738 can be provided to the supervisory MCU. If the outputs from the main computer and the auxiliary computer conflict, then the supervisory MCU must determine how to reconcile the conflict to ensure safe operation.

[0201] In some examples, the host computer may be configured to provide a confidence score to the supervisory MCU indicating the host computer's confidence in the selected result. If the confidence score exceeds a threshold, then the supervisory MCU may follow the direction of the host computer regardless of whether the secondary computer provides conflicting or inconsistent results. In cases where the confidence score does not meet the threshold and where the host computer and the secondary computer indicate different results (e.g., conflict), the supervisory MCU may arbitrate between these computers to determine an appropriate result.

[0202] The supervisory MCU may be configured to run a neural network that is trained and configured to determine, based on the outputs from the host computer and the secondary computer, the conditions under which the secondary computer provides a false alarm. Thus, the neural network in the supervisory MCU can learn when the output of the secondary computer can be trusted and when it cannot. For example, when the secondary computer is a RADAR-based FCW system, the neural network in the supervisory MCU can learn when the FCW system is identifying a metallic object that is not in fact dangerous, such as a drainage grate or manhole cover that triggers an alarm. Similarly, when the secondary computer is a camera-based LDW system, the neural network in the supervisory MCU can learn to disregard the LDW when a cyclist or pedestrian is present and lane departure is actually the safest strategy. In embodiments that include a neural network running on the supervisory MCU, the supervisory MCU may include at least one of a DLA or a GPU suitable for running the neural network with associated memory. In a preferred embodiment, the supervisory MCU may include components of the SoC 704 and / or be included as components of the SoC 704.

[0203] In other examples, the ADAS system 738 may include a secondary computer that performs ADAS functions using traditional computer vision rules. As such, the secondary computer may use classical computer vision rules (if - then), and the presence of a neural network in the supervisory MCU can improve reliability, safety, and performance. For example, the diverse implementations and intentional non-identity make the overall system more fault-tolerant, especially for failures caused by software (or software-hardware interface) functions. For example, if there is a software vulnerability or error in the software running on the host computer and the non-identical software code running on the secondary computer provides the same overall result, then the supervisory MCU can be more confident that the overall result is correct and that the vulnerability in the software or hardware on the host computer does not cause a substantial error.

[0204] In some examples, the output of the ADAS system 738 can be fed to the perception block of the main computer and / or the dynamic driving task block of the main computer. For example, if the ADAS system 738 indicates a forward collision warning due to an object being immediately in front, the perception block can use this information when identifying the object. In other examples, the secondary computer can have its own neural network, which is trained and thus reduces the risk of false positives as described herein.

[0205] The vehicle 700 can further include an infotainment SoC 730 (e.g., an in-vehicle infotainment system (IVI)). Although illustrated and described as an SoC, the infotainment system can not be an SoC and can include two or more discrete components. The infotainment SoC 730 can include a combination of hardware and software that can be used to provide audio (e.g., music, personal digital assistant, navigation instructions, news, radio, etc.), video (e.g., TV, movies, streaming, etc.), telephone (e.g., hands-free calling), network connectivity (e.g., LTE, Wi-Fi, etc.), and / or information services (e.g., navigation system, rear parking assistance, radio data system, vehicle-related information such as fuel level, total distance covered, brake fuel level, oil level, door open / close, air filter information, etc.) to the vehicle 700. For example, the infotainment SoC 730 can include a radio, a disc player, a navigation system, a video player, USB and Bluetooth connectivity, an in-vehicle computer, in-vehicle entertainment, Wi-Fi, steering wheel audio controls, hands-free voice controls, a head-up display (HUD), an HMI display 734, a telematics device, a control panel (e.g., for controlling various components, features, and / or systems, and / or interacting therewith), and / or other components. The infotainment SoC 730 can further be used to provide information (e.g., visual and / or auditory) to the user of the vehicle, such as information from the ADAS system 738, autonomous driving information such as planned vehicle maneuvers, trajectories, surrounding environment information (e.g., intersection information, vehicle information, road information, etc.), and / or other information.

[0206] The infotainment SoC 730 can include GPU functionality. The infotainment SoC 730 can communicate with other devices, systems, and / or components of the vehicle 700 via a bus 702 (e.g., a CAN bus, Ethernet, etc.). In some examples, the infotainment SoC 730 can be coupled to a supervisory MCU such that in the event of a failure of the main controller 736 (e.g., the main and / or backup computer of the vehicle 700), the GPU of the infotainment system can perform some autonomous driving functions. In such examples, the infotainment SoC 730 can place the vehicle 700 in a driver safe parking mode as described herein.

[0207] Vehicle 700 may further include an instrument cluster 732 (such as a digital dashboard, an electronic instrument cluster, a digital instrument panel, etc.). The instrument cluster 732 may include a controller and / or a supercomputer (such as a discrete controller or supercomputer). The instrument cluster 732 may include a set of instruments, such as a speedometer, fuel level, oil pressure, tachometer, odometer, turn indicator, shift position indicator, seat belt warning light, parking brake warning light, engine fault light, airbag (SRS) system information, lighting controls, safety system controls, navigation information, etc. In some examples, information may be displayed and / or shared between the infotainment SoC 730 and the instrument cluster 732. In other words, the instrument cluster 732 may be included as part of the infotainment SoC 730, or vice versa.

[0208] Figure 7D A system schematic diagram of communication between a cloud-based server and Figure 7A an exemplary autonomous vehicle 700 according to some embodiments of the present disclosure. System 776 may include a server 778, a network 790, and vehicles including vehicle 700. The server 778 may include multiple GPUs 784(A)-784(H) (collectively referred to herein as GPUs 784), PCIe switches 782(A)-782(H) (collectively referred to herein as PCIe switches 782), and / or CPUs 780(A)-780(B) (collectively referred to herein as CPUs 780). The GPUs 784, CPUs 780, and PCIe switches may be interconnected by high-speed interconnects such as, for example, and without limitation, the NVLink interface 788 developed by NVIDIA and / or PCIe connections 786. In some examples, the GPUs 784 are connected via NVLink and / or an NVSwitch SoC, and the GPUs 784 and the PCIe switches 782 are connected via a PCIe interconnect. Although eight GPUs 784, two CPUs 780, and two PCIe switches are illustrated, this is not intended to be limiting. Depending on the embodiment, each of the servers 778 may include any number of GPUs 784, CPUs 780, and / or PCIe switches. For example, each of the servers 778 may include eight, sixteen, thirty-two, and / or more GPUs 784.

[0209] Server 778 can receive image data via network 790 and from a vehicle, the image data representing an image showing an unexpected or changed road condition such as a recently started road work. Server 778 can transmit neural network 792, updated neural network 792, and / or map information 794 via network 790 and to the vehicle, including information about traffic and road conditions. Updates to map information 794 can include updates to HD map 722, such as information about construction sites, potholes, curves, floods, or other obstacles. In some examples, neural network 792, updated neural network 792, and / or map information 794 can be represented and / or generated based on data received from new training and / or from any number of vehicles in the environment and / or experience of training performed at a data center (e.g., using server 778 and / or other servers).

[0210] Server 778 can be used to train a machine learning model (e.g., a neural network) based on training data. The training data can be generated by a vehicle and / or can be generated in a simulation (e.g., using a game engine). In some examples, the training data is labeled (e.g., in cases where the neural network benefits from supervised learning) and / or undergoes other preprocessing, while in other examples, the training data is not labeled and / or preprocessed (e.g., in cases where the neural network does not require supervised learning). The training can be performed according to any one or more classes of machine learning techniques, including but not limited to the following classes: supervised training, semi-supervised training, unsupervised training, self-learning, reinforcement learning, federated learning, transfer learning, feature learning (including principal component and clustering analysis), multilinear subspace learning, manifold learning, representation learning (including alternate dictionary learning), rule-based machine learning, anomaly detection, and any variants or combinations thereof. Once the machine learning model is trained, the machine learning model can be used by the vehicle (e.g., transmitted to the vehicle via network 790), and / or the machine learning model can be used by server 778 to remotely monitor the vehicle.

[0211] In some examples, server 778 can receive data from a vehicle and apply the data to the latest real-time neural network for real-time intelligent inference. Server 778 can include a deep learning supercomputer powered by GPU 784 and / or a dedicated AI computer, such as DGX and DGX Station machines developed by NVIDIA. However, in some examples, server 778 can include a deep learning infrastructure of a data center powered only by a CPU.

[0212] The deep learning infrastructure of server 778 may be capable of fast real-time inference and can use this ability to evaluate and verify the health of the processors, software, and / or associated hardware in vehicle 700. For example, the deep learning infrastructure can receive periodic updates from vehicle 700, such as an image sequence and / or objects located in that image sequence that vehicle 700 has identified (e.g., via computer vision and / or other machine learning object classification techniques). The deep learning infrastructure can run its own neural network to identify the objects and compare them with the objects identified by vehicle 700. If the results do not match and the infrastructure concludes that the AI in vehicle 700 has malfunctioned, then server 778 can transmit a signal to vehicle 700 instructing the fail-safe computer in vehicle 700 to take control, notify the passengers, and complete a safe parking operation.

[0213] For inference, server 778 can include a GPU 784 and one or more programmable inference accelerators (e.g., NVIDIA's TensorRT 3). The combination of a GPU-powered server and inference acceleration can enable real-time response. In other examples, such as when performance is less critical, a CPU, FPGA, and other processor-powered servers can be used for inference.

[0214] Example computing device

[0215] Figure 8 FIG. is a block diagram of an example computing device 800 suitable for implementing some embodiments of the present disclosure. Computing device 800 may include an interconnect system 802 that directly or indirectly couples the following devices: memory 804, one or more central processing units (CPUs) 806, one or more graphics processing units (GPUs) 808, a communication interface 810, input / output (I / O) ports 812, input / output components 814, a power supply 816, one or more presentation components 818 (e.g., a display), and one or more logic units 820. In at least one embodiment, computing device 800 may include one or more virtual machines (VMs), and / or any of its components may include virtual components (e.g., virtual hardware components). For non-limiting examples, one or more GPUs 808 may include one or more vGPUs, one or more CPUs 806 may include one or more vCPUs, and / or one or more logic units 820 may include one or more virtual logic units. Thus, computing device 800 may include discrete components (e.g., a complete GPU dedicated to computing device 800), virtual components (e.g., a portion of a GPU dedicated to computing device 800), or a combination thereof.

[0216] Although Figure 8The respective blocks are shown as being connected via an interconnect system 802 having circuitry, but this is not intended to be limiting and is for clarity only. For example, in some embodiments, a rendering component 818, such as a display device, may be considered an I / O component 814 (e.g., if the display is a touchscreen). As another example, the CPU 806 and / or GPU 808 may include memory (e.g., memory 804 may represent a storage device in addition to the memory of the GPU 808, CPU 806, and / or other components). In other words, Figure 8 the computing devices are merely illustrative. No distinction is made among categories such as “workstation,” “server,” “laptop,” “desktop,” “tablet,” “client device,” “mobile device,” “handheld device,” “gaming console,” “electronic control unit (ECU),” “virtual reality system,” and / or other device or system types because all of these are considered within the Figure 8 scope of the computing devices.

[0217] The interconnect system 802 may represent one or more links or buses, such as an address bus, a data bus, a control bus, or a combination thereof. The interconnect system 802 may include one or more types of links or buses, such as an Industry Standard Architecture (ISA) bus, an Extended Industry Standard Architecture (EISA) bus, a Video Electronics Standards Association (VESA) bus, a Peripheral Component Interconnect (PCI) bus, a Peripheral Component Interconnect Express (PCIe) bus, and / or another type of bus or link. In some embodiments, there are direct connections between components. As an example, the CPU 806 may be directly connected to the memory 804. Additionally, the CPU 806 may be directly connected to the GPU 808. In cases where there are direct or point-to-point connections between components, the interconnect system 802 may include a PCIe link to effectuate the connection. In these examples, a PCI bus need not be included in the computing device 800.

[0218] The memory 804 may include any of a variety of computer-readable media. Computer-readable media can be any available media that can be accessed by the computing device 800. Computer-readable media can include volatile and nonvolatile media as well as removable and non-removable media. By way of example and not limitation, computer-readable media may include computer storage media and communication media.

[0219] A computer storage medium can include volatile and non-volatile media and / or removable and non-removable media implemented in any method or technology for storing information such as computer-readable instructions, data structures, program modules, and / or other data types. For example, memory 804 can store computer-readable instructions (e.g., which represent programs and / or program elements such as an operating system). Computer storage media can include, but are not limited to, RAM, ROM, EEPROM, flash memory or other storage technologies, CD-ROM, digital versatile disks (DVDs) or other optical disk storage devices, magnetic tape cassettes, magnetic tape, magnetic disk storage devices or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by computing device 800. As used herein, a computer storage medium does not include the signal itself.

[0220] A computer storage medium can include computer-readable instructions, data structures, program modules, and / or other data types in a modulated data signal such as a carrier wave or other transmission mechanism, and includes any information conveyance medium. The term "modulated data signal" can refer to a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. By way of example and not limitation, a computer storage medium can include wired media such as a wired network or direct wired connection, and wireless media such as sound, RF, infrared, and other wireless media. Any of the foregoing combinations should also be included within the scope of computer-readable media.

[0221] The CPU 806 can be configured to execute at least some of the computer-readable instructions to control one or more components of computing device 800 to perform one or more of the methods and / or processes described herein. Each of the CPUs 806 can include one or more cores (e.g., one, two, four, eight, twenty-eight, seventy-two, etc.) capable of simultaneously processing a large number of software threads. The CPU 806 can include any type of processor and can include different types of processors depending on the type of computing device 800 being implemented (e.g., a processor with fewer cores for a mobile device and a processor with more cores for a server). For example, depending on the type of computing device 800, the processor can be an advanced RISC machine (ARM) processor implemented using reduced instruction set computing (RISC) or an x86 processor implemented using complex instruction set computing (CISC). In addition to one or more microprocessors or complementary coprocessors such as a math coprocessor, computing device 800 can also include one or more CPUs 806.

[0222] In addition to or instead of the CPU 806, the GPU 808 can also be configured to execute at least some computer-readable instructions to control one or more components of the computing device 800 to perform one or more of the methods and / or processes described herein. One or more GPUs 808 can be an integrated GPU (e.g., having one or more CPUs 806) and / or one or more GPUs 808 can be a discrete GPU. In an embodiment, one or more GPUs 808 can be a coprocessor of one or more CPUs 806. The computing device 800 can use the GPU 808 to render graphics (e.g., 3D graphics) or perform general computing. For example, the GPU 808 can be used for general-purpose computing on the GPU (GPGPU). The GPU 808 can include hundreds or thousands of cores capable of processing hundreds or thousands of software threads simultaneously. The GPU 808 can generate pixel data for an output image in response to a rendering command (e.g., a rendering command received from the CPU 806 via a host interface). The GPU 808 can include graphics memory such as display memory for storing pixel data or any other suitable data (e.g., GPGPU data). The display memory can be included as part of the memory 804. The GPU 808 can include two or more GPUs operating in parallel (e.g., via a link). The link can directly connect the GPUs (e.g., using NVLINK) or can connect the GPUs through a switch (e.g., using NVSwitch). When combined, each GPU 808 can generate pixel data or GPGPU data for different parts of the output or for different outputs (e.g., the first GPU for the first image and the second GPU for the second image). Each GPU can include its own memory or can share memory with other GPUs.

[0223] In addition to or instead of the CPU 806 and / or the GPU 808, the logic unit 820 can be configured to execute at least some computer-readable instructions to control one or more components of the computing device 800 to perform one or more of the methods and / or processes described herein. In an embodiment, the CPU 806, the GPU 808, and / or the logic unit 820 can discretely or jointly execute any combination of methods, processes, and / or portions thereof. One or more logic units 820 can be part of and / or integrated in one or more CPUs 806 and / or one or more GPUs 808, and / or one or more logic units 820 can be discrete components of the CPU 806 and / or the GPU 808 or otherwise external to them. In an embodiment, one or more logic units 820 can be a processor of one or more CPUs 806 and / or one or more GPUs 808.

[0224] Examples of the logic unit 820 include one or more processing cores and / or their components, such as data processing units (DPUs), tensor cores (TCs), tensor processing units (TPUs), pixel vision cores (PVCs), vision processing units (VPUs), graphics processing clusters (GPCs), texture processing clusters (TPCs), streaming multiprocessors (SMs), tree traversal units (TTUs), artificial intelligence accelerators (AIAs), deep learning accelerators (DLAs), arithmetic logic units (ALUs)), application-specific integrated circuits (ASICs), floating-point units (FPUs), input / output (I / O) elements, Peripheral Component Interconnect (PCI) or Peripheral Component Interconnect Express (PCIe) elements, etc.

[0225] In various embodiments, one or more CPUs 806, GPUs 808, and / or logic units 820 are configured to execute one or more instances of the sensor fault detector 122. The sensor output frames 216 generated by the sensor fault detector 122 can then be used by other components to perform additional processing, such as planning and control functions.

[0226] The communication interface 810 can include one or more receivers, transmitters, and / or transceivers that enable the computing device 800 to communicate with other computing devices via an electronic communication network, including wired and / or wireless communication. The communication interface 810 can include components and functions that enable communication through any of several different networks, such as wireless networks (e.g., Wi-Fi, Z-Wave, Bluetooth, Bluetooth LE, ZigBee, etc.), wired networks (e.g., communication via Ethernet or InfiniBand), low-power wide area networks (e.g., LoRaWAN, SigFox, etc.), and / or the Internet. In one or more embodiments, the logic unit 820 and / or the communication interface 810 can include one or more data processing units (DPUs) to directly transfer data received through the network and / or through the interconnect system 802 to one or more GPUs 808 (e.g., the memory in the GPUs 808).

[0227] The I / O port 812 can enable the computing device 800 to be logically coupled to other devices including I / O components 814, presentation components 818, and / or other components, some of which may be built into (e.g., integrated into) the computing device 800. Exemplary I / O components 814 include microphones, mice, keyboards, joysticks, game pads, game controllers, dish satellite antennas, scanners, printers, wireless devices, and so on. The I / O components 814 can provide a natural user interface (NUI) that processes user-generated air gestures, voice, or other physiological inputs. In some instances, the input can be transmitted to appropriate network elements for further processing. The NUI can implement any combination of speech recognition, stylus recognition, face recognition, biometric recognition, on-screen and near-screen gesture recognition, air gestures, head and eye tracking, and touch recognition associated with the display of the computing device 800 (described in more detail below). The computing device 800 can include depth cameras such as stereo camera systems, infrared camera systems, RGB camera systems, touch screen technologies, and combinations thereof for gesture detection and recognition. Additionally, the computing device 800 can include an accelerometer or gyroscope that enables motion detection (e.g., as part of an inertial measurement unit (IMU)). In some examples, the output of the accelerometer or gyroscope can be used by the computing device 800 to render immersive augmented reality or virtual reality.

[0228] The power supply 816 can include hardwired power, battery power, or a combination thereof. The power supply 816 can power the computing device 800 so that the components of the computing device 800 can operate.

[0229] The presentation component 818 can include a display (e.g., a monitor, touch screen, TV screen, head-up display (HUD), other display types, or a combination thereof), speakers, and / or other presentation components. The presentation component 818 can receive data from other components (e.g., GPU 808, CPU 806, DPU, etc.) and output the data (e.g., as images, videos, sounds, etc.).

[0230] Example data center

[0231] Figure 9 An example data center 900 is shown, which can be used in at least one embodiment of the present disclosure. The data center 900 can include a data center infrastructure layer 910, a framework layer 920, a software layer 930, and an application layer 940.

[0232] As Figure 9As shown, the data center infrastructure layer 910 may include a resource coordinator 912, grouped computing resources 914, and node computing resources ("node C.R.") 916(1)-916(N), where "N" represents any whole positive integer. In at least one embodiment, the node C.R. 916(1)-916(N) may include, but is not limited to, any number of central processing units (CPUs) or other processors (including DPUs, accelerators, field programmable gate arrays (FPGAs), graphics processors or graphics processing units (GPUs), etc.), memory devices (such as dynamic read-only memory), storage devices (such as solid-state drives or disk drives), network input / output (NW I / O) devices, network switches, virtual machines (VMs), power modules, and cooling modules, etc. In some embodiments, one or more of the node C.R. 916(1)-916(N) may correspond to a server having one or more of the above computing resources. Additionally, in some embodiments, the node C.R. 916(1)-916(N) may include one or more virtual components, such as vGPUs, vCPUs, etc., and / or one or more of the node C.R. 916(1)-916(N) may correspond to a virtual machine (VM).

[0233] In at least one embodiment, the grouped computing resources 914 may include separate groupings (not shown) of node C.R. 916 housed within one or more racks, or numerous racks (also not shown) within data centers located in various geographical locations. The separate groupings of node C.R. 916 within the grouped computing resources 914 may include grouped computing, network, memory, or storage resources that can be configured or allocated to support one or more workloads. In at least one embodiment, several node C.R. 916 including CPUs, GPUs, DPUs, and / or other processors may be grouped within one or more racks to provide computing resources to support one or more workloads. One or more racks may also include any number of power modules, cooling modules, and / or network switches in any combination.

[0234] The resource coordinator 912 may configure or otherwise control one or more of the node C.R. 916(1)-916(N) and / or the grouped computing resources 914. In at least one embodiment, the resource coordinator 912 may include a software design infrastructure (SDI) management entity for the data center 900. The resource coordinator 912 may include hardware, software, or some combination thereof.

[0235] In at least one embodiment, as Figure 9As shown, the framework layer 920 may include a job scheduler 933, a configuration manager 934, a resource manager 936, and a distributed file system 938. The framework layer 920 may include a framework for software 932 that supports the software layer 930 and / or one or more applications 942 of the application layer 940. The software 932 or the application 942 may respectively include web-based service software or applications, such as service software or applications provided by Amazon Web Services, Google Cloud, and Microsoft Azure. The framework layer 920 may be, but is not limited to, a free and open-source software web application framework, such as Apache Spark that can utilize the distributed file system 938 for large-scale data processing (e.g., "big data"). TM (hereinafter referred to as "Spark"). In at least one embodiment, the job scheduler 933 may include a Spark driver for facilitating the scheduling of workloads supported by the various layers of the data center 900. In at least one embodiment, the configuration manager 934 may be capable of configuring different layers, such as the software layer 930 and the framework layer 920 including Spark and the distributed file system 938 for supporting large-scale data processing. The resource manager 936 is capable of managing the cluster or grouped computing resources mapped to or allocated for supporting the distributed file system 938 and the job scheduler 933. In at least one embodiment, the cluster or grouped computing resources may include grouped computing resources 914 at the data center infrastructure layer 910. The resource manager 936 may coordinate with the resource coordinator 912 to manage these mapped or allocated computing resources.

[0236] In at least one embodiment, the software 932 included in the software layer 930 may include software used by at least a portion of the nodes C.R. 916(1)-916(N), the grouped computing resources 914, and / or the distributed file system 938 of the framework layer 920. One or more types of software may include, but are not limited to, Internet web search software, email virus browsing software, database software, and streaming video content software.

[0237] In at least one embodiment, one or more applications 942 included in the application layer 940 may include one or more types of applications used by at least a portion of nodes C.R. 916(1)-916(N), the grouped computing resources 914, and / or the distributed file system 938 of the framework layer 920. The one or more types of applications may include, but are not limited to, any number of genomics applications, cognitive computing and machine learning applications, including training or inference software, machine learning framework software (such as PyTorch, TensorFlow, Caffe, etc.) and / or other machine learning applications used in conjunction with one or more embodiments.

[0238] In at least one embodiment, any one of the configuration manager 934, the resource manager 936, and the resource coordinator 912 may implement any number and type of self-modifying actions based on any amount and type of data obtained in any technically feasible manner. The self-modifying actions may relieve the data center operator of the data center 900 from making potentially bad configuration decisions and may avoid underutilized and / or poorly performing portions of the data center.

[0239] The data center 900 may include tools, services, software, or other resources for training one or more machine learning models or using one or more machine learning models to predict or infer information according to one or more embodiments described herein. For example, a machine learning model may be trained by calculating weight parameters according to a neural network architecture by using the software and computing resources described above with respect to the data center 900. In at least one embodiment, by using the weight parameters calculated by one or more training techniques, the resources described above with respect to the data center 900 may be used to infer or predict information using the trained machine learning model corresponding to one or more neural networks, such as, but not limited to, those described herein.

[0240] In at least one embodiment, the data center 900 may use a CPU, an application specific integrated circuit (ASIC), a GPU, an FPGA, and / or other hardware (or corresponding virtual computing resources) to perform training and / or inference using the above resources. In addition, one or more of the above software and / or hardware resources may be configured as a service to allow a user to train or perform information inference, such as image recognition, speech recognition, or other artificial intelligence services.

[0241] Example Network Environment

[0242] The network environment suitable for implementing the embodiments of the present disclosure may include one or more client devices, servers, network attached storage (NAS), other backend devices, and / or other device types. The client devices, servers, and / or other device types (e.g., each device) may be implemented on one or more instances of the Figure 8 computing device 800 - for example, each device may include similar components, features, and / or functions of the computing device 800. Additionally, in the case of implementing backend devices (e.g., servers, NAS, etc.), the backend devices may be included as part of a data center 900, an example of which is described in more detail herein with respect to Figure 9 more detail.

[0243] The components of the network environment may communicate with each other via a network, which may be wired, wireless, or both. The network may include multiple networks, or networks within multiple networks. For example, the network may include one or more wide area networks (WANs), one or more local area networks (LANs), one or more public networks (e.g., the Internet and / or the public switched telephone network (PSTN)), and / or one or more private networks. In the case where the network includes a wireless telecommunications network, components such as base stations, communication towers, or even access points (and other components) may provide wireless connectivity.

[0244] A compatible network environment may include one or more peer - to - peer network environments (in which case servers may not be included in the network environment), and one or more client - server network environments (in which case one or more servers may be included in the network environment). In a peer - to - peer network environment, the functions described herein with respect to servers may be implemented on any number of client devices.

[0245] In at least one embodiment, the network environment may include one or more cloud - based network environments, distributed computing environments, combinations thereof, etc. A cloud - based network environment may include a framework layer, a job scheduler, a resource manager, and a distributed file system implemented on one or more servers, which may include one or more core network servers and / or edge servers. The framework layer may include a framework for supporting one or more applications of a software layer and / or an application layer. The software or application may respectively include network - based service software or applications. In an embodiment, one or more client devices may use network - based service software or applications (e.g., by accessing the service software and / or applications via one or more application programming interfaces (APIs)). The framework layer may be, but is not limited to, a type of free and open - source software web application framework, for example, which may use a distributed file system for large - scale data processing (e.g., "big data").

[0246] A cloud-based network environment can provide cloud computing and / or cloud storage that perform any combination of the computing and / or data storage functions (or one or more portions thereof) described herein. Any one of these various functions can be distributed across multiple locations from a central or core server (e.g., one or more data centers that can be distributed across states, regions, countries, globally, etc.). If the connection to a user (e.g., a client device) is relatively close to an edge server, the core server can assign at least a portion of the function to the edge server. A cloud-based network environment can be private (e.g., limited to a single organization), public (e.g., available to many organizations), and / or a combination thereof (e.g., a hybrid cloud environment).

[0247] A client device can include at least some of the components, features, and functions of the example computing device 800 described herein. By way of example and not limitation, a client device can be embodied as a personal computer (PC), laptop computer, mobile device, smartphone, tablet computer, smartwatch, wearable computer, personal digital assistant (PDA), MP3 player, virtual reality headset, global positioning system (GPS) or device, video player, camera, surveillance device or system, vehicle, boat, aircraft, virtual machine, drone, robot, handheld communication device, hospital device, gaming device or system, entertainment system, in-vehicle computer system, embedded system controller, remote control, appliance, consumer electronic device, workstation, edge device, any combination of these described devices, or any other suitable device. Figure 8

[0248] 1. In some embodiments, a method includes: receiving sensor output data and at least one sensor output setting associated with the sensor output data from a sensor; determining whether a queue of pending sensor input settings includes a matching sensor input setting that matches the sensor output setting; in response to determining that the queue of pending sensor input settings includes the matching sensor input setting, identifying, in the queue of pending sensor input settings, the pending sensor input settings in the queue that are before the matching sensor input setting; determining whether a sensor setting failure has occurred based at least on a pending request count associated with the pending sensor input settings; in response to determining that the sensor setting failure has occurred, generating an error indication based at least on the pending request count; and modifying one or more downstream operations that depend on the sensor output data based at least on the error indication.

[0249] 2. The method according to clause 1 further includes: in response to identifying the matching sensor input setting, reducing the count of pending requests associated with the matching sensor input setting, where the count of pending requests is at least based on the difference between the number of consecutive requests applying the pending sensor input setting to the sensor and the number of times the corresponding matching sensor output setting has been received from the sensor.

[0250] 3. The method according to clause 1 or 2, wherein determining whether the queue of pending sensor input settings includes the matching sensor input setting includes: determining whether the matching sensor input setting matches the sensor output setting according to a threshold tolerance.

[0251] 4. The method according to any one of clauses 1 - 3, wherein the threshold tolerance is at least based on one or more of the matching sensor input setting or the sensor output setting.

[0252] 5. The method according to any one of clauses 1 - 4 further includes: in response to determining that the queue of pending sensor input settings does not include the matching sensor input setting, generating an error indicating that a data corruption error has occurred.

[0253] 6. The method according to any one of clauses 1 - 5, wherein determining whether a sensor setting failure has occurred based at least on the count of pending requests associated with the pending sensor input setting includes: comparing the count of pending requests with zero.

[0254] 7. The method according to any one of clauses 1 - 6, wherein generating the error indication based at least on the count of pending requests includes: determining the type of sensor setting failure based at least on the count of pending requests.

[0255] 8. The method according to any one of clauses 1 - 7, wherein if the count of pending requests is zero, the type of sensor setting failure corresponds to no failure.

[0256] 9. The method according to any one of clauses 1 - 8, wherein if the count of pending requests associated with the pending sensor input setting is less than zero, the type of sensor setting failure corresponds to a sensor setting update delay or a sensor setting loss.

[0257] 10. The method according to any one of clauses 1 - 9, wherein if the count of pending requests associated with the pending sensor input setting is less than zero, and the count of the next pending request associated with the next queue element after the pending sensor input setting is zero, the type of sensor setting failure corresponds to a sensor setting update delay.

[0258] 11. The method according to any one of clauses 1 - 10, wherein if the pending request count associated with the pending sensor input setting is less than zero, and the next pending request count associated with the next queue element after the pending sensor input setting is greater than zero, then the sensor setting fault type corresponds to a lost sensor setting.

[0259] 12. The method according to any one of clauses 1 - 11, wherein if the pending request count associated with the pending sensor input setting is greater than zero, then the sensor setting fault type corresponds to a lost frame or a lost sensor setting.

[0260] 13. The method according to any one of clauses 1 - 12, wherein if the pending request count associated with the pending sensor input setting is greater than zero, and the previous pending request count associated with the previous queue element before the pending sensor input setting is zero, then the sensor setting fault type corresponds to a lost frame.

[0261] 14. The method according to any one of clauses 1 - 13, wherein if the pending request count associated with the pending sensor input setting is greater than zero, and the next pending request count associated with the previous queue element before the pending sensor input setting is less than zero, then the sensor setting fault type corresponds to a lost sensor setting.

[0262] 15. The method according to any one of clauses 1 - 14, further comprising: removing the pending sensor input setting from the queue in response to identifying the pending sensor input setting in the pending sensor input setting queue.

[0263] 16. The method according to any one of clauses 1 - 15, wherein the method further comprises: receiving an input setting; determining whether the input setting matches a previous setting in the pending sensor input setting queue; and in response to determining that the input setting matches the previous setting, incrementing the pending request count associated with the previous setting.

[0264] 17. The method according to any one of clauses 1 - 16, wherein the method further comprises: in response to determining that the input setting does not match the previous setting, adding the pending request count to the tail of the queue.

[0265] 18. In some embodiments, a processor includes: one or more processing units for performing operations, the operations including: receiving sensor output data and at least one sensor output setting associated with the sensor output data from a sensor; determining whether a queue of pending sensor input settings includes a matching sensor input setting that matches the sensor output setting; in response to determining that the queue of pending sensor input settings includes the matching sensor input setting, identifying, in the queue of pending sensor input settings, the pending sensor input settings in the queue that are before the matching sensor input setting; determining whether a sensor setting failure has occurred based at least on a pending request count associated with the pending sensor input setting; and in response to determining that the sensor setting failure has occurred, generating an error indication based at least on the pending request count.

[0266] 19. The processor according to clause 18, wherein the processor is included in at least one of the following: a control system for an autonomous or semi-autonomous machine; a perception system for an autonomous or semi-autonomous machine; a system for performing one or more simulation operations; a system for performing one or more digital twin operations; a system for performing optical transmission simulation; a system for performing collaborative content creation of 3D assets; a system for performing one or more deep learning operations; a system implemented using an edge device; a system for generating or presenting at least one of virtual reality content, augmented reality content, or mixed reality content; a system implemented using a robot; a system for performing one or more conversational AI operations; a system for performing one or more generative AI operations; a system implementing one or more large language models (LLMs); a system for generating synthetic data; a system including one or more virtual machines (VMs); a system implemented at least partially in a data center; or a system implemented at least partially using cloud computing resources.

[0267] 20. In some embodiments, a system includes: one or more processing units, the one or more processing units for performing operations, the operations including: receiving sensor output data and at least one sensor output setting associated with the sensor output data from a sensor; determining whether a queue of pending sensor input settings includes a matching sensor input setting that matches the sensor output setting; in response to determining that the queue of pending sensor input settings includes the matching sensor input setting, identifying, in the queue of pending sensor input settings, the pending sensor input settings in the queue that are before the matching sensor input setting; determining whether a sensor setting failure has occurred based at least on a pending request count associated with the pending sensor input setting; and in response to determining that the sensor setting failure has occurred, generating an error indication based at least on the pending request count.

[0268] The present disclosure may be described in the general context of machine - usable instructions or computer code, including computer - executable instructions such as program modules, executed by a computer or other machine, such as a personal digital assistant or other handheld device. Generally, program modules, including routines, programs, objects, components, data structures, etc., refer to code that performs particular tasks or implements particular abstract data types. The present disclosure may be practiced in a variety of system configurations, including handheld devices, consumer electronics, general - purpose computers, more specialized computing devices, etc. The present disclosure may also be practiced in a distributed computing environment where tasks are performed by remote processing devices linked through a communications network.

[0269] As used herein, the recitation of "and / or" with respect to two or more elements should be construed to refer to only one element or a combination of elements. For example, "element A, element B, and / or element C" may include only element A, only element B, only element C, element A and element B, element A and element C, element B and element C, or element A, B, and C. Further, "at least one of element A or element B" may include at least one of element A, at least one of element B, or at least one of element A and at least one of element B. Still further, "at least one of element A and element B" may include at least one of element A, at least one of element B, or at least one of element A and at least one of element B.

[0270] The subject matter of the present disclosure is described in detail herein to meet statutory requirements. However, the description itself is not intended to limit the scope of the present disclosure. On the contrary, the inventors have contemplated that the claimed subject matter may also be embodied in other ways, including combinations of steps different from or similar to the steps described herein in connection with other current or future technologies. Moreover, although the terms "step" and / or "block" may be used herein to imply different elements of the methods employed, these terms should not be construed to imply any particular order among or between the various steps disclosed herein unless the order of the steps is expressly described.

Claims

1. A method, comprising: Receiving sensor output data and at least one sensor output setting associated with the sensor output data from a sensor; Determining whether a queue of sensor input settings to be processed includes a matching sensor input setting that matches the sensor output setting; In response to determining that the queue of sensor input settings to be processed includes the matching sensor input setting, identifying, in the queue of sensor input settings to be processed, the sensor input settings to be processed that are before the matching sensor input setting in the queue; Determining whether a sensor setting failure has occurred based at least on a pending request count associated with the sensor input setting to be processed; In response to determining that the sensor setting failure has occurred, generating an error indication based at least on the pending request count; And Modifying one or more downstream operations that depend on the sensor output data based at least on the error indication.

2. The method according to claim 1, further comprising: In response to identifying the matching sensor input setting, reducing the pending request count associated with the matching sensor input setting, wherein the pending request count is based at least on a difference between a number of consecutive requests to apply the sensor input setting to be processed to the sensor and a number of times the corresponding matching sensor output setting has been received from the sensor.

3. The method according to claim 1, wherein, Determining whether the queue of sensor input settings to be processed includes the matching sensor input setting includes: determining whether the matching sensor input setting matches the sensor output setting according to a threshold tolerance.

4. The method according to claim 3, wherein The threshold tolerance is based at least on one or more of the matching sensor input setting or the sensor output setting.

5. The method according to claim 1, further comprising: In response to determining that the queue of sensor input settings to be processed does not include the matching sensor input setting, generating an error indicating that a data corruption error has occurred.

6. The method according to claim 1, wherein Determining whether a sensor setting failure has occurred based at least on the pending request count associated with the sensor input setting to be processed includes: comparing the pending request count with zero.

7. The method according to claim 1, wherein, Generating the error indication based at least on the pending request count includes: determining a sensor setting failure type based at least on the pending request count.

8. The method according to claim 7, wherein If the pending request count is zero, the sensor setting failure type corresponds to no failure.

9. The method according to claim 1, wherein, If the pending request count associated with the sensor input setting to be processed is less than zero, the sensor setting failure type corresponds to a sensor setting update delay or a sensor setting loss.

10. The method according to claim 1, wherein If the pending request count associated with the sensor input setting to be processed is less than zero, and the next pending request count associated with the next queue element after the sensor input setting to be processed is zero, the sensor setting failure type corresponds to a sensor setting update delay.

11. The method according to claim 1, wherein If the pending request count associated with the pending sensor input setting is less than zero and the next pending request count associated with the next queue element after the pending sensor input setting is greater than zero, the sensor setting fault type corresponds to a lost sensor setting.

12. The method according to claim 1, wherein, If the pending request count associated with the pending sensor input setting is greater than zero, the sensor setting fault type corresponds to a lost frame or a lost sensor setting.

13. The method according to claim 1, wherein If the pending request count associated with the pending sensor input setting is greater than zero and the previous pending request count associated with the previous queue element before the pending sensor input setting is zero, the sensor setting fault type corresponds to a lost frame.

14. The method according to claim 1, wherein, If the pending request count associated with the pending sensor input setting is greater than zero and the next pending request count associated with the previous queue element before the pending sensor input setting is less than zero, the sensor setting fault type corresponds to a lost sensor setting.

15. The method according to claim 1, further comprising: Removing the pending sensor input setting from the queue in response to identifying the pending sensor input setting in the pending sensor input setting queue.

16. The method according to claim 1, wherein, The method further comprises: Receiving an input setting; Determining whether the input setting matches a previous setting in the pending sensor input setting queue; and Increasing the pending request count associated with the previous setting in response to determining that the input setting matches the previous setting.

17. The method according to claim 16, wherein, The method further comprises: Adding the pending request count to the tail of the queue in response to determining that the input setting does not match the previous setting.

18. A processor, comprising: One or more processing units for performing operations, the operations including: Receiving sensor output data and at least one sensor output setting associated with the sensor output data from a sensor; Determining whether a pending sensor input setting queue includes a matching sensor input setting that matches the sensor output setting; Identifying, in the pending sensor input setting queue, a pending sensor input setting in the queue that is before the matching sensor input setting in response to determining that the pending sensor input setting queue includes the matching sensor input setting; Determining whether a sensor setting fault has occurred based at least on a pending request count associated with the pending sensor input setting; and Generating an error indication based at least on the pending request count in response to determining that the sensor setting fault has occurred.

19. The processor according to claim 18, wherein, The processor is included in at least one of the following: A control system for an autonomous or semi-autonomous machine; A perception system for an autonomous or semi-autonomous machine; A system for performing one or more simulation operations; A system for performing one or more digital twin operations; A system for performing optical transmission simulation; A system for performing collaborative content creation of 3D assets; A system for performing one or more deep learning operations; Systems implemented using edge devices; Systems for generating or presenting at least one of virtual reality content, augmented reality content, or mixed reality content; Systems implemented using robots; Systems for performing one or more conversational AI operations; Systems for performing one or more generative AI operations; Systems implementing one or more large language models (LLMs); Systems for generating synthetic data; Systems including one or more virtual machines (VMs); Systems implemented at least in part in a data center; or Systems implemented at least in part using cloud computing resources.

20. A system comprising: One or more processing units configured to perform operations including: Receiving sensor output data and at least one sensor output setting associated with the sensor output data from a sensor; Determining whether a queue of pending sensor input settings includes a matching sensor input setting that matches the sensor output setting; In response to determining that the queue of pending sensor input settings includes the matching sensor input setting, identifying, in the queue of pending sensor input settings, the pending sensor input settings in the queue that are before the matching sensor input setting; Determining whether a sensor setting failure has occurred based at least on a pending request count associated with the pending sensor input settings; and In response to determining that the sensor setting failure has occurred, generating an error indication based at least on the pending request count.

Images