Multi-modal Sensing for Autonomous Driving Vehicles with Self-repair Capability

Abstract

An apparatus for an autonomous vehicle, the apparatus including a perception pipeline having an independent classification process that operates in parallel to identify objects belonging to a particular object type respectively based on sensor data streams from a plurality of sensors among a plurality of different types of sensors. The apparatus also includes a sensor monitoring stage that is operative to operate in parallel with the perception pipeline and to use the sensor data streams to estimate and track a confidence level of each of the plurality of different types of sensors, and to invalidate a defective sensor when the confidence level associated with the defective sensor fails to meet a confidence threshold.

Description

Technical Field

[0001] Embodiments of the present disclosure are in the field of autonomous or semi-autonomous devices, and more particularly relate to multimodal sensing in autonomous vehicles with self-healing capabilities. Background Art

[0002] The background description provided herein is for the purpose of generally presenting the context of the present disclosure. Unless otherwise indicated herein, the materials described in this section are not prior art to the claims in this application and are not admitted to be prior art by virtue of inclusion in this section.

[0003] Autonomous or semi-autonomous devices (such as, autonomous vehicles, unmanned aerial vehicles (UAVs, also known as drones), or robots) may rely on a multimodal set of sensors to sense, map, and track the surrounding environment. The sensors may include several types, such as long-range radar, mid-range front radar, night vision cameras, cameras, rearview cameras, ultrasonic, mid-range rear radar, etc. Each type of sensor may have its own advantages and disadvantages.

[0004] Once a minimum set of sensors is defined, each of these sensors is essential for safely operating the vehicle. Whenever a sensor in this minimum set fails, the current vehicle behavior will take action to safely stop the vehicle by engaging an Emergency Brake System or a similar emergency stopping method applicable to driverless vehicles, or else it will compromise the vehicle's autonomy and safety. For this reason, autonomous vehicle manufacturers choose to provide a superset of the minimum set of sensors to provide redundancy for each type of sensor, which ultimately results in higher costs and higher peripheral device complexity in vehicle design. Brief Description of the Drawings

[0005] Embodiments will be readily understood by the following detailed description in conjunction with the accompanying drawings. For the sake of convenience of this description, the same reference numerals denote the same structural elements. In the figures of the accompanying drawings, embodiments are illustrated by way of example and not by way of limitation.

[0006] Figure 1 Illustrates a representative multimodal heterogeneous sensor array on an autonomous or semi-autonomous vehicle.

[0007] Figure 2 Illustrates a pipeline for multimodal sensor perception with continuous sensor monitoring and self-healing capabilities.

[0008] Figure 3 Is a flowchart illustrating a process performed by a multimodal perception pipeline with sensor monitoring and self-healing capabilities according to an embodiment of the disclosure.

[0009] Figure 4 Illustrates a Bayesian Network that serves as a management program for estimating the probability of a sensor failure given the behavior of all other relevant sensors.

[0010] Figure 5 Illustrates the equations used by the Bayesian Network management program to estimate the probability of a sensor failure.

[0011] Figure 6 Is a graph comparing one component of the vector descriptors interpreted from different sensors.

[0012] Figure 7 Illustrates an example clustering algorithm that can be used to detect outlier sensors.

[0013] Figure 8A Illustrates for Figure 6 The example shown Figure 7 The clustering results output by the clustering algorithm of

[0014] Figure 8B Is a graph illustrating the error metric of a faulty sensor.

[0015] Figure 9 Is a block diagram illustrating an example sensor suite management program process based on a sensor transducer model.

[0016] Figure 10 Illustrates an example computing device in which the apparatus and / or methods described herein can be employed according to various embodiments. Detailed Description

[0017] Describes apparatuses, methods, and storage media associated with multimodal sensing in autonomous vehicles with self-healing capabilities.

[0018] In the following detailed description, reference is made to the accompanying drawings, which form a part hereof. The same reference numerals throughout the drawings indicate the same parts, and embodiments that can be implemented are shown by way of illustration. It is to be understood that other embodiments can be utilized and structural or logical changes can be made without departing from the scope of the present disclosure. Therefore, the following detailed description should not be taken in a limiting sense, and the scope of the embodiments is defined by the appended claims and their equivalents.

[0019] Multiple aspects of the present disclosure are disclosed in the accompanying specification. Alternative embodiments and their equivalents of the present disclosure can be conceived without departing from the spirit or scope of the present disclosure. It should be noted that similar elements disclosed below are indicated by similar reference numerals in the drawings.

[0020] The operations may be described as a number of discrete actions or operations in series in a manner that is most helpful in understanding the claimed subject matter. However, the order of the description should not be construed as implying that these operations necessarily depend on the order. Specifically, these operations may not be performed in the order presented. The described operations may be performed in an order different from the described embodiments. In additional embodiments, various additional operations may be performed and / or the described operations may be omitted.

[0021] For the purposes of this disclosure, the phrase "A and / or B" means (A), (B), or (A and B). For the purposes of this disclosure, the phrase "A, B, and / or C" means (A), (B), (C), (A and B), (A and C), (B and C), or (A, B, and C).

[0022] The specification may use the phrases "in an embodiment" or "in embodiments," which may each refer to one or more of the same or different embodiments. Additionally, the terms "comprising," "including," "having," etc., as used with respect to embodiments of the present disclosure, are synonymous.

[0023] As used herein, the term "circuitry" may refer to, may be part of, or may include: an application specific integrated circuit (ASIC), an electronic circuit, a processor (shared, dedicated, or group) and / or memory (shared, dedicated, or group) that executes one or more software or firmware programs, combinational logic circuitry, and / or other suitable components that provide the described functionality.

[0024] Embodiments herein provide a multimodal method for object detection and classification to an autonomous vehicle or other device, which is used as an input to a traditional fusion grid mechanism. In a device that utilizes a multimodal sensor set, the device may detect and compensate for sensor defects in another modality by leveraging information obtained from one modality (e.g., sensor inactivity, sensor failure, sensor malfunction, sensor reset, a safety event associated with at least one of the sensors). This enables a robust multi-sensor based perception method to maintain autonomy after a sensor failure. In further embodiments, the sensor fault detection capability is extended to a self-healing capability for the detected faulty sensor. This enables safe operation without equipping a larger superset of the minimum sensor set, thereby reducing cost and system-level complexity. The device may be a component of a vehicle (such as a wheeled road vehicle or an off-road vehicle or any other type of vehicle such as a vehicle operating on a track, an aircraft, a ship, a robot, etc., or a combination thereof) or a non-vehicle (such as a stationary surveillance system that uses more than one sensor type).

[0025] Figure 1Illustrated is a representative multi-modal heterogeneous sensor array on an autonomous or semi-autonomous vehicle operating a perception pipeline 200 with a self-healing function according to an example embodiment (as Figure 2 described). In some applications, each heterogeneous sensor in the sensor array can monitor a different physical space. For example, in the heterogeneous sensor array in a motor vehicle embodiment, the sensors can monitor different spaces around the motor vehicle, such as a specific area in front of the motor vehicle, or a specific area behind the motor vehicle. Examples of the monitored areas around the motor vehicle can include: a long-range radar area 10, a mid-range radar front-end area 12, a night vision camera area 14, a camera area 16, an ultrasonic area 18, a mid-range radar rear-end area 20, and a rear-view camera area 22, some of which areas overlap (e.g., partially overlap). Different types of sensors that can monitor various areas in front of the motor vehicle can include: ultrasonic, cameras (day vision and / or night vision), mid-range radar, and long-range radar, LIDAR (Light Imaging, Detection and Ranging), etc. Different types of sensors that can monitor various areas behind the motor vehicle can include: ultrasonic and rear-view cameras, etc.

[0026] In a traditional perception pipeline based on autonomous driving sensors, the information of each sensor is independently used for single-sensor (single-modal) classification of a specific type of object, and these characteristic types of objects can be easily identifiable from the information provided by that single sensor. This means that there are objects detected only from LIDAR, radar, or cameras, which are then fused into a single representation by a fusion grid and are crucial for the trajectory planning of the motor vehicle. Although this method can generally be classified as sensor data fusion, this fusion does not utilize the entire available information (from all sensors) to optimize the detection / classification of each object.

[0027] In a traditional perception pipeline setup, a single sensor failure may cause the vehicle to lose the ability to classify a specific object that can only be sensed by that sensor. Therefore, the current behavior is to trigger an emergency stop behavior because the autonomy and safety of the vehicle are compromised. Since it is completely dependent on each sensor in the minimum sensor set, this reflects a lack of robustness in the system-level solution. The solution in a traditional autonomous driving perception pipeline is based on the addition of redundant / repeat sensors, which results in the disadvantages of increased cost and system-level complexity.

[0028] According to the exemplary embodiments disclosed herein, a perception pipeline for autonomous or semi-autonomous vehicles is disclosed that provides a multi-modal approach for object detection and classification with self-healing capabilities, as opposed to existing sensor type-based approaches. Independent multi-modal classification processes for different object types (e.g., lane markings, traffic signs, pedestrians, animals, other vehicles, etc.) operate in parallel, and each of the classification processes subscribes to synchronized sensor data streams from multiple sensors. Object detection and classification utilize low-level (raw data or extracted feature level) multi-sensor data fusion to maximize available information and improve classification accuracy. The multi-modal approach is pre-trained using worst-case scenarios in which one or more sensors in the multi-modal sensor suite feeding each classification process are missing, thereby allowing the classification processes to maintain a predetermined accuracy level for all required combinations of sensor defects during self-healing operations.

[0029] Figure 2 Diagram showing a multi-modal sensor perception pipeline with continuous sensor monitoring and self-healing capabilities, according to some embodiments. According to one embodiment, the multi-modal sensor perception pipeline 200 can be divided into four main components: a data acquisition stage 202, a multi-modal object classification stage 204, a sensor monitoring and repair stage 206, and a sensor repair feedback path 207.

[0030] The data acquisition stage 202 obtains data from multiple sensors 208A - 208N and passes the data as a synchronized sensor data stream 214 to the sensor monitoring and repair stage 206 (and indirectly to the multi-modal classification stage 204). In one embodiment, the data acquisition stage 202 can include a raw sensor data stage 208, a sensor data preprocessing stage 210, and a sensor data synchronization stage 212.

[0031] The multi-modal classification stage 204 includes multiple classification processes 204A - 204M that are assigned to identify different object types (e.g., lane markings, traffic signs, pedestrians, animals, other vehicles, etc.) in parallel. In one embodiment, each of the classification processes 204A - 204M subscribes to any number of synchronized sensor data streams 214 from multiple different sensors 208A - 208N. The multi-modal classification stage 204 utilizes low-level (raw data or extracted feature level) multi-sensor data fusion to maximize available information and increase classification accuracy.

[0032] In accordance with one aspect of the disclosed embodiments, the sensor monitoring and repair stage 206 uses all synchronized sensor data streams 214 to separately estimate and track the sensor confidence levels of each of the different sensors 208A - 208N with respect to the relevant sensors, where the synchronized sensor data streams 214 include pre - processed data (extracted features) from the sensor data pre - processing stage 210. The sensor monitoring and repair stage 206 operates in parallel with the data acquisition stage 202 and the multi - modal classification stage 204. As used herein, the term "parallel" means that the processes operate simultaneously, but not necessarily on the same data. In one embodiment, the sensor monitoring and repair stage 206 includes a sensor suite management program process 216, a sensor repair process 218, and a sensor data gateway 226.

[0033] Whenever the sensor confidence levels of the corresponding sensors 208A - 208N fail to meet the confidence threshold, the sensor repair feedback path 207 attempts to correct / re - configure the raw sensor data stage 208, the sensor data pre - processing stage 210, and optionally the sensor data synchronization stage 212 to attempt to restore the defective sensors to an operable confidence level. In one embodiment, the sensor repair feedback path 207 includes the sensor repair process 218 and the sensor tuning control signal 220 within the sensor monitoring and repair stage 206. The sensor tuning control signal 220 is output by the sensor monitoring and repair stage 206 to any defective sensors 208A - 208N, the corresponding sensor data pre - processors 210A - 210N, and optionally the sensor data synchronization stage 212. In some cases, when the sensors 208A - 208N are operating correctly, but the sensor data pre - processors 210A - 210N need to be tuned and / or the sensor data synchronization stage 212 needs to be re - synchronized, the sensor monitoring and repair stage 206 can output the sensor tuning control signal 220 only to the sensor data pre - processors 210A - 210N and / or the sensor data synchronization stage 212.

[0034] Figure 3 is a flowchart illustrating a process performed by a multi - modal perception pipeline having sensor monitoring and self - repair capabilities according to one disclosed embodiment. The process can begin by: operating a perception pipeline 200 having multiple classification processes 204A - 204M in parallel to separately identify objects belonging to a particular object type based on sensor data streams 214 from multiple different sensors 208A - 208N (block 300).

[0035] The sensor monitoring and repair stage 206 operates in parallel with the perception pipeline and uses the sensor data stream 214 to estimate and track the operating confidence level of each of multiple different types of sensors (block 302). In an embodiment, the sensor confidence level is the likelihood that the sensor is operating correctly.

[0036] When the sensor monitoring and repair stage 206 determines that the confidence level associated with a defective sensor fails to meet the operating confidence threshold (block 304), the sensor monitoring and repair stage 206 disables the defective sensor (block 306). It should be noted that the sensor disable signal 224 has no effect on the operation of the defective sensor because, as opposed to being "turned off", the defective sensor continues to operate and generate raw sensor data. In one embodiment, the sensor suite supervisor process 216 determines and tracks the confidence level associated with each of the sensors 208A - 208N and sends the sensor disable signal 224 to both the sensor repair process 218 and the sensor data gateway 226 when the confidence level associated with a target sensor fails to meet the confidence threshold. In one embodiment, the sensor suite supervisor process 216 can include individual sensor supervisor processes for monitoring each sensor or a subset of sensors among the sensors 208A - 208N. In one embodiment, the confidence threshold can represent the minimum likelihood that a defective sensor is operating correctly. In one embodiment, the confidence threshold can be a user - configurable value for each sensor.

[0037] In one embodiment, the sensor suite supervisor process 216 can also send the sensor confidence level 230 to the sensor repair process 218. In one embodiment, the sensor confidence level 230 can be sent whenever the sensor disable signal 224 is sent. In another embodiment, the sensor confidence levels 230 of all the sensors 208A - 208N can be continuously sent to the sensor repair process 218.

[0038] In response to disabling a defective sensor, the sensor monitoring and repair stage 206 sends a sensor tuning control signal 220 to the target sensor to attempt to restore the defective sensor to an operable confidence level (block 308). More specifically, in response to receiving a sensor disable signal 224 from the sensor suite manager process 216 by the self-healing process 218, the self-healing process 218 sends the sensor tuning control signal 220 to the defective sensor. In one embodiment, the sensor tuning control signal 220 is also sent to the corresponding sensor data preprocessing stage 210 and optionally to the sensor data synchronization stage 212. The sensor data synchronization stage 212 may also be tuned so that the "repaired" sensor can be synchronized with the remaining sensors (when possible / required) before declaring that the defective sensor is operating correctly again.

[0039] The sensor monitoring and repair stage 206 also causes the actual sensed data values output from the disabled defective sensor to be replaced by an invalid value data stream (e.g., a zero-valued constant data stream) (block 310). According to one embodiment, once the sensor data gateway 226 receives the sensor disable signal 224 from the sensor suite manager process 216, the sensor data gateway 226 switches the input to the multimodal classification stage 204 from the sensor data stream 214 of the disabled defective sensor to a constant invalid value data stream.

[0040] In response to one of the classification processes 204A - 204M receiving a constant invalid value data stream as the input of the disabled target sensor, the classification processes 204A - 204M operate in a degraded mode, in which the classification processes 204A - 204M continue to identify the respective object types using the sensor data streams of the properly operating sensors without the defective actual sensor data of the disabled sensor, while maintaining a predetermined accuracy level (block 320). If the number of simultaneously disabled sensors exceeds the ability of the classification processes 204A - 204M to maintain the predetermined accuracy level, the perception pipeline 200 may fall back to the normal behavior of detecting sensor failures, which is to call an emergency braking procedure (etc.) to stop the operation of the vehicle in a responsible manner.

[0041] According to one embodiment, replacing the output of the disabled target sensor data with an invalid value data stream, rather than completely stopping the data stream, effectively disables the target sensor from the perception pipeline 200 to ensure that the multimodal classification stage 204 only processes the sensor data from the properly operating sensors without disturbing the object classification process.

[0042] In a further embodiment, the classification processes 204A - 204M are trained a priori to operate in a degraded mode in the case of one or more defective sensors, while allowing the perception pipeline 200 to attempt to correct the defective sensors. These training cases are added to cover sensor failure scenarios, i.e., the training cases assume that any faulty sensor data (non-responsive, untuned, artifacts in the data stream, etc.) is replaced with a constant data stream of invalid or zero values. In one embodiment, the training covers all possible scenarios in which different subsets of the sensor data stream are invalidated, which allows the classification processes 204A - 204M to maintain a predetermined level of accuracy for all desired combinations of sensor defects during operation. Replacing the actual sensor "corrupted" data from the defective sensors with zero-valued data (which, by definition, has no information at all) makes the classification processes 204A - 204M easier to train and enables high accuracy of operation in the degraded mode.

[0043] This ability is crucial for a commercial vehicle to operate safely in the case where one sensor or one type of sensor is temporarily incapacitated (e.g., blinded), permanently incapacitated, or under attack. Additionally, this ability allows manufacturers to avoid high-order sensor redundancy, thereby reducing the added cost and system-level complexity.

[0044] The disclosed embodiments also add robustness without sacrificing real-time operation. That is, since the sensor monitoring and repair stage 206 is parallel to the multi-modal classification stage 204 (which is the critical path of the perception pipeline), no latency is added to the perception pipeline 200 and no pipeline process reset is required. Additionally, in the case of a detected sensor failure, the disclosed embodiments will not require resynchronization of the properly operating sensor data streams 214. Note that during the self-repair process, resynchronization of the sensor may be required before the previously invalidated target sensor is un-invalidated. Nevertheless, resynchronization may not be required when the sensor is invalidated (once it is detected to have low confidence).

[0045] Additional details of the components including the perception pipeline 200 are disclosed below.

[0046] Referring again to Figure 2, the raw sensor data level 208 may include a heterogeneous sensor array having a plurality of different types of sensors 208A - 208N, and the plurality of different types of sensors 208A - 208N preferably have individually configurable or tunable characteristics. Initially, the sensors 208A - 208N continuously generate data (raw data or interpreted data). Data from each of the sensors 208A - 208N can be input into one or more simultaneously reconfigurable sensor data preprocessors 210A - 210N, which can perform simple data adaptation functions (such as simple filtering, reformatting, etc., or more complex feature extraction). For the same sensor 208A - 208N, there can be multiple sensor data preprocessors 210A - 210N, thus enabling various features to be extracted from the same sensor 208A - 208N simultaneously. In one embodiment, there can be a one - to - many relationship between the raw sensor data from each of the sensors 208A - 208N and the corresponding sensor data preprocessors 210A - 210N. That is, for example, the raw data from a single sensor 208A can be fed into multiple parallel sensor data preprocessors 210A (each sensor data preprocessor extracts different features from the data). For this reason, the number of signals (or streams) in the synchronized sensor data stream 214 will be greater than or equal to the number of sensors (N).

[0047] Then, all the pre - processed sensor data streams 210B (≥N) are fed into the sensor data synchronization level 212, which synchronizes all the pre - processed sensor data streams 210B in time (taking into account the data capture time differences and pre - processing time differences between the sensors), and outputs the synchronized sensor data stream 214 to the sensor monitoring and repair level 206.

[0048] Now referring to the multimodal classification level 204, the disclosed multimodal perception pipeline 200 is characterized by having M parallel and independent multimodal object classification processes 204A - 204M, with one multimodal object classification process for each object type (e.g., lane, sign, pedestrian, other vehicle, etc.), where each of the classification processes 204A - 204M can simultaneously use data from multiple sensors (up to all sensors) to increase the available information for classification.

[0049] The classification processes 204A - 204M require multi - sensor data fusion (MSDF) at the level of raw data or extracted features. MSDF can be performed as a pre - stage for each classification process 204A - 204M (e.g., by some parametric method like model - based non - linear filtering), or MSDF can also be embedded into the classification processes 204A - 204M (e.g., training the classification processes to use multi - sensor data and / or features as inputs and implicitly solve the fusion at the hidden layer).

[0050] To implement the multi - modal aspects of the classification processes 204A - 204M, this embodiment can utilize a publish - subscribe (pub / sub) mechanism 222 to enable all simultaneously required sensor data streams 214 from the synchronized sensor data pre - processors 210A - 210N to the sensor monitoring and repair stage 206 for forwarding to the corresponding classification process 204A - 204M data inputs. The final configuration of the supported sensor data streams 214 can be hardened at the design stage, in which case the subscriptions should not change at runtime.

[0051] The potential implementation of the publish - subscribe mechanism 222 depends on whether the classification is done as a software process in a symmetric multi - processing (SMP) multi - core - like architecture or as a hardware stream processor (such as an application - specific instruction processor (ASIP) or a field - programmable gate array (FPGA) engine that can process streams in real - time). When the classification processes 204A - 204M are implemented as independent hardware components, this publish / subscribe mechanism 222 can be implemented as a fully - connected interconnection network between all synchronized sensor data outputs to all object classifiers, where each classification process 204A - 204M has multiple ports to the interconnection network (one port per parallel data stream). When the classification processes 204A - 204M are implemented as software processes mapped to different physical cores in a multi - core system, the publish - subscribe mechanism 222 can be implemented as a coherent SMP multi - processor mesh network with a distributed NUMA memory architecture that has memory access latency guarantees (illustrating consistency).

[0052] To achieve the robustness of the classification processes 204A - 204M, each classification process 204A - 204M must be trained to be able to perform adequately on all required combinations of sensor failures. To achieve this, it is assumed that any faulty and invalid sensor data stream will be replaced by a constant zero - valued data stream. The zero - valued data stream assumption for faulty sensors simplifies the training of the classification processes 204A - 204M for robustness (the classifier is pre - trained against sensor failures), allowing the classification processes 204A - 204M to provide a sufficient functional safety margin known at design time.

[0053] The sensor suite management program process 216 analyzes all synchronized sensor data streams 214 in parallel with the multimodal classification stage 204 and estimates the confidence level of each of the sensors 208A - 208N with respect to data received from all other statistically relevant sensors. As used herein, the phrase "statistically relevant sensors" refers to data output from a set of sensors that are statistically related (as opposed to related physical sensing mechanisms). In one embodiment, the sensor suite management program 216 needs to consider the confidence levels of sensors 208A - 208N when calculating the confidence levels of other sensors 208A - 208N. That is, when estimating the confidence levels of sensors other than a sensor with a low confidence level, the sensor with the low confidence level should be considered less (or discarded).

[0054] According to an embodiment disclosed herein, the sensor suite management program process 216 can estimate the confidence level of a target sensor by (in different ways) comparing the synchronized sensor data stream of the target sensor with the sensor data streams of a set of statistically relevant sensors. According to an embodiment disclosed herein, several different methods can be used by the sensor suite management program process 216 to compare the synchronized sensor data stream of the target sensor with the sensor data streams of a set of statistically relevant sensors.

[0055] In a first embodiment, a pre-trained sensor transformation model is provided for each sensor, which takes as input the synchronized sensor data streams of statistically relevant reference sensors and generates a predicted output for the target sensor. Thus, the sensor suite management program process 216 accesses the pre-trained model for the target sensor to transform the sensor data generated from the relevant sensors into modeled sensor data for the target sensor (e.g., modeling LIDAR data based on relevant camera and radar data). The pre-trained model can be used to transform the raw sensor data (or pre-extracted feature data) from the relevant sensors into a representation space corresponding to the target sensor. Note that "target" refers to the sensor for which the confidence level is to be calculated, and "relevant" or "reference" sensors are all other statistically relevant sensors. Then, a similarity metric between the predicted sensor output from the sensor transformation model and the actual target sensor data is calculated. This can be, for example, the distance (in the context of a vector space) between the features extracted from the predicted sensor data and the features of the actual sensor data.

[0056] In a second embodiment, the sensor suite management program process 216 can compare the synchronized sensor data stream of a target sensor with the synchronized sensor data streams of a statistically relevant set of sensors by accessing a pre-trained belief network that calculates a posterior probability estimate of a feature generated from the target sensor given the values of features generated by the relevant sensors. By using the confidence levels of the relevant sensors as the strength (weights) of the belief in such sensors, the values of the features of the relevant sensors are treated as soft evidence or uncertain evidence. The confidence levels are then approximated by the calculated posterior probability estimates.

[0057] In a third embodiment, the sensor suite management program process 216 can compare the synchronized sensor data streams from a statistically relevant set of sensors by: i) calculating a similarity / distance metric that is calculated as a weighted average of the pairwise similarities / distances of each pair of sensors in the statistically relevant set of sensors that includes the target sensor; and ii) performing real-time clustering and calculating the distance of each sensor to a cluster centroid that represents a confidence level, where outliers of the cluster centroid represent defective sensors. Note that in this case, all sensors in the "statistically relevant set of sensors" are both target sensors and reference sensors. Thus, any sensor in the set can deviate from the centroid of the cluster and become a defective sensor.

[0058] In a fourth embodiment, the sensor suite management program process 216 can compare the synchronized sensor data stream of a target sensor with the synchronized sensor data streams of a statistically relevant set of sensors by: accessing a parametric non-linear filter to fuse the features generated from the target sensor and the features from the relevant sensors into a common state space, and generating confidence level information from the error covariance matrix of the non-linear filter. By analyzing the elements of the non-linear filter error covariance matrix corresponding to the parameters of the features of the target sensor, the confidence levels can be inferred.

[0059] In a further embodiment, in addition to the object fusion grid, the outputs 228 of the classification processes 204A - 204M can be input to the sensor suite management program 216 to further improve the confidence level estimation.

[0060] In one embodiment, the sensor suite management program 216 can be configured with a per-sensor confidence level lower bound threshold that determines the minimum confidence level for declaring a sensor as operational. To avoid rapid switching between valid / invalid states, two different thresholds (lower bound for operational, upper bound for non-operational) can be used to create a hysteresis loop.

[0061] Whenever the confidence level of sensors 208A - 208N drops below a threshold, sensors 208A - 208N are invalidated. All sensor invalidation signals 224 for the complete sensor suite are fed forward to the sensor data gateway 226, and the sensor data gateway 226 in turn switches the publish / subscribe data 222 stream input of any invalidated sensors to a constant zero value input until the sensors are "uninvalidated". This is how the feed - forward path ensures that the multimodal classification stage 204 always operates in a pre - trained scenario, i.e., no erroneous data is forwarded to the classification processes 204A - 204M. Instead, the zero - value data stream effectively nullifies the impact of faulty sensors on object classification.

[0062] Since the sensor suite management program process 216 runs in parallel with the multimodal classification stage 204, the processing latency in the sensor suite management program process 216 does not introduce latency or bubbles in the multimodal classification stage 204. It also does not force resynchronization of currently working sensor data (i.e., the invalidated sensor data stream may need to be resynchronized before being uninvalidated, but this resynchronization does not affect the remaining active (uninvalidated) data streams), and thus, it does not have any negative impact on real - time constraints.

[0063] Now referring to the details of the sensor repair feedback path 207, whenever a sensor becomes invalid, a sensor repair process 218 for that sensor can be triggered to execute in parallel with all the previously described processes (so it does not add latency and it does not affect the real - time constraints of the platform). The sensor repair process 218 receives the confidence level and the invalidation status as inputs via the sensor confidence level 230 and the sensor invalidation signal 224 for each of the sensors 208A - 208N from the sensor suite management program process 216.

[0064] The sensor repair process 218 will attempt to "repair" the invalidated sensor by sending a sensor tuning control signal 220. In one embodiment, the sensor tuning control signal 220 can modify the tuning parameters of the sensor, reconfigure the sensor, reconfigure the sensor data pre - processing stage 210, reset the sensor, resynchronize the sensor (without disturbing the synchronization of working sensors), etc., while the sensor repair process 218 monitors how the confidence level 230 and the invalidation status of the sensor respond to the changes. For this reason, when the sensor repair process 218 is executing, the invalidated sensor and its corresponding sensor data pre - processor 210A should continue to operate, and the sensor suite management program process 216 can continue to monitor the sensor data stream 214 of the invalidated sensor.

[0065] Whenever the sensor repair operation (by recalibrating, readjusting, transforming, or filtering the signals from the sensors) corrects the sensor faults, the sensor suite manager process 216 can reactivate (\"un-disable\") the sensor into the sensing pipeline 200 by activating the sensor disable signal 224, and this activation triggers the completion of the sensor repair process 218.

[0066] The operations conveyed in the sensor tuning control signal 220 sent by the sensor repair process 218 can be at least the following three example types: i) a directed recipe of the exact operation sequence specified by a \"sensor expert\", where the \"sensor expert\" is known to correct pre-identified sensor problems; ii) deterministic optimization, which numerically finds the multi-dimensional direction of the steepest ascent at a confidence level; iii) meta-heuristic optimization, which allows randomness / stochasticity while searching for the best configuration to increase the confidence level.

[0067] The following provides example implementations for the disclosed embodiments. Sensor confidence level estimation aims to identify sensors or subsets of sensors that deviate from the expected functionality, where the expected functionality is extracted from the collective of trusted sensors.

[0068] Example 1 - Bayesian belief network inference supervision. The first example involves the implementation of a pre-trained belief network that the sensor suite manager process 216 can use to predict the output of the target sensor. Considering the pre-processed input features of all other relevant sensors as evidence for evaluation (i.e., using the features of relevant sensors as multi-variable random variables), a simple Bayesian network (BN) can be constructed separately for each sensor manager process among multiple sensor manager processes including the sensor suite manager process 216 to estimate the probability of a defective / faulty sensor, as Figure 4 shown.

[0069] Figure 4 Illustrated is a Bayesian network used as a manager for estimating the probability of a sensor failure given the behavior of all other relevant sensors. Assuming that the other sensors are working correctly, the probability of the expected sensor failure is relatively high. Different sensors 208A - 208N in the original sensor data level 208 output the original sensor data to the sensor data pre-processing level 210 and generate corresponding features 210A - 210N. For each sensor Xi = {A, B, C, D, …, N S}, based on the observed features extracted from the original sensor data, there is a probability distribution P(X i)400. Thus, corresponding Bayesian network management programs 402A - 402N can be established to estimate the failure probability 404 of each sensor. As an example, the Bayesian network management program 402A estimates the probability that the sensor 208A fails, as shown in Figure 6 shown.

[0070] Figure 5 The figure illustrates the equation used by the Bayesian network management program 402A to estimate the probability of sensor 208A failure. In this definition, the A posteriori result 500 represents the quantitative probability of the sensor 208A failing at the input end based on the observed probability distribution 400 of all other sensor characteristics. The evidence 502 represents the joint probability distribution of the probability distributions of all other sensors and remains constant; the A priori probability distribution 504 of sensor 208A failure is heuristically estimated and regarded as the initial belief, and it will be continuously updated with new observations. Given the information of sensor 208A, the likelihood 506 is represented by the product of all probability distributions of all other sensors. When the probability of failure in sensor 208A is high (set to a predefined threshold), given the observed characteristic probabilities of other sensors, the Bayesian network management program 402A can alert the perception pipeline 200 in the form of a sensor confidence level 230, as shown in Figure 2 shown. In this method, there is a Bayesian network management program 402 for each sensor to perform personalized analysis of failures.

[0071] Example 2 - Clustering-based supervision. In this example, the sensor suite management program process 216 creates clusters from the synchronized sensor data stream 214 corresponding to sensors in a statistically related set of sensors. Such clusters must be continuously created and updated. Generally, the n-dimensional representation space for clustering depends on the number of dimensions in the vector descriptors from the original sensor data or the interpreted sensor data (extracted features), and the method for clustering is based on unsupervised learning and works in a way that is always learning (i.e., the centroid of the cluster is constantly updating its position). For example, the unsupervised method can be k-means, where k is initially equal to the number of sensors. The following exemplary scenario is used to clarify the "clustering method".

[0072] The system only uses the interpreted sensors, which means the data from the sensors has a certain meaning. This data can potentially represent the vehicle's pose, distance to obstacles, etc. In this scenario, the number of dimensions required to describe the attributes is small, usually less than 10. For simplicity of illustration, only one dimension of this descriptor will be graphically represented below.

[0073] Figure 6A graph comparing one component of vector descriptors interpreted from different sensors. In the example shown, as grouped by component 600, the output of estimating the components of the vector descriptor from the set of sensors each time is similar for the set of given components in a given scenario. However, there is one sensor that provides an unreliable estimate of the same component 602. After detecting the corresponding outlier sensor, the outlier sensor should be removed from the calculation of the average value of the "trusted sensor set" corresponding to component 600, and this average value is used to define the reference value and perform sensor fusion. Moreover, after detection, the outlier signal can be invalidated at the multimodal classification stage 204.

[0074] Figure 7 Illustrates an example clustering algorithm that can be used to detect outlier sensors. Algorithm 700 dynamically adjusts the number of clusters such that there is only one cluster when all sensors converge to the same value. In block 702, the variable sigma (σ) is set equal to a function of the sensor output. In block 704, the output of each sensor is associated with the closest cluster. The error is calculated in block 706, and when all sensors are operating normally, the error should be a small value. The centroid of each of the multiple clusters is updated in block 708. If a sensor fails and produces incorrect data, the incorrect data may fall into different clusters. It is determined in block 709 whether the clusters are stable. Then it is determined in block 710 whether the error is significant. It is determined in block 712 whether the error is greater than a threshold. If so, the number of clusters is increased in block 714, and the process continues. Clustering algorithm 700 dynamically adjusts the number of clusters such that there is only one cluster when all sensors converge to the same value.

[0075] Figure 8A Illustrates for Figure 6 the example shown by Figure 7 the clustering results output by the clustering algorithm. In this example, clustering algorithm 700 generates two clusters: one cluster 800 for the trusted sensor set (average value) and another cluster 802 for the divergent sensors.

[0076] Figure 8B Is a graph illustrating the error metric (inverse similarity) of a faulty sensor. By calculating the Euclidean distance between the divergent sensor and the average value of the trusted sensor set, it is possible to generate an instantaneous error 900 (the reciprocal of the similarity / distance metric). Finally, the error metric is converted to a confidence level within a standardized range so that all sensors produce similar confidence levels. An example of standardization would be an example of max( C s,norm / error), where is the maximum confidence level, and C s,norm is the standardization coefficient specific to sensor S.

[0077] Example 3 - Supervision Based on Sensor Transducer Model Figure 9 is a block diagram illustrating an example sensor suite management program process 900 based on a sensor transducer model 908, where Figure 2 similar components have similar reference numerals. In this example, for each sensor, the sensor suite management program process 900 has a corresponding sensor management program process 902A - 902N. The sensor management program processes 902A - 902N subscribe to the synchronized sensor data stream 214 through a publish - subscribe (pub / sub) mechanism 901 and determine a confidence level 230. The confidence level 230 of each of the multiple sensors is compared with a confidence level threshold 904. If the confidence level 230 fails the comparison, a sensor invalidation signal 224 is sent to the sensor data gateway 226.

[0078] Each of the sensor management program processes 902A - 902N has a corresponding multimodal sensor conversion model 908 (as shown in the enlarged box for Sensor X). The multimodal sensor conversion model 908 is created at design time based on supervised learning. The multimodal sensor transducer model 908 receives the sensor data stream 214 from other sensors while receiving the current confidence level 230 of Sensor X. In this way, the sensor conversion model 908 should depend on all its inputs having a high confidence level, otherwise the conversion must be aborted. The output of the sensor transducer model 908 is the modeled sensor data 909 of Sensor X. A data similarity metric 910 is defined (the data similarity metric 910 will depend on the sensor type and its raw data format) to compare the modeled sensor data 909 with the actual raw data 912 of Sensor X. This similarity metric serves as the confidence 914 of Sensor X.

[0079] Figure 10 illustrates an example computing device 1000 that can employ the devices and / or methods described herein (e.g., any device and / or method associated with any of the computing devices or electronic devices described previously with respect to Figures 1-9 ). In an embodiment, the example computing device 1000 can be installed in an autonomous or semi - autonomous vehicle (i.e., a self - driving car, UAV, robot, etc.). As shown, the example computing device 1000 can include multiple components, such as one or more processors 1004 (one shown), at least one communication chip 1006, and different types of sensors 1007. The at least one communication chip 1006 can have an interface for docking with a network to obtain a trained sensor conversion model and / or receive raw sensor data from additional remote sensors (not shown).

[0080] In various embodiments, each of one or more processors 1004 may include one or more processor cores. In various embodiments, at least one communication chip 1006 may be physically or electrically coupled to one or more processors 1004. In a further implementation, at least one communication chip 1006 may be part of one or more processors 1004. In various embodiments, the computing device 1000 may include a printed circuit board (PCB) 1002. For these embodiments, one or more processors 1004 and at least one communication chip 1006 may be disposed on the printed circuit board.

[0081] Depending on its application, the computing device 1000 may include other components that may or may not be physically or electrically coupled to the PCB 1002. These other components include, but are not limited to: a memory controller (not shown), volatile memory (e.g., dynamic random access memory (DRAM) 1020), non-volatile memory such as flash memory 1022, a hardware accelerator 1024, an I / O controller (not shown), a digital signal processor (not shown), a cryptographic processor (not shown), a graphics processor 1030, one or more antennas 1028, a display (not shown), a touchscreen display 1032, a touchscreen controller 1046, a battery 1036, an audio codec (not shown), a video codec (not shown), a global positioning system (GPS) device 1040, a compass 1042, an accelerometer (not shown), a gyroscope (not shown), a speaker 1000, and a mass storage device (such as a hard disk drive, a solid state drive, a compact disc (CD), a digital versatile disc (DVD)) (not shown), and so on.

[0082] In some embodiments, one or more processors 1004, DRAM 1020, flash memory 1022, and / or a storage device (not shown) may include associated firmware (not shown) storing programming instructions that are configured to enable the computing device 1000 to perform the methods described herein in response to execution of the programming instructions by one or more processors 1004, such as compensating for sensor defects in a heterogeneous sensor array. In various embodiments, these aspects may additionally or alternatively be implemented using hardware separate from one or more processors 1004 or flash memory 1022, such as a hardware accelerator 1024 (which may be a field programmable gate array (FPGA)). In some embodiments, the hardware accelerator 1024 may be part of the processor 1004.

[0083] At least one communication chip 1006 can enable wired and / or wireless communication for data transmission to and from the computing device 1000. The term "wireless" and its derivatives can be used to describe circuits, devices, systems, methods, technologies, communication channels, etc. that can transfer data by using electromagnetic radiation modulated via a non-solid medium. The term does not imply that the associated devices do not contain any wires, although in some embodiments they may not contain any wires. At least one communication chip 1006 can implement any one of a large number of wireless standards or protocols, including but not limited to IEEE 702.20, Long Term Evolution (LTE), LTE Advanced (LTE-A), General Packet Radio Service (GPRS), Evolution-Data Optimized (Ev-DO), High Speed Packet Access Plus (HSPA+), High Speed Downlink Packet Access Plus (HSDPA+), High Speed Uplink Packet Access Plus (HSUPA+), Global System for Mobile Communications (GSM), GSM Enhanced Data Rates for GSM Evolution (EDGE), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Digital Enhanced Cordless Telecommunications (DECT), Worldwide Interoperability for Microwave Access (WiMAX), Bluetooth, its derivatives, and any other wireless protocols referred to as 3G, 4G, 5G, and higher generations. At least one communication chip 1006 can include multiple communication chips 1006. For example, a first communication chip 1006 can be dedicated to short-range wireless communication such as Wi-Fi and Bluetooth, while a second communication chip 1006 can be dedicated to long-range wireless communication such as GPS, EDGE, GPRS, CDMA, WiMAX, LTE, Ev-DO, and others.

[0084] In various implementations, the computing device 1000 can be a component of a vehicle, a component of a robot, a component of a surveillance system, a laptop computer, a netbook, a notebook, a superbook, a smart phone, a computing tablet, a personal digital assistant (PDA), an ultra-mobile PC, a mobile phone, a desktop computer, a server, a printer, a scanner, a monitor, a set-top box, an entertainment control unit (e.g., a game console or an in-vehicle entertainment unit), a digital camera, a household appliance, a portable music player, or a digital video recorder. In further implementations, the computing device 1000 can be any other electronic device that processes data.

[0085] One or more networks and / or data centers can be used to generate a sensor conversion model for use by computing device 1000. These networks and / or data centers can include systems of distributed computing devices, each of which can include components similar to any of the components of computing device 1000. The computing devices of the network and / or data center may not require sensor 1007, since such computing devices can receive input sensor data collected from computing device 1000 or some other similar computing device having a sensor similar to sensor 1007 (if it is a prototype of computing device 1000).

[0086] Any combination of one or more computer-usable media or computer-readable media can be utilized. The computer-usable media or computer-readable media can be, by way of example and not limitation, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, device, or propagation medium. More specific examples (a non-exhaustive list) of the computer-readable media would include the following: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a transmission medium such as those supporting the Internet or an intranet, or a magnetic storage device. Note that the computer-usable media or computer-readable media could even be paper or another suitable medium upon which the program is printed, as the program can be electronically captured via, for instance, optical scanning of the paper or other medium, then compiled, interpreted, or otherwise processed in a suitable manner and then stored in a computer memory. In the context of this document, the computer-usable media or computer-readable media can be any medium that can contain, store, communicate, propagate, or transport the program for use by or in connection with an instruction execution system, apparatus, or device. The computer-usable media can include a propagated data signal and computer-usable program code embodied therein either in baseband or as part of a carrier wave. Any suitable medium can be used to transmit the computer-usable program code, and suitable media include, but are not limited to, wireless, wireline, fiber optic cable, RF, etc.

[0087] Computer program code for performing the operations of this disclosure may be written in any combination of one or more programming languages, including object-oriented programming languages such as Java, Smalltalk, C++, etc., and conventional procedural programming languages such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, execute as a stand-alone software package, partly on the user's computer and partly on a remote computer, or entirely on the remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection to the external computer may be made (e.g., through the Internet using an Internet service provider).

[0088] In view of the above detailed description, these modifications can be made to this disclosure. The terms used in the appended claims should not be construed as limiting this disclosure to the specific implementations disclosed in the specification and the claims. Instead, the scope of the present invention should be determined entirely by the appended claims, which should be understood in accordance with the established rules of claim interpretation.

[0089] Example Embodiment 1 is a device for autonomous driving. The device includes a perception pipeline having an independent classification process that operates in parallel to identify object types respectively based on sensor data streams from multiple sensors of multiple different types. The device also includes a sensor monitoring stage that operates in parallel with the perception pipeline and uses the sensor data streams to estimate and track the confidence level of each of the multiple different types of sensors, and invalidates a target sensor when the confidence level associated with the target sensor fails to meet a confidence threshold.

[0090] Example Embodiment 2: The device of Example Embodiment 1, wherein in response to a defective sensor being invalidated, the sensor monitoring stage sends a sensor control signal to the invalidated sensor to attempt to restore the invalidated sensor to an operable confidence level and causes the actual sensed data values output by the invalidated sensor to be replaced by an invalid value data stream.

[0091] Example Embodiment 3: The device of Example Embodiment 2, wherein in response to one of the multiple classification processes receiving an invalid value data stream as an input corresponding to the invalidated sensor, the classification process operates in a degraded mode, in which the classification process continues to identify objects of each object type using the sensor data streams of the properly functioning sensors without the actual sensor data of the invalidated sensor while maintaining a predetermined accuracy level.

[0092] Example Embodiment 4: The apparatus of Example Embodiment 1, 2, or 3, wherein estimating a confidence level of a target sensor among a plurality of sensors includes: comparing a synchronized sensor data stream of the target sensor with sensor data streams of a statistically relevant set of sensors.

[0093] Example Embodiment 5: The apparatus of Example Embodiment 4, wherein comparing a synchronized sensor data stream of the target sensor with sensor data streams of statistically relevant sensors includes: accessing a pre-trained sensor conversion model of the target sensor, the pre-trained sensor conversion model taking data of a statistically relevant reference sensor as input and generating a predicted output of the target sensor, and calculating a similarity measure between the predicted output and actual target sensor data.

[0094] Example Embodiment 6: The apparatus of Example Embodiment 4, wherein comparing a synchronized sensor data stream of the target sensor with sensor data streams of statistically relevant sensors includes: accessing a pre-trained belief network that calculates a posterior probability estimate of features generated from the target sensor given values of features generated by relevant sensors.

[0095] Example Embodiment 7: The apparatus of Example Embodiment 4, wherein comparing a synchronized sensor data stream of the target sensor with sensor data streams of a statistically relevant set of sensors includes: calculating a similarity / distance measure that is calculated as a weighted average of pairwise similarities / distances of each pair of sensors in the statistically relevant set of sensors of the target sensor; and performing real-time clustering and calculating the distance of each sensor to a cluster centroid representing the confidence level of the sensor, wherein outliers in the cluster represent defective sensors.

[0096] Example Embodiment 8: The apparatus of Example Embodiment 4, wherein comparing a synchronized sensor data stream of the target sensor with sensor data streams of a statistically relevant set of sensors includes: accessing a parametric non-linear filter to fuse features generated from the target sensor and features from relevant sensors into a common state space, and generating confidence level information from an error covariance matrix of the non-linear filter.

[0097] Example Embodiment 9: The apparatus of Example Embodiment 1, 2, 3, 4, 5, 6, 7, or 8, wherein a sensor control signal is sent along a sensor repair feedback path coupled between a plurality of different sensors and a sensor monitoring stage to reconfigure an invalid target sensor and restore the invalid target sensor to an operable confidence level.

[0098] Example Embodiment 10: The apparatus of Example Embodiment 9, wherein when the confidence level associated with an invalid target sensor meets a confidence threshold, the sensor monitoring stage re-enables the target sensor to output normal sensor data stream values to the perception pipeline.

[0099] Example Embodiment 11: The apparatus of Example Embodiment 9 or 10, wherein the plurality of different types of sensors include respective data preprocessors for extracting features from the raw sensor data and outputting preprocessed sensor data streams, and these preprocessed sensor data streams are fed to the sensor data synchronization stage to synchronize the preprocessed sensor data streams in a timely manner and output the synchronized sensor data streams to the classification stage and the sensor monitoring stage.

[0100] Example Embodiment 12: The apparatus of Example Embodiment 11, wherein the sensor monitoring stage sends sensor control signals to the sensor data preprocessors of the respective defective sensors and to the sensor data synchronization stage.

[0101] Example Embodiment 13: The apparatus of Example Embodiment 11 or 12, wherein when the sensor is operating correctly but the sensor data preprocessor needs to be tuned or when the sensor data synchronization stage needs to be resynchronized, the sensor monitoring stage only sends sensor control signals to the sensor data preprocessor and the sensor data synchronization stage.

[0102] Example Embodiment 14: The apparatus of Example Embodiments 1, 2, 3, 4, or 9, wherein the classification process is pre-trained to maintain a predetermined accuracy level during operation for all required combinations of sensor defects, and the sensor data streams of any defective sensors are replaced with invalid value data to simplify the training.

[0103] Example Embodiment 15 is a method for an autonomous vehicle, the method including identifying objects belonging to a specific object type using respective independent classification processes operating in parallel based on sensor data streams from a plurality of sensors among a plurality of different types of sensors. In parallel with identifying the object type, the confidence level of each sensor among the plurality of different types of sensors is estimated and tracked from the sensor data streams, and defective sensors are invalidated when the confidence level associated with a defective sensor fails to meet the confidence threshold.

[0104] Example Embodiment 16: The apparatus of Example Embodiment 15, further comprising: in response to a defective sensor being invalidated, sending a sensor control signal to the invalidated defective sensor to attempt to restore the invalidated sensor to an operable confidence level, and causing the actual sensed data values output by the invalidated defective sensor to be replaced with an invalid value data stream.

[0105] Example Embodiment 17: The apparatus of Example Embodiment 16, wherein, in response to receiving an invalid value data stream as input corresponding to an invalid defect sensor during one of a plurality of classification processes, the classification process operates in a degraded mode in which the classification process continues to identify objects of respective object types using the sensor data streams of properly functioning sensors without the invalid defect sensor while maintaining a predetermined accuracy level.

[0106] Example Embodiment 18: The apparatus of Example Embodiment 15, 16, or 17, wherein estimating a confidence level of a target sensor among a plurality of sensors further comprises: comparing the synchronized sensor data stream of the target sensor with the sensor data streams of a statistically relevant set of sensors.

[0107] Example Embodiment 19: The apparatus of Example Embodiment 18, wherein comparing the synchronized sensor data stream of the target sensor with the sensor data streams of statistically relevant sensors further comprises: accessing a pre-trained sensor transformation model of the target sensor, the pre-trained sensor transformation model taking the data of a statistically relevant reference sensor as input and generating a predicted output of a defect sensor, and calculating a similarity metric between the predicted output and the actual target sensor data.

[0108] Example Embodiment 20: The apparatus of Example Embodiment 18, wherein comparing the synchronized sensor data stream of the target sensor with the sensor data streams of statistically relevant sensors further comprises: accessing a pre-trained belief network that calculates a posterior probability estimate of a feature generated from the target sensor given the values of features generated by relevant sensors.

[0109] Example Embodiment 21: The apparatus of Example Embodiment 18, wherein comparing the synchronized sensor data stream of the target sensor with the sensor data streams of a statistically relevant set of sensors further comprises: calculating a similarity / distance metric that is calculated as a weighted average of pairwise similarities / distances of each pair of sensors in the statistically relevant set of sensors of the target sensor; and performing real-time clustering and calculating the distance of each sensor to a cluster centroid representing the confidence level of the sensor, wherein outliers in the cluster represent defect sensors.

[0110] Example Embodiment 22: The apparatus of Example Embodiment 18, wherein comparing the synchronized sensor data stream of a defect sensor with the sensor data streams of a statistically relevant set of sensors further comprises: accessing a parametric non-linear filter to fuse the features generated from the target sensor and the features from the relevant sensors into a common state space, and generating confidence level information from the error covariance matrix of the non-linear filter.

[0111] Example Embodiment 23: The apparatus of Example Embodiments 15, 16, 17, 18, 19, 20, 21, or 22, wherein sensor control signals are sent along a sensor repair feedback path coupled between a plurality of different sensors and a sensor monitoring stage to reconfigure an invalid target sensor and restore the invalid target sensor to an operable confidence level.

[0112] Example Embodiment 24: The apparatus of Example Embodiment 23, wherein when the confidence level associated with an invalid target sensor meets a confidence threshold, the sensor monitoring stage re-enables the target sensor to output normal sensor data stream values to a perception pipeline.

[0113] Example Embodiment 25: The apparatus of Example Embodiment 23 or 24, wherein the plurality of different types of sensors include respective data preprocessors for extracting features from raw sensor data and outputting preprocessed sensor data streams, and the preprocessed sensor data streams are fed to a sensor data synchronization stage to synchronize the preprocessed sensor data streams in a timely manner and output the synchronized sensor data streams to a classification stage and a sensor monitoring stage.

[0114] Example Embodiment 26: The apparatus of Example Embodiment 25, wherein the sensor monitoring stage sends sensor control signals to the sensor data preprocessor of a corresponding defective sensor and to the sensor data synchronization stage.

[0115] Example Embodiment 27: The apparatus of Example Embodiment 25 or 26, wherein when a sensor is operating correctly but the sensor data preprocessor needs to be tuned or when the sensor data synchronization stage needs to be resynchronized, the sensor monitoring stage sends sensor control signals only to the sensor data preprocessor and the sensor data synchronization stage.

[0116] Example Embodiment 28: The apparatus of Example Embodiments 15, 16, 17, 18, or 23, wherein a classification process is pre-trained to maintain a predetermined accuracy level during operation over all desired combinations of sensor defects, and the sensor data stream of any defective sensor is replaced with invalid value data to simplify training.

[0117] Example Embodiment 29: A computer-readable medium including executable instructions, wherein the instructions, in response to execution of the instructions by a processor, cause the processor to identify an object belonging to a specific object type based on sensor data streams from a plurality of sensors among a plurality of different types of sensors. In parallel with identifying the object type, the confidence level of each of the plurality of different types of sensors is estimated and tracked from the sensor data streams, and a defective sensor is invalidated when the confidence level associated with the defective sensor fails to meet a confidence threshold.

[0118] Example Embodiment 30: The apparatus of Example Embodiment 29, wherein the instructions further cause the processor to: in response to a defect sensor being invalidated, send a sensor control signal to the invalidated defect sensor to attempt to restore the invalidated sensor to an operable confidence level, and cause the actual sensed data value output by the invalidated defect sensor to be replaced by an invalidated value data stream.

[0119] Example Embodiment 31: The apparatus of Example Embodiment 30, wherein the instructions further cause the processor to: in response to one of a plurality of classification processes receiving an invalidated value data stream as an input corresponding to an invalidated defect sensor, operate the classification process in a degraded mode, in which the classification process continues to identify respective object types using the sensor data stream of properly functioning sensors without the actual sensor data of the invalidated defect sensor while maintaining a predetermined level of accuracy.

[0120] Example Embodiment 32: The apparatus of Example Embodiment 29, 30, or 31, wherein estimating the confidence level of a target sensor among a plurality of sensors further comprises: comparing the synchronized sensor data stream of the target sensor with the sensor data streams of a statistically relevant set of sensors.

[0121] Example Embodiment 33: An apparatus for an autonomous vehicle, the apparatus comprising a perception pipeline having independent classification processes that operate in parallel to identify objects belonging to a particular object type based on sensor data streams from a plurality of sensors among a plurality of different types of sensors. A sensor monitoring stage apparatus for operating in parallel with the perception pipeline and for using the sensor data streams to estimate and track the confidence level of each of the plurality of different types of sensors, and invalidating a defect sensor when the confidence level associated with the defect sensor fails to meet a confidence threshold.

[0122] Example Embodiment 34: The apparatus of Example Embodiment 33, wherein in response to a defect sensor being invalidated, the sensor monitoring stage apparatus sends a sensor control signal to the invalidated sensor to attempt to restore the invalidated sensor to an operable confidence level, and causes the actual sensed data value output by the invalidated sensor to be replaced by an invalidated value data stream.

[0123] Example Embodiment 35: The apparatus of Example Embodiment 34, wherein in response to one of a plurality of classification processes receiving an invalidated value data stream as an input corresponding to an invalidated sensor, the classification process operates in a degraded mode, in which the classification process continues to identify objects of respective object types using the sensor data stream of properly functioning sensors without the actual sensor data of the invalidated sensor while maintaining a predetermined level of accuracy.

[0124] Example Embodiment 36: The apparatus of Example Embodiment 33 or 34, wherein estimating a confidence level of a target sensor among a plurality of sensors includes: comparing a synchronized sensor data stream of the target sensor with sensor data streams of a statistically relevant set of sensors.

[0125] Example Embodiment 37: A multimodal sensor perception pipeline apparatus for an autonomous vehicle includes: a data acquisition stage for acquiring data from a plurality of different types of sensors and outputting synchronized sensor data streams corresponding to the plurality of different types of sensors. A multimodal classification stage having a plurality of classification processes for parallelly identifying different object types based on the synchronized sensor data streams from a plurality of sensors among the plurality of different types of sensors. A sensor monitoring stage for operating in parallel with the data acquisition stage and the multimodal classification stage, the sensor monitoring stage inputting the synchronized sensor data streams to estimate and track a confidence level of each of the plurality of different types of sensors. A defective sensor that is invalidated when a confidence level associated with the defective sensor fails to meet a confidence threshold. A sensor repair feedback path that sends a sensor control signal to the invalidated defective sensor in response to the invalidation of the defective sensor to attempt to restore the invalidated defective sensor to an operable confidence level.

[0126] Example Embodiment 38: The apparatus of Example Embodiment 37, wherein in response to the defective sensor being invalidated, the sensor monitoring stage causes an actual sensed data value output by the invalidated defective sensor to be replaced with an invalid value data stream.

[0127] Example Embodiment 39: The apparatus of Example Embodiment 38, wherein in response to one of the plurality of classification processes receiving an invalid value data stream as an input of the invalidated defective sensor, the classification process operates in a degraded mode, in which the classification process continues to identify respective object types using sensor data streams of properly operating sensors without actual sensor data of the invalidated defective sensor while maintaining a predetermined accuracy level.

[0128] Example Embodiment 40: The apparatus of Example Embodiment 37, 38, or 39, wherein estimating a confidence level of a target sensor among a plurality of sensors includes: comparing a synchronized sensor data stream of the target sensor with sensor data streams of a statistically relevant set of sensors.

Claims

1. An apparatus for an autonomous vehicle, the apparatus comprising: a perception pipeline having an independent classification process that operates in parallel to identify objects belonging to a specific object type based on sensor data streams from multiple sensors of multiple different types; and a sensor monitoring stage for operating in parallel with the perception pipeline and for using the sensor data streams to estimate and track the confidence level of each of the multiple different types of sensors, and invalidating a defective sensor when the confidence level associated with the defective sensor fails to meet a confidence threshold, wherein the sensor confidence level is the likelihood that the sensor is operating correctly, and wherein in response to the defective sensor being invalidated, the sensor monitoring stage sends a sensor control signal to the invalidated sensor to attempt to restore the invalidated sensor to an operable confidence level.

2. The apparatus according to claim 1, wherein in response to the defective sensor being invalidated, the sensor monitoring stage causes the actual sensed data values output by the invalidated sensor to be replaced by an invalid value data stream.

3. The apparatus according to claim 2, wherein in response to one of the multiple classification processes receiving an invalid value data stream as input corresponding to the invalidated sensor, the classification process operates in a degraded mode in which the classification process continues to identify objects of each object type using the sensor data streams of the properly operating sensors without the actual sensor data of the invalidated sensor while maintaining a predetermined level of accuracy.

4. The apparatus according to claim 1, 2 or 3, wherein estimating the confidence level of a target sensor among the multiple sensors includes: comparing the synchronized sensor data stream of the target sensor with the sensor data streams of a statistically relevant set of sensors.

5. The apparatus according to claim 4, wherein comparing the synchronized sensor data stream of the target sensor with the sensor data streams of a statistically relevant set of sensors includes: accessing a pre-trained sensor transformation model of the target sensor, the pre-trained sensor transformation model taking as input the data of a statistically relevant reference sensor and generating a predicted output of the target sensor, and calculating a similarity metric between the predicted output and the actual target sensor data.

6. The apparatus according to claim 4, wherein comparing the synchronized sensor data stream of the target sensor with the sensor data streams of a statistically relevant set of sensors includes: accessing a pre-trained belief network that calculates a posterior probability estimate of the features generated from the target sensor given the values of the features generated by the relevant sensors.

7. The apparatus according to claim 4, wherein Comparing the synchronized sensor data stream of the target sensor with the sensor data streams of a statistically relevant set of sensors includes: calculating a similarity / distance metric, which is calculated as a weighted average of pairwise similarities / distances of each pair of sensors in the statistically relevant set of sensors of the target sensor; and performing real-time clustering and calculating the distance of each sensor to a cluster centroid representing the confidence level of the sensor, where outliers in the cluster represent the defective sensors.

8. The apparatus according to claim 4, wherein, comparing the synchronized sensor data stream of the target sensor with the sensor data streams of a statistically relevant set of sensors includes: accessing a parametric non-linear filter to fuse features generated from the target sensor and features from the relevant sensors into a common state space, and generating confidence level information from the error covariance matrix of the non-linear filter.

9. The apparatus according to claim 1, 2 or 3, wherein, sending the sensor control signal along a sensor repair feedback path coupled between a plurality of different sensors and the sensor monitoring stage to reconfigure the ineffective target sensor and restore the ineffective target sensor to an operable confidence level.

10. The apparatus according to claim 9, wherein, when the confidence level associated with the ineffective target sensor meets the confidence threshold, the sensor monitoring stage re-enables the target sensor to output normal sensor data stream values to the perception pipeline.

11. The apparatus according to claim 9, wherein, the plurality of different types of sensors include respective data preprocessors for extracting features from the raw sensor data and outputting preprocessed sensor data streams, and these preprocessed sensor data streams are fed to a sensor data synchronization stage to synchronize the preprocessed sensor data streams in a timely manner and output the synchronized sensor data streams to the classification process and the sensor monitoring stage.

12. The apparatus according to claim 11, wherein, the sensor monitoring stage sends the sensor control signal to the sensor data preprocessor of the respective defective sensor and to the sensor data synchronization stage.

13. The apparatus according to claim 11 or 12, wherein, when the sensor is operating correctly but the sensor data preprocessor needs to be tuned or when the sensor data synchronization stage needs to be resynchronized, the sensor monitoring stage only sends the sensor control signal to the sensor data preprocessor and the sensor data synchronization stage.

14. The apparatus according to claim 1, 2, or 3, wherein, the classification process is pre-trained to maintain a predetermined accuracy level during operation on all required combinations of sensor defects, where the sensor data stream of any defective sensor is replaced with the invalid value data to simplify the training.

15. A method for an autonomous vehicle, comprising: identifying objects belonging to a specific object type using respective independent classification processes operating in parallel based on sensor data streams from a plurality of sensors of a plurality of different types; estimating and tracking a confidence level of each of the plurality of different types of sensors from the sensor data streams in parallel with identifying the object type, and invalidating a defective sensor when the confidence level associated with the defective sensor fails to meet a confidence threshold, wherein the sensor confidence level is the likelihood that the sensor is operating correctly, and sending a sensor control signal to the invalidated defective sensor in response to the defective sensor being invalidated to attempt to restore the invalidated sensor to an operable confidence level.

16. The method according to claim 15, further comprising: causing actual sensed data values output by the invalidated defective sensor to be replaced by an invalid value data stream in response to the defective sensor being invalidated.

17. The method according to claim 16, wherein, in response to one of the plurality of classification processes receiving the invalid value data stream as an input corresponding to the invalidated defective sensor, operating the classification process in a degraded mode, in which the classification process continues to identify objects of respective object types using the sensor data streams of sensors that are operating correctly without the invalidated defective sensor while maintaining a predetermined level of accuracy.

18. The method according to claim 15, 16 or 17, wherein, estimating a confidence level of a target sensor among the plurality of sensors includes: comparing the synchronized sensor data stream of the target sensor with the sensor data streams of a statistically relevant set of sensors.

19. The method according to claim 18, wherein, comparing the synchronized sensor data stream of the target sensor with the sensor data streams of a statistically relevant set of sensors further includes: accessing a pre-trained sensor transformation model of the target sensor, the pre-trained sensor transformation model taking data of a statistically relevant reference sensor as an input and generating a predicted output of the defective sensor, and calculating a similarity metric between the predicted output and the actual target sensor data.

20. The method according to claim 18, wherein, comparing the synchronized sensor data stream of the target sensor with the sensor data streams of a statistically relevant set of sensors further includes: accessing a pre-trained belief network, the pre-trained belief network calculating a posterior probability estimate of a feature generated from the target sensor given values of features generated by relevant sensors.

21. The method according to claim 18, wherein, Further including comparing the synchronized sensor data stream of the target sensor with the sensor data streams of a statistically relevant set of sensors: calculating a similarity / distance metric, which is calculated as a weighted average of pairwise similarities / differences of each pair of sensors in the statistically relevant set of sensors of the target sensor; and performing real-time clustering and calculating the distance of each sensor to a cluster centroid representing the confidence level of the sensor, wherein outliers in the cluster represent the defective sensors.

22. A computer-readable medium including executable instructions, wherein, the instructions, upon execution by a processor, cause the processor to: Based on sensor data streams from multiple sensors of multiple different types, use individual independent classification processes operating in parallel to identify objects belonging to a specific object type; Estimate and track the confidence level of each of the multiple different types of sensors from the sensor data streams in parallel with identifying the object type, and invalidate a defective sensor when the confidence level associated with the defective sensor fails to meet a confidence threshold, wherein the sensor confidence level is the likelihood that the sensor is operating correctly, and In response to the defective sensor being invalidated, send a sensor control signal to the invalidated defective sensor to attempt to restore the invalidated sensor to an operable confidence level.

23. The computer-readable medium according to claim 22, wherein, the instructions further cause the processor to: in response to the defective sensor being invalidated, cause the actual sensed data values output by the invalidated defective sensor to be replaced by an invalid value data stream.

24. A multi-modal sensor perception pipeline apparatus for an autonomous vehicle, the apparatus comprising: A data acquisition stage for acquiring data from multiple different types of sensors and outputting a synchronized sensor data stream corresponding to the multiple different types of sensors; A multi-modal classification stage having multiple classification processes for identifying different object types in parallel based on the synchronized sensor data streams from multiple sensors of the multiple different types; A sensor monitoring stage for operating in parallel with the data acquisition stage and the multi-modal classification stage, the sensor monitoring stage outputting the synchronized sensor data stream to estimate and track the confidence level of each of the multiple different types of sensors; And invalidating a defective sensor when the confidence level associated with the defective sensor fails to meet a confidence threshold; And A sensor repair feedback path that, in response to the invalidation of the defective sensor, sends a sensor control signal to the invalidated defective sensor to attempt to restore the invalidated defective sensor to an operable confidence level.

25. The apparatus according to claim 24, wherein, The sensor monitoring stage responds to the defective sensor being invalidated such that the actual sensed data value output by the invalidated defective sensor is replaced by an invalid value data stream.

Images