Method and device for monitoring failure of autonomous driving perception hardware function

Abstract

The application relates to an automatic driving perception hardware function failure monitoring method and an automatic driving perception hardware function failure monitoring device and belongs to the technical field of deep learning.The method comprises the following steps: compressing perception hardware data based on an encoder to obtain corresponding low-dimensional potential representation; establishing a perception hardware data model according to the hidden variable of the perception hardware data potential representation; introducing the conditional probability distribution between perception hardware, constructing an elastic probability graph between different perception hardware through the perception hardware data model; and combining the elastic probability graph to monitor each perception hardware in real time and perform risk assessment.The application can comprehensively consider the complex relationship between each perception hardware and monitor the safety of the automatic driving system in real time.

Description

TECHNICAL FIELD

[0001] The present application relates to an automatic driving perception hardware function failure monitoring method, and simultaneously relates to an automatic driving perception hardware function failure monitoring device, and belongs to the technical field of deep learning. BACKGROUND

[0002] With the rapid development of artificial intelligence technology, autonomous vehicles gradually transform from a concept to reality, however, its safety problem has always been the focus of attention. In the autonomous driving system, perception hardware is a crucial component, responsible for collecting and interpreting information about the surrounding environment. However, these sensor systems are not immune to failure, and the safety risks caused by perception hardware failure have become a major challenge to the development of autonomous driving technology.

[0003] Perception hardware failure can be caused by a variety of reasons, including sensor damage, environmental condition changes, severe weather, electromagnetic interference, etc. When the perception hardware fails, the autonomous vehicle may not accurately perceive the surrounding environment, leading to misjudgment, wrong decision, and even causing traffic accidents. Therefore, timely monitoring the state of perception hardware and taking appropriate safety measures is a crucial link to ensure the safety of autonomous vehicles.

[0004] Currently, in the sensor system, the diagnosis method of perception hardware failure is mainly divided into two categories: internal diagnosis and external monitoring. The method based on internal diagnosis usually integrates a self-checking mechanism inside the sensor system, which detects the working state of the sensor through regular self-checking. This includes self-checking of real-time data output by the sensor to detect abnormal or deviating values from the normal range. In addition, the system may be equipped with multiple sensors of the same or different types, and by comparing their output results, it can be detected whether there is an abnormality. If a sensor fails, other normally working sensors can provide backup information.

[0005] The method based on external monitoring uses additional external monitoring devices, such as cameras, laser radars, etc., to monitor the actual working condition of the sensor. These devices can capture environmental information around the sensor and compare it with the sensor output.

[0006] Although these methods can detect sensor failures to some extent, there are still some limitations. Current technologies mainly focus on simple failure detection of a single sensor, such as wire breakage, offset, etc. However, for complex failures such as incorrect interpretation of sensor data, errors caused by environmental changes, etc., existing technologies are difficult to cope with. In addition, existing technologies often lack comprehensive risk monitoring and evaluation of the entire system configuration and algorithm. Since autonomous driving systems usually work collaboratively by multiple sensors and algorithms, a failure of one sensor can have a chain effect on the entire system. Therefore, a more comprehensive and comprehensive method is needed to evaluate the safety risk of the system in the case of sensor failure, in order to better ensure the safety of autonomous vehicles. SUMMARY

[0007] The technical problem to be solved by the present application is to overcome the shortcomings of the prior art and provide an autonomous driving perception hardware function failure monitoring method and device that can comprehensively consider the complex relationship between each perception hardware and monitor the safety of the autonomous driving system in real time.

[0008] To achieve the above technical purpose, in one aspect, the present application provides an autonomous driving perception hardware function failure monitoring method, comprising:

[0009] compressing the perception hardware data based on an encoder to obtain a corresponding low-dimensional latent representation;

[0010] establishing a perception hardware data model according to the latent variable of the perception hardware data latent representation;

[0011] introducing the conditional probability distribution between the perception hardware, and constructing an elastic probability graph between different perception hardware through the perception hardware data model;

[0012] combining the elastic probability graph to monitor and evaluate the risk of each perception hardware in real time.

[0013] Preferably, the compression of the perception hardware data based on the encoder to obtain the corresponding low-dimensional latent representation further comprises:

[0014] the encoder maps the perception hardware training data to the latent space to obtain the corresponding latent representation;

[0015] the decoder restores the latent representation to the original data;

[0016] based on the latent representation and the original data, using variational lower bound to maximize the marginal log-likelihood function of the perception hardware training data, thereby training the encoder.

[0017] Preferably, the establishment of the perception hardware data model according to the latent variable of the perception hardware data latent representation specifically comprises:

[0018] introducing a latent variable of a potential representation of the perception hardware data, calculating a prior distribution of the latent variable;

[0019] obtaining a corresponding joint distribution through a product of the marginal distribution and the prior distribution of the latent variable;

[0020] calculating a corresponding posterior distribution through the marginal distribution and the prior distribution of the latent variable by using the Bayes theorem;

[0021] representing the perception hardware data model through the posterior distribution of the perception hardware latent variable.

[0022] More preferably, the conditional probability distribution between the perception hardware is introduced, and an elastic probabilistic graph between different perception hardware is constructed through the perception hardware data model, specifically including:

[0023] Each node of the elastic probabilistic graph is represented by a different perception hardware data model, and each perception hardware corresponds to a parent node set;

[0024] The conditional probability distribution is introduced, and the joint probability distribution of different perception hardware with a dependency relationship is calculated in combination with the parent nodes corresponding to each perception hardware;

[0025] The edges corresponding to different nodes of the elastic probabilistic graph are represented by the joint probability distribution of different perception hardware under a given condition.

[0026] More preferably, the construction of the elastic probabilistic graph between different perception hardware further includes:

[0027] Parameter learning of the elastic probabilistic graph is performed by using existing data;

[0028] The learning goal is to maximize the joint probability distribution of different perception hardware with a dependency relationship under a given condition in the elastic probabilistic graph;

[0029] A complete elastic probabilistic graph is constructed through the parameters.

[0030] More preferably, the real-time monitoring and risk assessment of each perception hardware by the elastic probabilistic graph specifically include:

[0031] The likelihood probability of a target perception hardware under a current condition is calculated by using the elastic probabilistic graph; the likelihood probability is the joint probability distribution of the target perception hardware and the perception hardware with a dependency relationship;

[0032] The likelihood probability of the target perception hardware under the current condition is compared with the likelihood probability under a normal working state, and a set threshold is used to determine whether the current perception hardware is abnormal.

[0033] More preferably, the real-time monitoring and risk assessment of each perception hardware by the elastic probabilistic graph further include:

[0034] If the current perception hardware data is abnormal, the likelihood probability of the current perception hardware under abnormal conditions is calculated according to the elastic probabilistic graph;

[0035] According to the likelihood probability of the current perception hardware under abnormal conditions, the risk assessment index of the current perception hardware is calculated.

[0036] Preferably, the joint probability distribution of different perception hardware with dependency relationship is:

[0037] (10)

[0038] In formula (10), represents the conditional probability distribution between nodes, V represents a node set, and represents two perception hardware data models with dependency relationship, represents the parent node set of node represents the parent node set of node

[0039] In another aspect, the automatic driving perception hardware function failure monitoring device provided by the application comprises:

[0040] A deep feature extraction system is used to compress perception hardware data based on an encoder and obtain corresponding low-dimensional latent representation;

[0041] A latent variable system is used to establish a perception hardware data model according to the latent variable of the perception hardware data latent representation;

[0042] An elastic probabilistic graph system is used to introduce the conditional probability distribution between perception hardware, and to construct an elastic probabilistic graph between different perception hardware through the perception hardware data model;

[0043] An anomaly detection and risk assessment system is used to combine the elastic probabilistic graph to monitor and perform risk assessment on each perception hardware in real time.

[0044] In another aspect, the automatic driving perception hardware function failure monitoring device provided by the application comprises: a processor and a memory, the processor reads the computer program in the memory, and is used to perform the following operations:

[0045] Compress perception hardware data based on an encoder and obtain corresponding low-dimensional latent representation;

[0046] According to the latent variable of the perception hardware data latent representation, a perception hardware data model is established;

[0047] ​​Introduce the conditional probability distribution between the perception hardware, construct the elastic probability graph between different perception hardware through the perception hardware data model;

[0048] Combine the elastic probability graph to monitor and evaluate the risk of each perception hardware in real time.

[0049] In the present application, by introducing hidden variable modeling and constructing an elastic probability graph, the complex relationship between sensors is comprehensively considered, not just limited to simple fault detection of a single perception hardware, but also considering the chain effect between different sensors. The present application comprehensively considers the risk of the entire autonomous driving system configuration and algorithm through the elastic probability graph model, and better ensures the overall safety of the autonomous driving system. BRIEF DESCRIPTION OF DRAWINGS

[0050] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or prior art description will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor on the basis of these drawings.

[0051] Figure 1 The method flowchart of the embodiments of the present application is shown in the figure.

[0052] Figure 2 The structure schematic diagram of the device in the embodiments of the present application is shown in the figure.

[0053] Figure 3 The structure schematic diagram of the system in the embodiments of the present application is shown in the figure. DETAILED DESCRIPTION

[0054] The technical solutions in the embodiments of the present application will be described clearly and completely in the following with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some embodiments of the present application, not all. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0055] Generally, multiple perception hardware will be installed on the autonomous driving system, including: steering sensor, brake sensor, throttle opening sensor, laser radar, camera, etc.

[0056] The present application aims to solve the safety risk problem of the functional failure of the perception hardware of the autonomous driving vehicle, such as Figure 1 As shown, an autonomous driving perception hardware functional failure monitoring method is proposed, which comprises:

[0057] 101, based on the encoder to the perception hardware data compression, obtain the corresponding low-dimensional latent representation;

[0058] 102, according to the latent variable of the perception hardware data latent representation, the perception hardware data model is established;

[0059] 103, the conditional probability distribution between the perception hardware is introduced, and the elastic probability graph between different perception hardware is constructed through the perception hardware data model;

[0060] 104, combining the elastic probability graph, each perception hardware is monitored and risk assessment in real time.

[0061] In the embodiment, by introducing the latent variable modeling and constructing the elastic probability graph, the complex relationship between sensors is comprehensively considered, not only limited to simple fault detection of single perception hardware, but also considering the chain effect between different sensors. The present application comprehensively considers the risk of the whole automatic driving system configuration and algorithm through the elastic probability graph model, and better ensures the overall safety of the automatic driving system.

[0062] The 101 further comprises:

[0063] 201, the encoder maps the perception hardware training data to the latent space to obtain the corresponding latent representation;

[0064] 202, the decoder restores the latent representation to the original data;

[0065] 203, based on the latent representation and the original data, the marginal log-likelihood function of the perception hardware training data is maximized using the variational lower bound, so as to train the encoder.

[0066] In the embodiment, in the 201, the posterior distribution of the latent representation W of the perception hardware data X of the automatic driving system is defined as , and then the encoder neural network is used to parameterize the distribution, wherein the posterior distribution of W under the condition of X is a multivariate normal distribution:

[0067] (1)

[0068] In formula (1), and are the mean and variance output by the encoder neural network.

[0069] In the 202, the prior distribution of the latent representation W of the hardware data X is defined as , and the standard normal distribution is usually selected:

[0070] (2)

[0071] The process of generating the perceptual hardware raw data can be represented as a decoder The decoder neural network is parameterized.

[0072] Let the likelihood distribution of W given X be a multivariate normal distribution:

[0073] (3)

[0074] In formula (3), and are the mean and variance outputted by the decoder neural network.

[0075] In the 203, the marginal log-likelihood function of the perceptual hardware data is maximized using the variational lower bound.

[0076] The variational lower bound is:

[0077] (4)

[0078] In formula (4), represents the KL divergence; is the marginal probability distribution of X;

[0079] (5)

[0080] In formula (5), i represents the i-th dimension in the vector, i represents the variance of the i-th dimension, represents the mean of the i-th dimension; i i

[0081] (6)

[0082] In formula (6), represents the expectation of the posterior distribution of the latent variable, represents the variance of the i-th dimension output, i represents the i-th dimension of the observation variable X, represents the variance of the i-th dimension output; i i

[0083] By maximizing the variational lower bound, the parameters of the encoder and the decoder are optimized; finally, the trained encoder can generate a low-dimensional latent representation of the perceptual hardware data.

[0084] In this embodiment, the encoder can use a variational autoencoder (VAE) to process the perceptual hardware data to obtain a low-dimensional latent representation of the perceptual hardware data. ​​​​​​

[0085] The 102, specifically includes:

[0086] 1021, introduce a latent variable of a potential representation of the perceptual hardware data, calculate a prior distribution of the latent variable;

[0087] 1022, obtain a corresponding joint distribution through a product of the marginal distribution and the prior distribution of the latent variable;

[0088] 1023, calculate a corresponding posterior distribution through the marginal distribution and the prior distribution of the latent variable by using Bayes' theorem;

[0089] 1024, represent a perceptual hardware data model through the posterior distribution of the perceptual hardware latent variable.

[0090] In the embodiment, in the 1021, first introduce a latent variable Z of a low-dimensional representation W of the perceptual hardware data X, the latent variable Z is used to describe the implicit relationship between different perceptual hardware; the prior distribution of the latent variable Z is , and is usually selected as a standard normal distribution:

[0091] (7)

[0092] Consider a joint distribution between the perceptual hardware data X and the latent variable Z .

[0093] The joint distribution can be expressed as a product of a marginal distribution and a prior distribution :

[0094] (8)

[0095] Calculate the posterior distribution between the perceptual hardware data X and the latent variable Z by using Bayes' theorem :

[0096] (9)

[0097] In formula (9), the denominator is a normalization factor, which can be calculated by marginalizing the latent variable Z of the perceptual hardware data.

[0098] Usually, the posterior distribution is generally difficult to express analytically, so the model parameters are learned by using variational inference or maximum likelihood function estimation, so that the optimized posterior distribution is closer to the true distribution. The likelihood function is the marginal distribution . The optimized posterior distribution represents the perceptual hardware data model.

[0099] In the embodiment, the diversity of perception hardware data is more comprehensively considered by introducing latent variables to model the perception hardware data, and the complex relationship between different perception hardware can be better captured, thereby improving the accuracy of safety risk monitoring of perception hardware functional failure. The method described in the embodiment can effectively handle complex faults such as incorrect interpretation of perception hardware data and errors caused by environmental changes.

[0100] The 103 specifically includes:

[0101] 1031, each node of the elastic probabilistic graph is represented by a different perception hardware data model, and each perception hardware corresponds to a parent node set;

[0102] 1032, a conditional probability distribution is introduced, and the joint probability distribution of different perception hardware with dependency is calculated in combination with the parent nodes corresponding to each perception hardware;

[0103] 1033, the edges corresponding to different nodes of the elastic probabilistic graph are represented by the joint probability distribution of different perception hardware under a given condition.

[0104] In 1031, the posterior distribution obtained from 102 is used to reflect the change of the fault probability distribution of the perception hardware over time or after an event occurs. In 1031, each perception hardware is regarded as a node, and the parent nodes of the perception hardware are represented by the node in combination with the corresponding posterior distribution. The parent nodes can include all perception hardware of the autonomous driving system, or all perception hardware of the same type as the perception hardware.

[0105] The conditional probability distribution is introduced to represent the dependency relationship between nodes, which can be represented as or for each pair of nodes.

[0106] The elastic probabilistic graph can be represented as a graph , where V is a node set corresponding to perception hardware and latent variables, and E is an edge set corresponding to the probability dependency relationship between nodes.

[0107] The joint probability distribution of different perception hardware with dependency is:

[0108] (10)

[0109] In formula (10), represents the conditional probability distribution between nodes, V represents the node set, and represent the data models of two perception hardware with dependency, representing a node a set of parent nodes of the node representing a node a set of parent nodes of the node.

[0110] The 103 further comprises:

[0111] 1034. Parameter learning of the elastic probabilistic graph using existing data;

[0112] 1035. The learning objective is to maximize the joint probability distribution of different perception hardware with dependencies under given conditions in the elastic probabilistic graph;

[0113] 1036. Construct a complete elastic probabilistic graph using the parameters.

[0114] In this embodiment, parameter learning is performed using existing data, i.e., estimating the parameters in the probability distribution. The goal of parameter learning is to maximize the likelihood function where θ is the parameter of the probability distribution, and an optimization algorithm such as gradient ascent is used to solve the parameter.

[0115] For example, a complete elastic probabilistic graph G is constructed, which contains two nodes of camera and lidar , and let and represent the conditional probability distribution of camera and lidar data, respectively. Here, the parent node set and may include other perception hardware data or latent variables according to the perception hardware model obtained in 102.

[0116] In the parameter learning process, the existing camera and lidar data are used for parameter learning to estimate the parameters of their conditional probability distributions. Assuming that Gaussian distribution is used to model these conditional probability distributions, the camera data and lidar data can be represented as and , and their probability distributions under given conditions are assumed to be and , respectively. Then, maximum likelihood estimation can be used to estimate the parameters of these probability distributions.

[0117] Assuming that the conditional probability distributions of camera data and lidar data are multivariate Gaussian distributions, their parameters are mean vectors and covariance matrices , The goal is to maximize the likelihood function, i.e., to maximize the conditional probability given the data. Assuming there are N sets of known camera and lidar data, the following likelihood function can be obtained:

[0118] (11)

[0119] In formula (11), and represent the i-th set of camera and lidar data, and represent the parent node set of the camera and lidar, respectively. Since a Gaussian distribution is assumed, the above probability can be represented by the density function of a multivariate Gaussian distribution. According to the learned probability dependency and parameters, a complete flexible probabilistic graph between the camera

[0120] and the lidar can be constructed. In this example, the conditional probability distribution between the camera and the lidar can be obtained. It is assumed that the output of the camera is affected by the lidar data, or both are affected by a certain hidden variable. In the flexible probabilistic graph, there will be a corresponding edge to represent this dependency. Specifically, the camera and the lidar can be taken as nodes in the graph, and an edge is established between them to represent that the output of the camera is affected by the lidar data. According to the learned probability distribution parameters, the conditional probability distribution on this edge is set. For example, the estimated multivariate Gaussian distribution parameters can be used to describe this dependency. In this way, a complete flexible probabilistic graph is obtained, which can characterize the global state of the camera and lidar sensor system.

[0121] In this embodiment, according to the learned probability dependency, a complete flexible probabilistic graph is constructed. The edges in the graph represent the conditional probability distribution between the nodes, and the nodes correspond to the perception hardware data and / or hidden variables; the topology of the flexible probabilistic graph reflects the dependency relationship in the perception hardware system, which helps to comprehensively consider the state of the system. By constructing the flexible probabilistic graph, the dependency relationship between the nodes in the perception hardware system can be more comprehensively modeled, so as to more accurately characterize the state of the entire system. The method described in this embodiment can help to monitor the abnormal situation in the sensor data in real time, and provide reliable evaluation for the perception hardware function failure safety risk.

[0122] The 104 includes two parts of anomaly detection and risk assessment, and the anomaly detection part specifically includes:

[0123]

[0124] ​1041、use the elastic probabilistic graph to calculate the likelihood probability of the target perception hardware under the current condition; the likelihood probability is the joint probability distribution of the target perception hardware and the perception hardware with dependent relationship;

[0125] 1042、compare the likelihood probability of the target perception hardware under the current condition with the likelihood probability under the normal working state, and determine whether the current perception hardware is abnormal through the set threshold.

[0126] In the embodiment, the learned complete elastic probabilistic graph model is used to calculate the likelihood probability of the target perception hardware under the current condition , wherein data represents the current condition.

[0127] Compare the current likelihood probability with the likelihood probability under the normal working state, and determine whether there is an anomaly through the set threshold:

[0128] (12)

[0129] In formula (12), Threshold represents the likelihood probability under the normal working state, i.e. the threshold, if the threshold is not exceeded, the target perception hardware is normal; if the threshold is exceeded, it is abnormal.

[0130] The 104 further comprises a risk assessment part, specifically comprising:

[0131] 1043, if the current perception hardware data is abnormal, calculate the likelihood probability of the current perception hardware under the abnormal condition according to the elastic probabilistic graph;

[0132] 1044, calculate the risk assessment index of the current perception hardware according to the likelihood probability of the current perception hardware under the abnormal condition.

[0133] In the embodiment, if an anomaly is detected, the overall risk of the system is evaluated by considering the node dependent relationship in the elastic probabilistic graph; the conditional probability distribution of the system under the sensor failure condition is calculated .

[0134] According to the specific risk assessment indexes: conditional entropy and conditional variance, the system risk is quantified.

[0135] (13)

[0136] In formula (13), Anomaly represents the abnormal condition.

[0137] In the comprehensive system state determination process, the conditional entropy and the conditional variance are used as the risk assessment indexes to quantify the system risk.

[0138] For random variables X and Y, the conditional entropy of X is represents the uncertainty of X given Y:

[0139] (14)

[0140] In formula (14), X and Y the joint probability distribution of X and Y given Z, Y the probability distribution of X given Y;

[0141] For random variables X and Y, the conditional variance of X given Y is represents the dispersion of X given Y:

[0142] (15)

[0143] In formula (15), Y = y the variance of X given Y. X

[0144] Therefore, in the comprehensive system state determination, the conditional entropy and the conditional variance are comprehensively considered to construct the comprehensive risk assessment index Z:

[0145] (16)

[0146] In formula (2), represents the conditional entropy of the perception hardware X and the perception hardware Y under abnormal conditions, represents the conditional variance of X and Y , and represents the coefficient of balancing the conditional entropy and the conditional variance.

[0147] By monitoring the actual sensor data, the conditional entropy and the conditional variance are calculated, and the formula of the comprehensive risk assessment index Z is substituted. If Z exceeds the threshold set in advance, it is determined that the perception hardware system has risks. Otherwise, the system state is normal.

[0148] Specifically, the conditional entropy measures the uncertainty of the system state determination, that is, the degree of uncertainty of the prediction of the system state given a set of conditions. While the conditional variance reflects the degree of change of the system state determination given the conditions. By comprehensively considering these two indicators, the comprehensive risk assessment index Z can more comprehensively quantify the safety risk of the system.

[0149] ​​​​Specifically, assume that an autonomous driving system is equipped with 1 lidar, 4 surround-view cameras, 1 main camera, and 4 millimeter-wave radars. For each perception hardware, the feedback data can measure its conditional entropy and conditional variance under different conditions. Then, according to the comprehensive risk assessment index Z, the feedback data comprehensively considers the risks of these perception devices to evaluate the safety of the entire perception system. If the conditional entropy of the lidar under a certain condition is high, it means that the perception result is relatively uncertain; at the same time, the large conditional variance indicates that the perception result changes greatly under this condition. This will lead to a relatively high comprehensive risk assessment index Z of the lidar under this condition. By performing similar comprehensive evaluations on all perception devices, the feedback data can obtain the risk state of the entire autonomous vehicle perception system.

[0150] The method described in the embodiment can be summarized as the following steps:

[0151] Step 1, obtaining raw data from each perception hardware of the autonomous driving system;

[0152] Step 2, processing the perception hardware data using a variational autoencoder in deep learning to achieve a low-dimensional and highly abstract representation of the data;

[0153] Step 3, introducing a latent variable to model the implicit relationship between perception hardware to more comprehensively consider the diversity and complexity of perception hardware data;

[0154] Step 4, constructing a flexible probabilistic graph according to the dependency relationship between perception hardware to fully depict the global state of the sensor system;

[0155] Step 5, learning the relationship of the probabilistic graph through parameter learning in the flexible probabilistic graph system, so that it can accurately reflect the complexity of the perception system;

[0156] Step 6, using the flexible probabilistic graph model, the system monitors the abnormal situation in the perception hardware data in real time, and finds the abnormality caused by the failure of the perception hardware function;

[0157] Step 7, on the basis of abnormality detection, risk assessment is performed to in-depth analyze the impact of the failure on the entire system;

[0158] Step 8, combining the results of deep feature extraction, latent variable modeling, and flexible probabilistic graph construction, to determine whether the overall system state is normal.

[0159] As shown in Figure 3 the embodiment of the application also provides an autonomous driving perception hardware failure monitoring device, comprising:

[0160] a deep feature extraction system for obtaining a corresponding low-dimensional latent representation based on encoder compression of the perception hardware data;

[0161] a latent variable system for establishing a perception hardware data model according to latent variables of the perception hardware data latent representation;

[0162] a flexible probabilistic graph system for introducing a conditional probability distribution between perception hardware and constructing a flexible probabilistic graph between different perception hardware through the perception hardware data model;

[0163] an anomaly detection and risk assessment system for real-time monitoring and risk assessment of each perception hardware in combination with the flexible probabilistic graph.

[0164] In this embodiment, four systems are constituted: a deep feature extraction system, a latent variable system, a flexible probabilistic graph system, and an anomaly detection and risk assessment system. First, the deep feature extraction system learns a low-dimensional high abstract representation of sensor data using a variational autoencoder to capture key features. Then, the latent variable system introduces latent variables to more comprehensively consider the diversity and complexity of sensor data by modeling the implicit relationship between sensors. Subsequently, the flexible probabilistic graph system constructs a flexible probabilistic graph based on the dependency relationship between sensors to depict the global state of the sensor system. Finally, the anomaly detection and risk assessment system uses the flexible probabilistic graph model to monitor anomalies in sensor data in real time and further performs risk assessment to determine the impact of a fault on the entire system.

[0165] By integrating the four systems, the device provided in this embodiment improves the limitations of current technology in sensor fault detection by comprehensively considering the complexity of multi-sensor cooperative work, and provides a higher level of protection for the safety risk monitoring of perception hardware function failure of an autonomous vehicle.

[0166] As shown in Figure 2 the embodiment of the present application also provides an autonomous driving perception hardware function failure monitoring device, which comprises a memory 10 and a processor 20, wherein the processor 20 reads a computer program in the memory 10 and is used to perform the following operations:

[0167] obtaining a corresponding low-dimensional latent representation based on encoder compression of the perception hardware data;

[0168] establishing a perception hardware data model according to latent variables of the perception hardware data latent representation;

[0169] introducing a conditional probability distribution between perception hardware and constructing a flexible probabilistic graph between different perception hardware through the perception hardware data model;

[0170] real-time monitoring and risk assessment of each perception hardware in combination with the flexible probabilistic graph.

[0171] The specific implementations have been described in detail above with reference to exemplary embodiments. However, other embodiments can exist that are within the scope of the disclosure, and what has been described above is not intended to limit the disclosure as there is a plurality of alternatives for replacing items, using equivalent equivalents and customizing the disclosed embodiments to a specific application.

[0172] It should be understood that the particular order in which steps of the processes have been presented and / or described is merely illustrative. Steps of the processes can be rearranged or otherwise re-ordered without departing from the spirit and / or scope of the disclosure. The accompanying method claims have been presented in a particular, exemplary order with respect to steps thereof for purposes of illustration and description. The steps as presented and / or described can be implemented in other orders than the presented and / or described, without departing from the scope of the present disclosure.

[0173] In order to make the description of the present disclosure more detailed and complete, the above describes the embodiments and specific examples of the present disclosure in an illustrative manner; but this is not the only form of implementing or using the specific embodiments of the present disclosure. The embodiments include the features of the specific embodiments and the method steps and their order for constructing and operating the specific embodiments. However, other specific embodiments can also be used to achieve the same or equivalent functions and step orders.

[0174] In the above detailed description, various features are grouped together in a single embodiment for the purpose of streamlining the disclosure. This disclosed approach is not to be interpreted as reflecting an intention that the claimed embodiments require more features than are explicitly recited in each claim. On the contrary, as reflected by the appended claims, the claimed disclosure is to be understood as comprising any novel features found within the scope of the claims. Thus, the appended claims are hereby expressly incorporated into the detailed description, with each claim standing on its own as a separate preferred embodiment.

[0175] The disclosed embodiments have been described above with the intent of enabling any persons skilled in the art to implement or use the disclosure. Various modifications of the embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein can be applied to other embodiments without departing from the spirit or scope of the disclosure. Thus, the present disclosure is not intended to be limited to the embodiments described herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0176] The above description includes examples of one or more embodiments. Of course, not all possible combinations of components or methods described above will be employed to make or use the embodiments nor will all of the following described examples necessarily be realized. One of ordinary skill in the art, however, having the benefit of the present description, can understand how to make and use variations of the embodiments under the teachings and concepts described herein. Thus, the embodiments described herein are intended to embrace all such alterations, modifications, and variations that fall within the scope of the appended claims. Furthermore, the terms "comprises", "comprising", "includes", "including", "has", "having" and the like are to be construed open-ended, as "comprising", "including" and "having" are to be interpreted in the same manner as "consisting of", "consisting essentially of" and "substantially consisting of" under 35 U.S.C. § 112, Paragraph 6, as that terminology is interpreted in the context of the specification as a whole. Additionally, the terms "a" and "an" are to be construed as "one or more" when used in this specification. Moreover, the use of any terms "or" is to be interpreted as "and / or" unless and except the context clearly indicates otherwise.

[0177] Those skilled in the art will further appreciate that the various illustrative logical blocks, modules, and steps described in connection with the embodiments disclosed herein can be implemented by electronic hardware, computer software, or combinations of both. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the overall system. Skilled artisans can implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present embodiments.

[0178] The various illustrative logical blocks, modules, and steps described in connection with the embodiments disclosed herein can be implemented or performed by a general purpose processor, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. A general purpose processor can be a microprocessor, but in the alternative, the general purpose processor can be any conventional processor, controller, microcontroller, or state machine. A processor can also be implemented as a combination of computing devices, e.g., a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration.

[0179] The steps of a method or algorithm described in connection with the embodiments disclosed herein can be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. A software module can reside in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor such the processor can read information from, and write information to, the storage medium. In the alternative, the storage medium can be integral to the processor. The processor and the storage medium can reside in an ASIC. The ASIC can reside in a user terminal. In the alternative, the processor and the storage medium can reside as discrete components in a user terminal.

[0180] In one or more exemplary designs, the functions described can be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions can be stored on or transmitted over as one or more instructions or code on a computer-readable medium. Computer-readable media include both computer storage media and communication media including any medium that facilitates transfer of a computer program from one place to another. Storage media can be any available media that can be accessed by a general purpose or special purpose computer. By way of example, such computer-readable media can comprise RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to carry or store desired program code means in the form of instructions or data structures and that can be accessed by a general-purpose or special-purpose computer, or a general-purpose or special-purpose processor. Also, any connection is properly termed a computer-readable medium. For example, if the software is transmitted from a website, server, or other remote source using a coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or data

[0181] The above detailed description describes the purpose, technical solutions and advantages of the present application. It should be understood that the above description is only a specific embodiment of the present application and is not intended to limit the scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application shall be included in the scope of the present application.

Claims

1. A method for monitoring failure of autonomous driving perception hardware, characterized in that: include: Based on the encoder, the perceptual hardware data is compressed to obtain the corresponding low-dimensional potential representation; The perception hardware includes at least: a steering sensor, a brake sensor, a throttle position sensor, a laser radar, and a camera; Establish a perception hardware data model based on the latent variables potentially represented by the perception hardware data; Introducing the conditional probability distribution between perception hardware, and building an elastic probability map between different perception hardware through the perception hardware data model; Combined with elastic probability graphs, each sensing hardware is monitored in real time and risk assessment is performed; The encoder-based compression of the perceptual hardware data to obtain the corresponding low-dimensional potential representation also includes: The encoder maps the perceptual hardware training data to the latent space to obtain the corresponding latent representation; The decoder converts the latent representation into original data; Based on the latent representation and the original data, the encoder is trained by maximizing the marginal log-likelihood of the perceptual hardware training data using a variational lower bound; The establishment of a perception hardware data model based on latent variables potentially represented by the perception hardware data specifically includes: Introducing latent variables that potentially represent the perception hardware data and calculating a priori distribution of the latent variables; The corresponding joint distribution is obtained by multiplying the marginal distribution of the latent variable and the prior distribution; Using Bayes' theorem, the corresponding posterior distribution is calculated through the marginal distribution and prior distribution of the latent variable; The perception hardware data model is represented by the posterior distribution of the perception hardware latent variables; The introduction of the conditional probability distribution between the perception hardware and the construction of the elasticity probability map between different perception hardware through the perception hardware data model specifically include: Each node of the elastic probability graph is represented by a different sensing hardware data model, and each sensing hardware corresponds to a parent node set; Conditional probability distribution is introduced, and the parent node corresponding to each sensing hardware is combined to calculate the joint probability distribution of different sensing hardware with dependencies. The edges corresponding to different nodes of the elastic probability graph are represented by the joint probability distribution of different sensing hardware under given conditions; The real-time monitoring and risk assessment of each sensing hardware using the elastic probability graph specifically includes: Using the elastic probability graph, calculate the likelihood probability of the target sensing hardware under the current conditions; the likelihood probability is the joint probability distribution of the target sensing hardware and the sensing hardware with which it has a dependent relationship; Compare the likelihood probability of the target perception hardware under current conditions with the likelihood probability under normal working conditions, and use the set threshold to determine whether the current perception hardware is abnormal.

2. The method for monitoring failure of autonomous driving perception hardware according to claim 1, characterized in that: The constructing of the elastic probability map between different perception hardware further includes: Utilize existing data to learn parameters of elastic probability maps; The learning goal is to maximize the joint probability distribution of different sensing hardware with dependencies under given conditions in the elastic probability graph; A complete elastic probability map is constructed from the parameters.

3. The method for monitoring failure of autonomous driving perception hardware according to claim 1, characterized in that: The real-time monitoring and risk assessment of each sensing hardware in combination with the elastic probability graph also includes: If the current perception hardware data is abnormal, the likelihood probability of the current perception hardware under abnormal conditions is calculated based on the elastic probability graph; According to the likelihood probability of the current perception hardware under abnormal conditions, the risk assessment index of the current perception hardware is calculated.

4. The method for monitoring failure of autonomous driving perception hardware according to claim 3, characterized in that: The joint probability distribution of the different sensing hardware with dependencies is: （10） In formula (10), represents the conditional probability distribution between nodes, V Represents a collection of nodes, and Represents two hardware-aware data models with dependencies, Representation node The parent node set of Representation node The parent node collection.

5. An autonomous driving perception hardware failure monitoring device, characterized in that: include: A deep feature extraction system, which compresses the perceptual hardware data based on the encoder to obtain the corresponding low-dimensional latent representation; The perception hardware includes at least: a steering sensor, a brake sensor, a throttle position sensor, a laser radar, and a camera; A latent variable system for building a perception hardware data model based on latent variables potentially representing the perception hardware data; The elastic probability graph system is used to introduce the conditional probability distribution between perception hardware and build elastic probability graphs between different perception hardware through the perception hardware data model; Anomaly detection and risk assessment system, which is used to monitor each sensing hardware in real time and conduct risk assessment based on elastic probability graphs; The encoder-based compression of the perceptual hardware data to obtain the corresponding low-dimensional potential representation also includes: The encoder maps the perceptual hardware training data to the latent space to obtain the corresponding latent representation; The decoder converts the latent representation into original data; Based on the latent representation and the original data, the encoder is trained by maximizing the marginal log-likelihood of the perceptual hardware training data using a variational lower bound; The establishment of a perception hardware data model based on latent variables potentially represented by the perception hardware data specifically includes: Introducing latent variables that potentially represent the perception hardware data and calculating a priori distribution of the latent variables; The corresponding joint distribution is obtained by multiplying the marginal distribution of the latent variable and the prior distribution; Using Bayes' theorem, the corresponding posterior distribution is calculated through the marginal distribution and prior distribution of the latent variable; The perception hardware data model is represented by the posterior distribution of the perception hardware latent variables; The introduction of the conditional probability distribution between the perception hardware and the construction of the elasticity probability map between different perception hardware through the perception hardware data model specifically include: Each node of the elastic probability graph is represented by a different sensing hardware data model, and each sensing hardware corresponds to a parent node set; Conditional probability distribution is introduced, and the parent node corresponding to each sensing hardware is combined to calculate the joint probability distribution of different sensing hardware with dependencies. The edges corresponding to different nodes of the elastic probability graph are represented by the joint probability distribution of different sensing hardware under given conditions; The real-time monitoring and risk assessment of each sensing hardware using the elastic probability graph specifically includes: Using the elastic probability graph, calculate the likelihood probability of the target sensing hardware under the current conditions; the likelihood probability is the joint probability distribution of the target sensing hardware and the sensing hardware with which it has a dependent relationship; Compare the likelihood probability of the target perception hardware under current conditions with the likelihood probability under normal working conditions, and use the set threshold to determine whether the current perception hardware is abnormal.

6. An autonomous driving perception hardware failure monitoring device, characterized in that: include: A processor and a memory, wherein the processor reads a computer program in the memory and is configured to perform the following operations: The encoder compresses the perception hardware data to obtain a corresponding low-dimensional potential representation; the perception hardware includes at least a steering sensor, a brake sensor, a throttle position sensor, a lidar, and a camera; Establish a perception hardware data model based on the latent variables potentially represented by the perception hardware data; Introducing the conditional probability distribution between perception hardware, and building an elastic probability map between different perception hardware through the perception hardware data model; Combined with elastic probability graphs, each sensing hardware is monitored in real time and risk assessment is performed; The encoder-based compression of the perceptual hardware data to obtain the corresponding low-dimensional potential representation also includes: The encoder maps the perceptual hardware training data to the latent space to obtain the corresponding latent representation; The decoder converts the latent representation into original data; Based on the latent representation and the original data, the encoder is trained by maximizing the marginal log-likelihood of the perceptual hardware training data using a variational lower bound; The establishment of a perception hardware data model based on latent variables potentially represented by the perception hardware data specifically includes: Introducing latent variables that potentially represent the perception hardware data and calculating a priori distribution of the latent variables; The corresponding joint distribution is obtained by multiplying the marginal distribution of the latent variable and the prior distribution; Using Bayes' theorem, the corresponding posterior distribution is calculated through the marginal distribution and prior distribution of the latent variable; The perception hardware data model is represented by the posterior distribution of the perception hardware latent variables; The introduction of the conditional probability distribution between the perception hardware and the construction of the elasticity probability map between different perception hardware through the perception hardware data model specifically include: Each node of the elastic probability graph is represented by a different sensing hardware data model, and each sensing hardware corresponds to a parent node set; Conditional probability distribution is introduced, and the parent node corresponding to each sensing hardware is combined to calculate the joint probability distribution of different sensing hardware with dependencies. The edges corresponding to different nodes of the elastic probability graph are represented by the joint probability distribution of different sensing hardware under given conditions; The real-time monitoring and risk assessment of each sensing hardware using the elastic probability graph specifically includes: Using the elastic probability graph, calculate the likelihood probability of the target sensing hardware under the current conditions; the likelihood probability is the joint probability distribution of the target sensing hardware and the sensing hardware with which it has a dependent relationship; Compare the likelihood probability of the target perception hardware under current conditions with the likelihood probability under normal working conditions, and use the set threshold to determine whether the current perception hardware is abnormal.

Images