Automatic driving sensing hardware function failure monitoring method and device

Abstract

The invention relates to an automatic driving perception hardware function failure monitoring method and an automatic driving perception hardware function failure monitoring device, and belongs to the technical field of deep learning. The method comprises the following steps: compressing sensing hardware data based on an encoder, and obtaining a corresponding low-dimensional potential representation; establishing a sensing hardware data model according to hidden variables potentially represented by the sensing hardware data; introducing conditional probability distribution between sensing hardware, and constructing an elastic probability graph between different sensing hardware through a sensing hardware data model; and performing real-time monitoring and risk assessment on each sensing hardware in combination with the elastic probability graph. According to the invention, the complex relationship among the sensing hardware can be comprehensively considered, and the safety of the automatic driving system can be monitored in real time.

Description

Technical Field

[0001] The present invention relates to a method for monitoring the functional failure of autonomous driving perception hardware, and also to a device for monitoring the functional failure of autonomous driving perception hardware, belonging to the field of deep learning technology. Background Art

[0002] With the rapid development of artificial intelligence (AI) technology, autonomous vehicles are gradually moving from concept to reality. However, their safety remains a key concern. Perception hardware is a crucial component of autonomous driving systems, responsible for collecting and interpreting information about the surrounding environment. However, these sensor systems are not immune to failure, and safety risks caused by malfunctioning perception hardware have become a major challenge hindering the development of autonomous driving technology.

[0003] Perception hardware failure can be caused by a variety of reasons, including sensor damage, changing environmental conditions, inclement weather, and electromagnetic interference. When this hardware fails, autonomous vehicles may be unable to accurately perceive their surroundings, leading to misjudgments, incorrect decisions, and even accidents. Therefore, timely monitoring of the perception hardware's status and implementing appropriate safety measures are crucial to ensuring the safety of autonomous vehicles.

[0004] Currently, diagnostic methods for sensing hardware failures in sensor systems fall into two main categories: internal diagnosis and external monitoring. Internal diagnostic methods typically integrate a self-checking mechanism within the sensor system, performing regular self-checks to monitor the sensor's operating status. This involves self-verifying the real-time sensor output data to detect anomalies or values that deviate from the normal range. Furthermore, the system may be configured with multiple sensors of the same or different types, comparing their outputs to detect any anomalies. If one sensor fails, other functioning sensors can provide backup information.

[0005] External monitoring-based methods use additional external monitoring devices, such as cameras and lidar, to monitor the actual working conditions of the sensors. These devices can capture environmental information around the sensors and compare it with the sensor output.

[0006] Although these methods can detect sensor failures to a certain extent, they still have some limitations. Current technologies mainly focus on detecting simple faults of single sensors, such as disconnection and offset. However, existing technologies have difficulty dealing with complex faults, such as incorrect interpretation of sensor data and errors caused by environmental changes. In addition, existing technologies often lack comprehensive risk monitoring and assessment of the entire system configuration and algorithm. Since autonomous driving systems usually rely on multiple sensors and algorithms working together, the failure of a single sensor may have a chain reaction on the entire system. Therefore, a more comprehensive and comprehensive method is needed to assess the safety risks of the system in the event of sensor failure to better ensure the safety of autonomous vehicles. Summary of the Invention

[0007] The technical problem to be solved by the present invention is to overcome the shortcomings of existing technologies and provide a method and device for monitoring the functional failure of autonomous driving perception hardware, which can comprehensively take into account the complex relationship between various perception hardware and monitor the safety of the autonomous driving system in real time.

[0008] To achieve the above technical objectives, the present invention provides a method for monitoring autonomous driving perception hardware failure, comprising: Based on the encoder, the perceptual hardware data is compressed to obtain the corresponding low-dimensional potential representation; Establish a perception hardware data model based on the latent variables potentially represented by the perception hardware data; Introducing the conditional probability distribution between perception hardware, and building an elastic probability map between different perception hardware through the perception hardware data model; Combined with the elastic probability graph, each sensing hardware is monitored in real time and risk assessment is performed.

[0009] Preferably, the encoder-based compression of the perceptual hardware data to obtain the corresponding low-dimensional potential representation further includes: The encoder maps the perceptual hardware training data to the latent space to obtain the corresponding latent representation; The decoder converts the latent representation into original data; Based on the latent representation and the original data, the encoder is trained using a variational lower bound to maximize the marginal log-likelihood of the perceptual hardware training data.

[0010] Preferably, the method of establishing a perception hardware data model based on latent variables potentially represented by the perception hardware data specifically includes: Introducing latent variables that potentially represent the perception hardware data and calculating a priori distribution of the latent variables; The corresponding joint distribution is obtained by multiplying the marginal distribution of the latent variable and the prior distribution; Using Bayes' theorem, the corresponding posterior distribution is calculated through the marginal distribution and prior distribution of the latent variable; The perception hardware data model is represented by the posterior distribution of the perception hardware latent variables.

[0011] Preferably, the conditional probability distribution between the sensing hardware is introduced, and an elastic probability map between different sensing hardware is constructed through the sensing hardware data model, specifically including: Each node of the elastic probability graph is represented by a different sensing hardware data model, and each sensing hardware corresponds to a parent node set; Conditional probability distribution is introduced, and the parent node corresponding to each sensing hardware is combined to calculate the joint probability distribution of different sensing hardware with dependencies. The edges corresponding to different nodes of the elastic probability graph are represented by the joint probability distribution of different sensing hardware under given conditions.

[0012] Preferably, the constructing of the elasticity probability map between different sensing hardware further includes: Utilize existing data to learn parameters of elastic probability maps; The learning goal is to maximize the joint probability distribution of different sensing hardware with dependencies under given conditions in the elastic probability graph; A complete elastic probability map is constructed from the parameters.

[0013] Preferably, the real-time monitoring and risk assessment of each sensing hardware in combination with the elastic probability map specifically includes: Using the elastic probability graph, calculate the likelihood probability of the target sensing hardware under the current conditions; the likelihood probability is the joint probability distribution of the target sensing hardware and the sensing hardware with which it has a dependent relationship; Compare the likelihood probability of the target perception hardware under current conditions with the likelihood probability under normal working conditions, and use the set threshold to determine whether the current perception hardware is abnormal.

[0014] Preferably, the real-time monitoring and risk assessment of each sensing hardware in combination with the elastic probability graph further includes: If the current perception hardware data is abnormal, the likelihood probability of the current perception hardware under abnormal conditions is calculated based on the elastic probability graph; According to the likelihood probability of the current perception hardware under abnormal conditions, the risk assessment index of the current perception hardware is calculated.

[0015] Preferably, the joint probability distribution of the different sensing hardware with dependencies is: (10) In formula (10), represents the conditional probability distribution between nodes,V Represents a collection of nodes, and Represents two hardware-aware data models with dependencies, Representation node The parent node set of Representation node The parent node collection.

[0016] In another aspect, the present invention provides an autonomous driving perception hardware function failure monitoring device, comprising: A deep feature extraction system, which compresses the perceptual hardware data based on the encoder to obtain the corresponding low-dimensional latent representation; A latent variable system for building a perception hardware data model based on latent variables potentially representing the perception hardware data; The elastic probability graph system is used to introduce the conditional probability distribution between perception hardware and build elastic probability graphs between different perception hardware through the perception hardware data model; The anomaly detection and risk assessment system is used to monitor each sensing hardware in real time and perform risk assessment in combination with elastic probability graphs.

[0017] In another aspect, the present invention provides an autonomous driving perception hardware failure monitoring device, comprising: a processor and a memory, wherein the processor reads a computer program in the memory and is configured to perform the following operations: Based on the encoder, the perceptual hardware data is compressed to obtain the corresponding low-dimensional potential representation; Establish a perception hardware data model based on the latent variables potentially represented by the perception hardware data; Introducing the conditional probability distribution between perception hardware, and building an elastic probability map between different perception hardware through the perception hardware data model; Combined with the elastic probability graph, each sensing hardware is monitored in real time and risk assessment is performed.

[0018] This invention, through the introduction of latent variable modeling and the construction of a resilient probabilistic graph, comprehensively considers the complex relationships between sensors. This goes beyond simple fault detection of a single sensing hardware component and takes into account the cascading effects between different sensors. This resilient probabilistic graph model comprehensively considers the risks of the entire autonomous driving system configuration and algorithm, further ensuring the overall safety of the system. BRIEF DESCRIPTION OF THE DRAWINGS In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0019] Figure 1 This is a schematic diagram of the method flow of an embodiment of the present application; Figure 2 This is a schematic diagram of the structure of the device in the embodiment of the present application; Figure 3 Schematic diagram of the structure of the system in the embodiment of the present application. DETAILED DESCRIPTION

[0020] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0021] Typically, an autonomous driving system is equipped with a variety of perception hardware, including steering sensors, brake sensors, throttle opening sensors, lidar, cameras, etc.

[0022] The present invention aims to solve the safety risk problem of failure of the perception hardware function of the autonomous vehicle, such as Figure 1 As shown in the figure, a method for monitoring the failure of autonomous driving perception hardware is proposed, including: 101. Compress the perceptual hardware data based on the encoder to obtain the corresponding low-dimensional potential representation; 102. Establish a perception hardware data model based on the latent variables potentially represented by the perception hardware data; 103. Introduce the conditional probability distribution between perception hardware and build the elastic probability map between different perception hardware through the perception hardware data model; 104. Combined with the elastic probability map, each sensing hardware is monitored in real time and risk assessment is performed.

[0023] This embodiment, by introducing latent variable modeling and constructing a resilient probability graph, comprehensively considers the complex relationships between sensors. This goes beyond simple fault detection of a single sensing hardware component and takes into account the cascading effects between different sensors. This resilient probability graph model comprehensively considers the risks of the entire autonomous driving system configuration and algorithm, further ensuring the overall safety of the autonomous driving system.

[0024] The 101 mentioned above also includes: 201. The encoder maps the perception hardware training data to the latent space to obtain the corresponding latent representation; 202. The decoder restores the latent representation to original data; 203. Based on the latent representation and the original data, use a variational lower bound to maximize the marginal log-likelihood function of the perceptual hardware training data to train the encoder.

[0025] In this embodiment, in step 201, the posterior distribution of the potential representation W of the autonomous driving system perception hardware data X is defined as , and then use an encoder neural network to parameterize this distribution, where the posterior distribution of W given X is a multivariate normal distribution: (1) In formula (1), and are the mean and variance of the output from the encoder neural network.

[0026] In the above 202, the prior distribution of the potential representation W of the hardware data X is defined as , which is usually chosen to be the standard normal distribution: (2) The process of generating raw data for perceptual hardware can be represented as a decoder , parameterized by a decoder neural network.

[0027] Assume that the likelihood distribution of W given X is a multivariate normal distribution: (3) In formula (3), and are the mean and variance of the output from the decoder neural network.

[0028] In 203 , a variational lower bound is used to maximize the marginal log-likelihood function of the perception hardware data.

[0029] The variational lower bound is: (4) In formula (4), represents KL divergence; is the marginal probability distribution of X; (5) In formula (5), i Indicates the first i dimensions, Indicates the i The variance of the dimensions, Indicates the i The mean of the dimensions; (6) In formula (6), represents the expectation of the posterior distribution of the latent variable, Indicates the i The variance of the output in each dimension, The first i dimension, Indicates the i The variance of the output in each dimension; By maximizing the variational lower bound, the parameters of the encoder and decoder are optimized; ultimately, the trained encoder can produce a low-dimensional latent representation of perceptual hardware data.

[0030] In this embodiment, the encoder may use a variational atomic encoder (VAE) to process the perceptual hardware data to obtain a low-dimensional latent representation of the perceptual hardware data.

[0031] The 102 specifically includes: 1021. Introduce latent variables that potentially represent the perception hardware data and calculate the prior distribution of the latent variables; 1022. Obtain a corresponding joint distribution by multiplying the marginal distribution of the latent variable and the prior distribution; 1023. Using Bayes’ theorem, calculate the corresponding posterior distribution through the marginal distribution and prior distribution of the latent variable; 1024. The perception hardware data model is represented by the posterior distribution of the perception hardware latent variables.

[0032] In this embodiment, in step 1021, the latent variable Z of the low-dimensional representation W of the perception hardware data X is first introduced. The latent variable Z is used to describe the implicit relationship between different perception hardware. The prior distribution of the latent variable Z is , which is usually chosen to be the standard normal distribution: (7) Consider the joint distribution between the perception hardware data X and the latent variable Z .

[0033] The joint distribution It can be expressed as a marginal distribution With prior distribution The product of: (8) Use Bayes' theorem to calculate the posterior distribution between perception hardware data X and latent variables Z : (9) In formula (9), the denominator is the normalization factor, which can be calculated by marginalizing the latent variable Z of the known hardware data.

[0034] Typically, the posterior distribution It is generally difficult to express analytically, so variational inference or maximum likelihood function estimation is used to learn the model parameters so that the optimized posterior distribution is closer to the true distribution. The likelihood function is the marginal distribution Optimized posterior distribution To represent the perception hardware data model.

[0035] In this embodiment, by introducing latent variables to model sensor hardware data, the diversity of sensor hardware data is more comprehensively considered, enabling better capture of the complex relationships between different sensor hardware components, thereby improving the accuracy of safety risk monitoring for sensor hardware functional failures. The method described in this embodiment can effectively handle complex faults, such as errors caused by incorrect interpretation of sensor hardware data and environmental changes.

[0036] The 103 specifically includes: 1031. Each node of the elastic probability graph is represented by a different sensing hardware data model, and each sensing hardware corresponds to a parent node set; 1032. Introduce conditional probability distribution, combine the parent nodes corresponding to each sensing hardware, and calculate the joint probability distribution of different sensing hardware with dependencies; 1033. The edges corresponding to different nodes of the elastic probability graph are represented by the joint probability distribution of different sensing hardware under given conditions.

[0037] In 1031, the posterior distribution obtained from 102 This is used to reflect changes in the probability distribution of sensor hardware failures over time or after an event. In 1031, each sensor hardware component is considered a node, and the parent node of that sensor hardware component is characterized by combining that node with the corresponding posterior distribution. The parent node can include all sensor hardware components of the autonomous driving system, or all sensor hardware components of the same type as the sensor hardware component.

[0038] Introducing conditional probability distribution Represents the dependency relationship between nodes. For each pair of nodes, it can be expressed as or .

[0039] The elastic probability graph can be represented as a graph , where V is the set of nodes, corresponding to the sensing hardware and latent variables, and E is the set of edges, corresponding to the probabilistic dependencies between nodes.

[0040] The joint probability distribution of the different sensing hardware with dependencies is: (10) In formula (10), represents the conditional probability distribution between nodes, V Represents a collection of nodes, and Represents two hardware-aware data models with dependencies, Representation node The parent node set of Representation node The parent node collection.

[0041] Said 103 further includes: 1034. Parameter learning of elastic probability maps using existing data; 1035. The learning goal is to maximize the joint probability distribution of different sensing hardware with dependencies under given conditions in the elastic probability graph; 1036 constructs a complete elastic probability map using the parameters.

[0042] In this embodiment, existing data is used for parameter learning, i.e., estimating the parameters in the probability distribution. The goal of parameter learning is to maximize the likelihood function , where θ is the parameter of the probability distribution, and the optimization algorithm such as gradient ascent method is used to solve the parameter.

[0043] For example, construct a complete elastic probability graph G containing cameras and lidar Two nodes, set and Represents the camera and lidar Conditional probability distribution of data. Here, the parent node set and ,The perception hardware model obtained according to 102 may include other perception hardware data or latent variables.

[0044] In the parameter learning process, the existing camera and lidar The data is used for parameter learning to estimate the parameters of their conditional probability distribution. Assuming that Gaussian distribution is used to model these conditional probability distributions, the camera data and lidar data can be represented as and , and assume that their probability distributions under given conditions are and , then, the parameters of these probability distributions can be estimated using maximum likelihood estimation.

[0045] Assuming camera data and lidar data The conditional probability distributions of are multivariate Gaussian distributions, whose parameters are the mean vector and covariance matrix , The goal is to maximize the likelihood function, that is, to maximize the conditional probability given the data. Assuming there are N sets of known camera and lidar data, the following likelihood function can be obtained: (11) In formula (11), and Respectively represent Set camera and lidar data, and Represent the parent node sets of camera and lidar respectively. Since Gaussian distribution is assumed, the above probability can be represented by the density function of multivariate Gaussian distribution.

[0046] Based on the learned probabilistic dependencies and parameters, a complete camera can be constructed and lidar In this example, we can get the elastic probability map between the camera and lidar The conditional probability distribution between . Suppose we find that the camera output is affected by the lidar data, or that both are affected by a latent variable. In the elastic probability graph, there will be corresponding edges to represent this dependency.

[0047] Specifically, the camera and lidar can be treated as nodes in a graph, with an edge established between them, indicating that the camera output is affected by the lidar data. The conditional probability distribution on this edge is set based on the learned probability distribution parameters. For example, the estimated parameters of a multivariate Gaussian distribution can be used to describe this dependency. This results in a complete elastic probability graph that captures the global state of the camera and lidar sensor system.

[0048] In this embodiment, a complete elastic probability graph is constructed based on the learned probabilistic dependencies. The edges in the graph represent the conditional probability distribution between nodes, and the nodes correspond to perception hardware data and / or hidden variables; the topological structure of the elastic probability graph reflects the dependencies in the perception hardware system, which helps to fully consider the state of the system. By constructing an elastic probability graph, the dependencies between nodes in the perception hardware system can be more comprehensively modeled, thereby more accurately characterizing the state of the entire system. The method described in this embodiment can help monitor anomalies in sensor data in real time and provide a reliable assessment of the safety risk of perception hardware function failure.

[0049] The above 104 includes two parts: anomaly detection and risk assessment. The anomaly detection part specifically includes: 1041. Calculate the likelihood probability of the target sensing hardware under the current conditions using the elastic probability graph; the likelihood probability is the joint probability distribution of the target sensing hardware and the sensing hardware with which it has a dependent relationship; 1042. Compare the likelihood probability of the target perception hardware under current conditions with the likelihood probability under normal working conditions, and determine whether the current perception hardware is abnormal through a set threshold.

[0050] In this embodiment, the learned complete elastic probability graph model is used to calculate the likelihood probability of the target perception hardware under the current conditions. , where data represents the current conditions.

[0051] Compare the current likelihood probability with the likelihood probability under normal working conditions, and determine whether there is an abnormality by setting a threshold: (12) In formula (12), Threshold represents the likelihood probability under normal working conditions, that is, the threshold. If the threshold is not exceeded, the target sensing hardware is normal; if the threshold is exceeded, it is abnormal.

[0052] The 104 also includes a risk assessment section, specifically including: 1043. If the current perception hardware data is abnormal, calculate the likelihood probability of the current perception hardware under abnormal conditions based on the elastic probability map; 1044. Calculate the risk assessment index of the current perception hardware based on the likelihood probability of the current perception hardware under abnormal conditions.

[0053] In this embodiment, if an anomaly is detected, the overall risk of the system is evaluated by considering the node dependencies in the elastic probability graph; the conditional probability distribution of the system in the case of sensor failure is calculated. . System risk is quantified based on specific risk assessment indicators: conditional entropy and conditional variance.

[0054] (13) In formula (13), Anomaly represents abnormal conditions.

[0055] In the process of comprehensive system status determination, two risk assessment indicators, conditional entropy and conditional variance, are used to quantify system risk.

[0056] For random variables X and Y, their conditional entropy Expresses the uncertainty of X given Y: (14) In formula (14), for X and Y The joint probability distribution of for Y The probability distribution of For random variables X and Y, their conditional variance is Indicates the degree of dispersion of X under the condition of given Y: (15) In formula (15), For a given Y = y Under the conditions X The variance of .

[0057] Therefore, in the comprehensive system state judgment, the conditional entropy and conditional variance are comprehensively considered to construct the comprehensive risk assessment index Z: (16) In formula (2), Representation-aware hardware X and perception hardware Y Conditional entropy under abnormal conditions, express X and Y The conditional variance of The coefficient representing the trade-off between conditional entropy and conditional variance.

[0058] By monitoring actual sensor data, we calculate the conditional entropy and conditional variance, and substitute them into the formula for the comprehensive risk assessment indicator Z. If Z exceeds a pre-set threshold, the perception hardware system is deemed to be at risk. Otherwise, the system is considered normal.

[0059] Specifically, conditional entropy measures the uncertainty of system state predictions—that is, the degree of uncertainty in the prediction of the system state under a given set of conditions. Conditional variance, on the other hand, reflects the degree of variability in the system state prediction under given conditions. By combining these two metrics, the comprehensive risk assessment metric Z can more comprehensively quantify the system's security risk.

[0060] Specifically, suppose an autonomous driving system is equipped with one lidar, four surround-view cameras, one main camera, and four millimeter-wave radars. Feedback data measures the conditional entropy and conditional variance of each perception hardware under different conditions. Then, based on the comprehensive risk assessment metric Z, the feedback data comprehensively considers the risks of these perception devices to assess the safety of the entire perception system. If the lidar's conditional entropy is high under certain conditions, it indicates that its perception results are relatively uncertain; conversely, a large conditional variance indicates that its perception results vary significantly under those conditions. This results in a relatively high comprehensive risk assessment metric Z for that lidar under those conditions. By performing a similar comprehensive assessment of all perception devices, the feedback data can be used to determine the risk status of the entire autonomous vehicle's perception system.

[0061] The method described in this embodiment can be summarized as follows: Step 1: Obtain raw data from the autonomous driving system’s various sensing hardware. Step 2: Use the variational autoencoder in deep learning to process the perception hardware data to achieve a low-dimensional, highly abstract representation of the data; Step 3: Introduce latent variables to model the implicit relationships between perception hardware to more comprehensively consider the diversity and complexity of perception hardware data; Step 4: Based on the dependencies between the sensing hardware, a resilient probability graph is constructed to describe the global state of the sensor system in detail. Step 5: In the elastic probability graph system, the relationship between the probability graphs is learned through parameter learning so that it can accurately reflect the complexity of the perception system; Step 6: Using the elastic probabilistic graph model, the system monitors anomalies in the perception hardware data in real time and detects anomalies caused by perception hardware failures. Step 7: Based on anomaly detection, conduct risk assessment and conduct in-depth analysis of the impact that the failure may have on the entire system; Step 8: Combine the results of deep feature extraction, latent variable modeling, and elastic probability map construction to determine whether the overall system status is normal.

[0062] like Figure 3 As shown, the embodiment of the present application also provides an autonomous driving perception hardware function failure monitoring device, comprising: A deep feature extraction system, which compresses the perceptual hardware data based on the encoder to obtain the corresponding low-dimensional latent representation; A latent variable system for building a perception hardware data model based on latent variables potentially representing the perception hardware data; The elastic probability graph system is used to introduce the conditional probability distribution between perception hardware and build elastic probability graphs between different perception hardware through the perception hardware data model; The anomaly detection and risk assessment system is used to monitor each sensing hardware in real time and perform risk assessment in combination with elastic probability graphs.

[0063] This embodiment consists of four systems: a deep feature extraction system, a latent variable system, an elastic probability graph system, and an anomaly detection and risk assessment system. First, the deep feature extraction system uses a variational autoencoder to learn a low-dimensional, highly abstract representation of sensor data to capture key features. Next, the latent variable system introduces latent variables and models implicit relationships between sensors to more comprehensively account for the diversity and complexity of sensor data. Subsequently, the elastic probability graph system constructs an elastic probability graph based on the dependencies between sensors to characterize the global state of the sensor system. Finally, the anomaly detection and risk assessment system uses the elastic probability graph model to monitor anomalies in sensor data in real time and further perform risk assessment to determine the potential impact of a failure on the entire system.

[0064] By integrating four systems, the device provided in this embodiment improves the limitations of current technology in sensor fault detection by comprehensively considering the complexity of multi-sensor collaborative operation, and provides a higher level of protection for the safety risk monitoring of perception hardware function failures in autonomous vehicles.

[0065] like Figure 2 As shown, the embodiment of the present application further provides an autonomous driving perception hardware function failure monitoring device, comprising: a memory 10 and a processor 20, wherein the processor 20 reads a computer program in the memory 10 and performs the following operations: Based on the encoder, the perceptual hardware data is compressed to obtain the corresponding low-dimensional potential representation; Establish a perception hardware data model based on the latent variables potentially represented by the perception hardware data; Introducing the conditional probability distribution between perception hardware, and building an elastic probability map between different perception hardware through the perception hardware data model; Combined with the elastic probability graph, each sensing hardware is monitored in real time and risk assessment is performed.

[0066] The specific implementation has been described in detail in the above embodiments and will not be repeated here.

[0067] It should be understood that the specific order or hierarchy of steps in the disclosed processes is an example of an exemplary method. Based on design preferences, it should be understood that the specific order or hierarchy of steps in the process can be rearranged without departing from the scope of the present disclosure. The accompanying method claims present elements of the various steps in an exemplary order and are not intended to be limited to the specific order or hierarchy described.

[0068] While the above descriptions of embodiments and examples of the present invention are provided for the purpose of providing a more detailed and complete description of the present disclosure, they are not intended to be the only ways to implement or use the embodiments of the present invention. The embodiments cover features of various embodiments, as well as the method steps and sequences for constructing and operating these embodiments. However, other embodiments may be used to achieve the same or equivalent functionality and sequence of steps.

[0069] In the foregoing detailed description, various features are grouped together in a single embodiment to simplify the disclosure. This method of disclosure should not be interpreted as reflecting an intention that embodiments of the claimed subject matter require more features than are expressly recited in each claim. On the contrary, as reflected in the appended claims, the invention comprises less than all the features of any individual disclosed embodiment. The appended claims are hereby expressly incorporated into the detailed description, with each claim standing on its own as a separate preferred embodiment of the invention.

[0070] The above description of the disclosed embodiments is intended to enable any person skilled in the art to implement or use the present invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other embodiments without departing from the spirit and scope of the present disclosure. Therefore, the present disclosure is not limited to the embodiments presented herein but is intended to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0071] The foregoing description includes examples of one or more embodiments. Of course, it is not possible to describe all possible combinations of components or methods for the purposes of describing the above embodiments, but one of ordinary skill in the art will recognize that the various embodiments may be further combined and arranged. Therefore, the embodiments described herein are intended to encompass all such changes, modifications and variations that fall within the scope of the appended claims. Furthermore, to the extent the term "comprising" is used in the specification or claims, the term is intended to be encompassed in a manner similar to the term "including," as explained in terms of "including," used as a transitional word in the claims. Furthermore, any use of the term "or" in the specification of the claims is intended to mean a "non-exclusive or."

[0072] Those skilled in the art will also understand that the various illustrative logical blocks, units, and steps listed in the embodiments of the present invention can be implemented by electronic hardware, computer software, or a combination of the two. To clearly demonstrate the interchangeability of hardware and software, the various illustrative components, units, and steps mentioned above have generally described their functions. Whether such functions are implemented by hardware or software depends on the specific application and the design requirements of the entire system. Those skilled in the art may use various methods to implement the described functions for each specific application, but such implementation should not be construed as exceeding the scope of protection of the embodiments of the present invention.

[0073] The various illustrative logic blocks or units described in the embodiments of the present invention may be implemented or operated using a general-purpose processor, a digital signal processor, an application-specific integrated circuit (ASIC), a field-programmable gate array or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof. A general-purpose processor may be a microprocessor, or alternatively, any conventional processor, controller, microcontroller, or state machine. A processor may also be implemented using a combination of computing devices, such as a digital signal processor and a microprocessor, multiple microprocessors, one or more microprocessors combined with a digital signal processor core, or any other similar configuration.

[0074] The steps of the methods or algorithms described in the embodiments of the present invention may be directly embedded in hardware, a software module executed by a processor, or a combination of the two. The software module may be stored in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art. For example, the storage medium may be connected to the processor so that the processor can read information from the storage medium and write information to the storage medium. Alternatively, the storage medium may also be integrated into the processor. The processor and storage medium may be provided in an ASIC, which may be provided in a user terminal. Alternatively, the processor and storage medium may also be provided in different components in the user terminal.

[0075] In one or more exemplary designs, the functions described in the embodiments of the present invention can be implemented in hardware, software, firmware, or any combination of the three. If implemented in software, these functions can be stored on a computer-readable medium or transmitted in the form of one or more instructions or codes on a computer-readable medium. Computer-readable media include computer storage media and communication media that facilitate the transfer of computer programs from one location to another. Storage media can be any usable medium that can be accessed by a general-purpose or specialized computer. For example, such computer-readable media can include, but are not limited to, RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to carry or store program code in the form of instructions, data structures, and other forms readable by a general-purpose or specialized computer or processor. In addition, any connection can be appropriately defined as a computer-readable medium. For example, if software is transmitted from a website, server, or other remote resource via a coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless methods such as infrared, wireless, and microwave, it is also included in the definition of computer-readable media. The aforementioned disks and discs include compact disks, laser disks, optical disks, DVDs, floppy disks, and Blu-ray discs. Disks typically reproduce data magnetically, while discs typically reproduce data optically using lasers. Combinations of the above may also be included in computer-readable media.

[0076] The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for monitoring failure of autonomous driving perception hardware, characterized in that: include: Based on the encoder, the perceptual hardware data is compressed to obtain the corresponding low-dimensional potential representation; Establish a perception hardware data model based on the latent variables potentially represented by the perception hardware data; Introducing the conditional probability distribution between perception hardware, and building an elastic probability map between different perception hardware through the perception hardware data model; Combined with the elastic probability graph, each sensing hardware is monitored in real time and risk assessment is performed.

2. The method for monitoring failure of autonomous driving perception hardware according to claim 1, characterized in that: The encoder-based compression of the perceptual hardware data to obtain the corresponding low-dimensional potential representation also includes: The encoder maps the perceptual hardware training data to the latent space to obtain the corresponding latent representation; The decoder converts the latent representation into original data; Based on the latent representation and the original data, the encoder is trained using a variational lower bound to maximize the marginal log-likelihood of the perceptual hardware training data.

3. The method for monitoring failure of autonomous driving perception hardware according to claim 1, characterized in that: The establishment of a perception hardware data model based on latent variables potentially represented by the perception hardware data specifically includes: Introducing latent variables that potentially represent the perception hardware data and calculating a priori distribution of the latent variables; The corresponding joint distribution is obtained by multiplying the marginal distribution of the latent variable and the prior distribution; Using Bayes' theorem, the corresponding posterior distribution is calculated through the marginal distribution and prior distribution of the latent variable; The perception hardware data model is represented by the posterior distribution of the perception hardware latent variables.

4. The method for monitoring failure of autonomous driving perception hardware according to claim 1, characterized in that: The introduction of the conditional probability distribution between the perception hardware and the construction of the elasticity probability map between different perception hardware through the perception hardware data model specifically include: Each node of the elastic probability graph is represented by a different sensing hardware data model, and each sensing hardware corresponds to a parent node set; Conditional probability distribution is introduced, and the parent node corresponding to each sensing hardware is combined to calculate the joint probability distribution of different sensing hardware with dependencies. The edges corresponding to different nodes of the elastic probability graph are represented by the joint probability distribution of different sensing hardware under given conditions.

5. The method for monitoring failure of autonomous driving perception hardware according to claim 4, characterized in that: The constructing of the elastic probability map between different perception hardware further includes: Utilize existing data to learn parameters of elastic probability maps; The learning goal is to maximize the joint probability distribution of different sensing hardware with dependencies under given conditions in the elastic probability graph; A complete elastic probability map is constructed from the parameters.

6. The method for monitoring failure of autonomous driving perception hardware according to claim 1, characterized in that: The real-time monitoring and risk assessment of each sensing hardware using the elastic probability graph specifically includes: Using the elastic probability graph, calculate the likelihood probability of the target sensing hardware under the current conditions; the likelihood probability is the joint probability distribution of the target sensing hardware and the sensing hardware with which it has a dependent relationship; Compare the likelihood probability of the target perception hardware under current conditions with the likelihood probability under normal working conditions, and use the set threshold to determine whether the current perception hardware is abnormal.

7. The method for monitoring failure of autonomous driving perception hardware according to claim 1, characterized in that: The real-time monitoring and risk assessment of each sensing hardware in combination with the elastic probability graph also includes: If the current perception hardware data is abnormal, the likelihood probability of the current perception hardware under abnormal conditions is calculated based on the elastic probability graph; According to the likelihood probability of the current perception hardware under abnormal conditions, the risk assessment index of the current perception hardware is calculated.

8. The method for monitoring failure of autonomous driving perception hardware according to claim 4, characterized in that: The joint probability distribution of the different sensing hardware with dependencies is: （10） In formula (10), represents the conditional probability distribution between nodes, V Represents a collection of nodes, and Represents two hardware-aware data models with dependencies, Representation node The parent node set of Representation node The parent node collection.

9. An autonomous driving perception hardware failure monitoring device, characterized in that: include: A deep feature extraction system, which compresses the perceptual hardware data based on the encoder to obtain the corresponding low-dimensional latent representation; A latent variable system for building a perception hardware data model based on latent variables potentially representing the perception hardware data; The elastic probability graph system is used to introduce the conditional probability distribution between perception hardware and build elastic probability graphs between different perception hardware through the perception hardware data model; The anomaly detection and risk assessment system is used to monitor each sensing hardware in real time and perform risk assessment in combination with elastic probability graphs.

10. An autonomous driving perception hardware failure monitoring device, characterized in that: include: A processor and a memory, wherein the processor reads a computer program in the memory and is configured to perform the following operations: Based on the encoder, the perceptual hardware data is compressed to obtain the corresponding low-dimensional potential representation; Establish a perception hardware data model based on the latent variables potentially represented by the perception hardware data; Introducing the conditional probability distribution between perception hardware, and building an elastic probability map between different perception hardware through the perception hardware data model; Combined with the elastic probability graph, each sensing hardware is monitored in real time and risk assessment is performed.

Images