Fault diagnosis method and device for perception sensor of unmanned vehicle
By constructing an over-picture of the vehicle-mounted equipment and using a wide-body vehicle fault detection model, the accuracy and efficiency of fault diagnosis of on-board sensing sensors of driverless vehicles are solved, and efficient identification and analysis of faults in harsh environments is achieved.
Patent Information
- Application Number
- CN202411353571.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-26
- Publication Date
- 2024-11-12
AI Technical Summary
Existing unmanned vehicle fault diagnosis technology is difficult to accurately identify and analyze the faults of on-board sensing sensors, especially in harsh environments, which ignore the faults caused by power supply abnormalities, wiring harness connection problems and environmental factors.
Using a wide-body vehicle fault detection model based on graph attention network convolution layer, timing-based neural network and multi-layer perceptron, by constructing a vehicle-mounted device hypergraph, parameter data at multiple moments is obtained and processed, and feature vectors are fused to predict Fault, realize accurate diagnosis of vehicle-based sensing sensor failures.
It realizes accurate and efficient diagnosis of on-board sensing sensor faults of driverless vehicles, and can extract effective fault information from the parameter data of multiple on-board equipment, improving the accuracy and efficiency of fault identification.
Smart Images

Figure CN118928447A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of unmanned driving technology, and in particular to a method and device for diagnosing faults of a perception sensor of an unmanned vehicle. Background Art
[0002] In the field of unmanned vehicles, especially wide-body vehicles that operate 24 hours a day in harsh environments such as mining areas, they are highly dependent on a variety of on-board perception sensors such as lidar, millimeter-wave radar, and cameras to achieve accurate environmental perception and decision-making planning. However, these on-board perception sensors face severe reliability challenges in practical applications. Due to the particularity of the mining environment, on-board perception sensors must not only withstand extreme weather conditions, but also cope with complex terrain and continuous vibration shocks, which makes on-board perception sensors and their related components (such as power supply systems, wiring harness connections, board-end docking, etc.) prone to failure. In related technologies, there are a variety of diagnostic and safety strategy formulation solutions for functional failures of on-board perception sensors.
[0003] However, the existing fault diagnosis technology for autonomous vehicles mainly focuses on whether the on-board perception sensors have faults such as frame loss and disconnection that directly affect their functional use, while ignoring the in-depth analysis and location of faults caused by the on-board perception sensors themselves, power supply anomalies, wiring harness connection problems, board-end docking anomalies, parsing errors, and environmental factors. In addition, the existing fault diagnosis methods have the problems of isolation and single monitoring indicators, making it difficult to accurately identify and analyze complex faults involving multiple software and hardware modules.
[0004] Therefore, how to accurately and efficiently diagnose faults of on-board perception sensors of driverless vehicles is a technical problem that needs to be solved urgently. Summary of the invention
[0005] In view of the above problems existing in the prior art, the present invention provides a method and device for diagnosing faults of perception sensors of unmanned vehicles, so as to accurately and efficiently diagnose faults of on-board perception sensors of unmanned vehicles.
[0006] The present invention provides a method for diagnosing faults of a perception sensor of an unmanned vehicle, comprising: Obtain parameter data of multiple on-board devices of a faulty vehicle at multiple times; wherein, the multiple on-board devices include: an on-board sensing sensor and on-board devices associated with the on-board sensing sensor; according to the parameter data of the multiple on-board devices at each of the multiple times, respectively construct a hypergraph of on-board devices at each time to obtain multiple hypergraphs of on-board devices; wherein, the on-board device hypergraph includes multiple supernodes, each supernode includes a subgraph, the subgraph corresponds to a type of on-board device among the multiple on-board devices, the subgraph includes at least one node, the value of the node is a type of parameter data of the on-board device corresponding to the subgraph; use a wide-body vehicle fault detection model to process the multiple on-board device hypergraphs to obtain a fault prediction result of the faulty vehicle; wherein, the wide-body vehicle fault detection model is obtained by training at least one graph attention network convolutional layer, a time-series-based neural network, and a multi-layer perceptron based on the parameter data of multiple on-board devices of historical faulty vehicles.
[0007] According to a method for diagnosing faults of perception sensors of unmanned vehicles provided by the present invention, the initial values of each supernode in the vehicle-mounted equipment hypergraph are obtained in the following manner: for the parameter data of the multiple vehicle-mounted equipment at each moment, the following operations are performed to determine the initial values of each supernode in the vehicle-mounted equipment hypergraph at the moment: based on the parameter data of each vehicle-mounted equipment, the node values of each subgraph corresponding to the vehicle-mounted equipment in the vehicle-mounted equipment hypergraph are obtained; using a preset algorithm, the node values of each subgraph in the vehicle-mounted equipment hypergraph are processed to obtain the node feature vectors of each subgraph in the vehicle-mounted equipment hypergraph; and the node feature vectors of each subgraph in the vehicle-mounted equipment hypergraph are integrated to obtain the initial values of each supernode in the vehicle-mounted equipment hypergraph.
[0008] According to a method for diagnosing faults of perception sensors of unmanned vehicles provided by the present invention, the method utilizes a wide-body vehicle fault detection model to process the multiple vehicle-mounted equipment hypergraphs to obtain a fault prediction result of the faulty vehicle, including: utilizing the at least one layer of graph attention network convolutional layer to iteratively process each of the vehicle-mounted equipment hypergraphs to obtain a first fused feature vector of each supernode in each of the vehicle-mounted equipment hypergraphs; obtaining a state feature of the vehicle-mounted perception sensor at each moment based on the first fused feature vector of the supernode corresponding to the vehicle-mounted perception sensor in each of the vehicle-mounted equipment hypergraphs; utilizing the time series-based neural network to obtain a time series state feature vector of the vehicle-mounted perception sensor based on the state features of the vehicle-mounted perception sensor at the multiple moments; utilizing the multi-layer perceptron to obtain a fault prediction result of the faulty vehicle based on the time series state feature vector of the vehicle-mounted perception sensor.
[0009] According to a method for diagnosing faults of perception sensors of unmanned vehicles provided by the present invention, the state characteristics of the on-board perception sensors at each moment are obtained based on the first fused feature vector of the supernode corresponding to the on-board perception sensor in each on-board equipment hypergraph, including: performing graph convolution operations and graph pooling operations on the supernodes corresponding to all the on-board perception sensors in each on-board equipment hypergraph to obtain the second fused feature vector of each on-board perception sensor corresponding to the supernode in each on-board equipment hypergraph; using the second fused feature vector of each on-board perception sensor corresponding to the supernode in each on-board equipment hypergraph as the state characteristics of the on-board perception sensor at each moment.
[0010] According to a method for diagnosing faults of perception sensors of unmanned vehicles provided by the present invention, the wide-body vehicle fault detection model is obtained by training in the following manner: a training data set is obtained based on parameter data of multiple on-board devices of historically faulty vehicles; the training data set includes parameter data as sample data and fault causes of the historically faulty vehicles as labels; and at least one initial graph attention network convolutional layer, an initial multi-layer perceptron, and an initial time-series-based neural network are trained using the training data set to obtain the wide-body vehicle fault detection model.
[0011] According to a method for diagnosing faults of perception sensors of unmanned vehicles provided by the present invention, a training data set is obtained based on parameter data of multiple on-board devices of historically faulty vehicles, including: in response to detecting that a fault has occurred in the historically faulty vehicle, collecting parameter data of multiple on-board devices of the historically faulty vehicle and using the data as sample data; determining the cause of the fault of the historically faulty vehicle and using the cause of the fault as a label for the corresponding sample data.
[0012] According to a method for diagnosing faults of perception sensors of unmanned vehicles provided by the present invention, the multiple on-board devices include: on-board perception sensors, domain control devices, gateway devices and combined navigation devices; the on-board device hypergraphs at each moment are constructed respectively according to the parameter data of the multiple on-board devices at each moment in the multiple moments to obtain multiple on-board device hypergraphs, including: according to the parameter data of the on-board perception sensors, the domain control devices, the gateway devices and the combined navigation devices at each moment, the correlation between the on-board perception sensors, the domain control devices, the gateway devices and the combined navigation devices is used as the edge value to construct the on-board device hypergraph at each moment to obtain multiple on-board device hypergraphs.
[0013] The present invention also provides a sensor fault diagnosis device for an unmanned vehicle, comprising: An acquisition module is used to acquire parameter data of multiple on-board devices of a faulty vehicle at multiple times; wherein the multiple on-board devices include: an on-board sensing sensor and on-board devices associated with the on-board sensing sensor; a construction module is used to construct a hypergraph of on-board devices at each time according to the parameter data of the multiple on-board devices at each time in the multiple times, so as to obtain multiple hypergraphs of on-board devices; wherein the hypergraph of on-board devices includes multiple supernodes, each supernode includes a subgraph, the subgraph corresponds to one of the multiple on-board devices, the subgraph includes at least one node, the value of the node is a parameter data of the on-board device corresponding to the subgraph; a prediction module is used to process the multiple on-board device hypergraphs using a wide-body vehicle fault detection model to obtain a fault prediction result of the faulty vehicle; wherein the wide-body vehicle fault detection model is obtained by training at least one graph attention network convolutional layer, a time-series-based neural network, and a multi-layer perceptron based on the parameter data of multiple on-board devices of historical faulty vehicles.
[0014] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of any of the above-described methods for diagnosing faults of perception sensors of unmanned vehicles are implemented.
[0015] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any of the above-described methods for diagnosing faults of perception sensors of unmanned vehicles.
[0016] The method and device for diagnosing faults of perception sensors of unmanned vehicles provided by the present invention respectively construct a hypergraph of vehicle-mounted devices at each moment according to the parameter data of multiple vehicle-mounted devices at each moment in multiple moments, thereby obtaining multiple hypergraphs of vehicle-mounted devices, so that the hypergraph of vehicle-mounted devices can be used to comprehensively and meticulously organize and represent the parameter data of multiple vehicle-mounted devices. The wide-body vehicle fault detection model is used to process multiple vehicle-mounted device hypergraphs to obtain the fault prediction result of the faulty vehicle. Since the wide-body vehicle fault detection model is based on the parameter data of multiple vehicle-mounted devices of historical faulty vehicles, and is obtained by training at least one graph attention network convolution layer, a time-series-based neural network, and a multi-layer perceptron, it has the powerful data processing and time-series feature extraction capabilities of the machine learning network, and can extract effective fault information from the parameter data of multiple vehicle-mounted devices at multiple moments, thereby accurately and efficiently realizing the diagnosis of vehicle-mounted perception sensor faults of unmanned vehicles. BRIEF DESCRIPTION OF THE DRAWINGS
[0017] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0018] Figure 1 This is one of the flow charts of the method for diagnosing a fault of a perception sensor of an unmanned vehicle provided by the present invention; Figure 2 It is a flow chart of a method for obtaining a fault prediction result of a faulty vehicle by using a wide-body vehicle fault detection model provided by the present invention; Figure 3 It is a flow chart of a training method for a wide-body vehicle fault detection model provided by the present invention; Figure 4 This is the second flow chart of the method for diagnosing faults of a perception sensor of an unmanned vehicle provided by the present invention; Figure 5 It is a structural schematic diagram of a wide-body vehicle fault detection model provided by the present invention; Figure 6 It is a schematic diagram of the data link structure of a wide-body vehicle; Figure 7 It is a schematic diagram of the process of the wide-body vehicle fault detection model provided by the present invention processing the vehicle-mounted equipment hypergraph; Figure 8 It is a structural schematic diagram of the vehicle-mounted equipment hypergraph provided by the present invention; Fig. 9 This is a schematic diagram of some of the reasons why LiDAR frames are lost and disconnected; Fig.10 This is a schematic diagram of some of the reasons why millimeter wave radar frames are lost and disconnected; Fig.11 This is a diagram of some of the reasons why the camera loses frames and goes offline; Fig.12 It is a structural schematic diagram of a perception sensor fault diagnosis device for an unmanned vehicle provided by the present invention; Fig.13 is a schematic diagram of the structure of an electronic device provided by the present invention; DETAILED DESCRIPTION
[0019] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0020] Combine the following Figures 1 to 11 The present invention describes a method for diagnosing faults of a perception sensor of an unmanned vehicle.
[0021] Figure 1 This is one of the flow charts of the method for diagnosing faults of a perception sensor of an unmanned vehicle provided by the present invention.
[0022] like Figure 1 As shown, the method includes the following steps.
[0023] Step 101: Obtain parameter data of multiple on-board devices of a faulty vehicle at multiple times.
[0024] The faulty vehicle is an unmanned vehicle whose fault cause needs to be detected, for example, a wide-body vehicle that has a fault.
[0025] The multiple vehicle-mounted devices include: a vehicle-mounted perception sensor and a vehicle-mounted device associated with the vehicle-mounted perception sensor.
[0026] Vehicle-mounted perception sensors include but are not limited to: laser radar, millimeter wave radar, camera, etc. For example, Figure 6 As shown, the vehicle-mounted perception sensors of the faulty vehicle include: front laser radar, rear laser radar, left laser radar, right laser radar, front millimeter wave radar, rear millimeter wave radar, left millimeter wave radar, right millimeter wave radar, front telephoto camera, front short-focus camera, etc. The vehicle-mounted devices associated with the vehicle-mounted perception sensors may include: domain control devices, gateway devices, and combined navigation devices, etc.
[0027] As an example only, the data communication link layout between multiple on-board devices of a wide-body vehicle is as follows: Figure 6 As shown. Specifically, the laser radar, millimeter-wave radar and camera realize data transmission with the domain controller through the vehicle Ethernet, CAN bus and GMSL (GigabitMultimedia Serial Link) respectively. Among them, the data transmission path of the front laser radar is also transferred through the intelligent gateway. The connection between the domain controller and the intelligent gateway relies on the vehicle Ethernet technology, while the integrated navigation system and the domain controller are connected through the CAN bus. Taking into account the large body size of the wide-body vehicle, the left and right rear cameras and four laser radars are designed with two-end wiring harness docking in the assembly process. The communication link between these on-board devices is quite complicated. From data collection at the sensor end to the final use in the application layer program, the data needs to go through multiple links of physical link transmission, domain control hardware driver processing and operating system kernel. Therefore, the challenge faced by the existing technology is that functional failures such as sensor data frame loss and disconnection are difficult to accurately attribute to a specific link in the data link.
[0028] In the specific implementation process, the collection interval of the vehicle-mounted equipment parameter data can be selected according to experience or experimental results, and multiple moments of the data can be determined.
[0029] Parameter data may include hardware status parameters of vehicle-mounted perception sensors, system environment parameters such as CPU frequency, CPU occupancy, memory occupancy, temperature, etc. of domain control and network management equipment, and IMU acceleration data RTK latitude and longitude elevation data in integrated navigation to associate the vehicle's operating environment conditions. As an example only, the parameter data of multiple vehicle-mounted devices of a faulty vehicle at multiple times are shown in Table 1 below.
[0030] Table 1
[0031] In the specific implementation process, the data collection function of the vehicle can be triggered when the on-board perception sensor of the unmanned vehicle has functional failures such as frame loss and disconnection, and the collected data can be uploaded to the cloud platform database for storage. The collected data can be exported and cleaned for use later.
[0032] In the specific implementation process, the reasons for frame loss and disconnection may be as follows: Fig. 9 , Fig.10 as well as Fig.11 shown.
[0033] For a diagnostic method for a fault type that can set diagnostic rules based on empirical thresholds, see Figure 4 The relevant content in will not be repeated here.
[0034] For fault types that cannot be diagnosed based on empirical thresholds, such as changes in current, voltage, and speed caused by loose connectors, the following steps can be used to diagnose the cause of the faulty vehicle.
[0035] Step 102: construct a hypergraph of the vehicle-mounted equipment at each moment according to the parameter data of the multiple vehicle-mounted equipment at each moment in the multiple moments, and obtain multiple hypergraphs of the vehicle-mounted equipment.
[0036] The vehicle-mounted equipment supergraph includes multiple supernodes, each supernode includes a subgraph, the subgraph corresponds to one of the multiple vehicle-mounted equipment, and the subgraph includes at least one node, and the value of the node is a parameter data of the vehicle-mounted equipment corresponding to the subgraph.
[0037] Just as an example, for unmanned wide-body vehicles, based on the parameter data of the on-board perception sensors, domain control devices, gateway devices and combined navigation devices at each moment, the correlation between the on-board perception sensors, domain control devices, gateway devices and combined navigation devices can be used as the edge value to construct a hypergraph of the on-board equipment at each moment, and obtain multiple hypergraphs of on-board equipment.
[0038] In the specific implementation process, the correlation between the vehicle-mounted perception sensors, domain control devices, gateway devices and integrated navigation devices can be determined in a variety of ways, which are not limited by the description of this specification. For example, the correlation between the vehicle-mounted devices can be determined based on the degree of data coupling between the vehicle-mounted devices.
[0039] As an example only, the vehicle-mounted device is constructed as shown in the figure Figure 8 As shown in the figure. The hypergraph contains multiple subgraphs, each of which represents a supernode corresponding to a device (such as Lidar, Camera, Radar, integrated navigation, gateway, domain control, etc.). Supernodes, nodes and supernodes, and nodes and nodes are connected by edges.
[0040] In some embodiments, the following operations may be performed for parameter data of multiple vehicle-mounted devices at each time to determine the initial value of each supernode of the vehicle-mounted device hypergraph at the time: According to the parameter data of each vehicle-mounted device, the node value of each vehicle-mounted device corresponding to the subgraph in the vehicle-mounted device hypergraph is obtained. In the specific implementation process, the node value can be obtained by assigning a value to the node corresponding to the subgraph in the vehicle-mounted device hypergraph according to the parameter data of each vehicle-mounted device.
[0041] Using a preset algorithm, the node values of each subgraph in the vehicle equipment hypergraph are processed to obtain the node feature vectors of each subgraph in the vehicle equipment hypergraph. For example, a fully connected layer or an embedding layer with an activation function can be used to map the node values of each subgraph in the vehicle equipment hypergraph to a feature vector space of the same dimension to obtain the node feature vectors of each subgraph.
[0042] The node feature vectors of each subgraph in the vehicle equipment hypergraph are integrated to obtain the initial value of each supernode in the vehicle equipment hypergraph. For example, the node feature vectors of each subgraph in the vehicle equipment hypergraph can be integrated by summing, concatenating, and sub-attention to obtain the initial value of each supernode in the vehicle equipment hypergraph.
[0043] Step 103: Using the wide-body vehicle fault detection model, multiple vehicle-mounted equipment hypergraphs are processed to obtain a fault prediction result of the faulty vehicle.
[0044] The wide-body vehicle fault detection model is obtained by training at least one graph attention network convolutional layer, a time-based neural network, and a multi-layer perceptron based on the parameter data of multiple on-board devices of historical faulty vehicles. For the training process of the wide-body vehicle fault detection model, see Figure 3 The relevant content in will not be repeated here.
[0045] The structure of the wide-body vehicle fault detection model is as follows: Figure 5 As shown, it includes at least one graph attention network convolution layer, a time-based neural network and a multi-layer perceptron.
[0046] During the specific implementation process, the number of graph attention network convolutional layers can be determined based on the model prediction effect. For example, it can include one graph attention network convolutional layer or multiple graph attention network convolutional layers, which is not limited to the description of this specification.
[0047] Time-series-based neural networks may include, but are not limited to, recurrent neural networks (RNN), long short-term memory networks (LSTM), gated recurrent units (GRU), etc.
[0048] Multilayer Perceptron (MLP) is a deep learning model based on a feedforward neural network. It consists of multiple neuron layers, each of which is fully connected to the previous layer. It can complete tasks such as classification, regression, and clustering. The basic structure of a multilayer perceptron includes an input layer, an output layer, and at least one or more hidden layers. Each layer consists of multiple neurons, and each neuron generates an output by performing a weighted summation of the input values and passing through an activation function.
[0049] Among them, the multi-layer perceptron includes a normalization layer, which can be implemented using the softmax function.
[0050] In the specific implementation process, multiple vehicle-mounted equipment hypergraphs can be used as input data of the wide-body vehicle fault detection model, and the wide-body vehicle fault detection model outputs multiple possible causes of the faulty vehicle and the probability corresponding to each cause.
[0051] For a detailed description of using the wide-body vehicle fault detection model to process multiple vehicle-mounted equipment hypergraphs and obtain fault prediction results for the faulty vehicle, see Figure 2 The relevant content in will not be repeated here.
[0052] During the specific implementation process, the vehicle fault diagnosis system can upload the fault prediction results to the cloud to guide the staff at the operation site to perform hardware repairs or perform necessary operations to quickly restore the normal operation of the vehicle.
[0053] Figure 2 It is a flow chart of a method for obtaining a fault prediction result of a faulty vehicle by using a wide-body vehicle fault detection model provided by the present invention.
[0054] like Figure 2 As shown, the method includes the following steps.
[0055] Step 201: Use at least one graph attention network convolution layer to iteratively process each vehicle-mounted equipment hypergraph to obtain the first fused feature vector of each supernode in each vehicle-mounted equipment hypergraph.
[0056] In the specific implementation process, the output of each graph attention network convolutional layer is used as the input of the next graph attention network convolutional layer, and the last graph attention network convolutional layer outputs the first fused feature vector of each super node in the vehicle-mounted equipment super graph.
[0057] In each round of iterative processing, the convolutional layer of the graph attention network calculates the attention weights between each supernode and other supernodes based on the edge values in the vehicle equipment hypergraph. For each supernode in the vehicle equipment hypergraph, the attention weights between it and other supernodes are used to perform weighted summation on the values of all supernodes in the vehicle equipment hypergraph to obtain the feature vector of the supernode.
[0058] After multiple rounds of iterative processing, each super node will obtain a feature vector that fully integrates the information of other nodes in the vehicle-mounted equipment hypergraph, that is, the first fused feature vector.
[0059] Step 202: Based on the first fused feature vector of the supernode corresponding to the vehicle-mounted perception sensor in each vehicle-mounted device hypergraph, obtain the state feature of the vehicle-mounted perception sensor at each moment.
[0060] In some embodiments, the first fused feature vector of the super node corresponding to the vehicle-mounted perception sensor in each vehicle-mounted device hypergraph can be used as the state feature of the vehicle-mounted perception sensor at each moment.
[0061] In some embodiments, Figure 7As shown, graph convolution and graph pooling operations can be performed on the super nodes corresponding to all vehicle-mounted perception sensors in each vehicle-mounted equipment hypergraph to obtain the second fused feature vector of each vehicle-mounted perception sensor corresponding to the super node in each vehicle-mounted equipment hypergraph; the second fused feature vector of each vehicle-mounted perception sensor corresponding to the super node in each vehicle-mounted equipment hypergraph is used as the state feature of the vehicle-mounted perception sensor at each moment.
[0062] Step 203: using a time-series-based neural network and based on the state characteristics of the vehicle-mounted perception sensor at multiple moments, obtain a time-series state feature vector of the vehicle-mounted perception sensor.
[0063] In the specific implementation process, the state features of the vehicle perception sensor at multiple times can be organized into a time series form, and each time step corresponds to the state features at one time. The time series is sequentially input into a time-series-based neural network (for example, GRU), and the feature vector of the last hidden layer of the time-series-based neural network is used as the time-series state feature vector of the vehicle perception sensor.
[0064] Step 204: using a multi-layer perceptron to obtain a fault prediction result of the faulty vehicle according to the time series state feature vector of the vehicle-mounted perception sensor.
[0065] In the specific implementation process, the time series state feature vector of the vehicle perception sensor can be used as the input of the multi-layer perceptron, and each neuron in the output layer of the multi-layer perceptron corresponds to a fault cause. Figure 7 As shown, in the output layer of the multi-layer perceptron (MLP), the softmax function is used to normalize the eigenvalue of each neuron in the output layer into a probability representation, and various fault causes of the faulty vehicle and the corresponding probabilities of various fault causes are obtained.
[0066] In the embodiment provided by the present invention, the vehicle-mounted equipment hypergraph is iteratively processed by at least one convolutional layer of the graph attention network, which can effectively fuse and extract the feature information of each supernode in the vehicle-mounted equipment hypergraph, not only considering the characteristics of the parameter data of the vehicle-mounted equipment corresponding to the supernode itself, but also fully considering the relationship and mutual influence between the parameter data of multiple vehicle-mounted equipment, thereby obtaining a more accurate and comprehensive first fusion feature vector. Based on the first fusion feature vector of the supernode corresponding to the vehicle-mounted perception sensor in the vehicle-mounted equipment hypergraph, the state characteristics of each vehicle-mounted perception sensor at each moment that can reflect its hardware operation state can be accurately obtained. Using a time-series-based neural network, according to the state characteristics of the vehicle-mounted perception sensor at multiple moments, the change information of the vehicle-mounted equipment fault information at different moments can be effectively extracted, thereby obtaining a time-series state feature vector that can accurately reflect the fault state information of the vehicle-mounted equipment. Finally, a multi-layer perceptron is used to accurately predict faults based on the time-series state feature vector of the vehicle-mounted perception sensor.
[0067] Figure 3 It is a flow chart of the training method of the wide-body vehicle fault detection model provided by the present invention.
[0068] like Figure 3 As shown, the method includes the following steps.
[0069] Step 301: obtain a training data set based on parameter data of multiple vehicle-mounted devices of historical faulty vehicles; the training data set includes parameter data as sample data and fault causes of historical faulty vehicles as labels.
[0070] When a fault is detected in a historical fault vehicle, parameter data of multiple on-board devices of the historical fault vehicle are collected and used as sample data; the fault cause of the historical fault vehicle is determined, and the fault cause is used as a label for the corresponding sample data.
[0071] In the specific implementation process, when a vehicle with a history of failure is detected to have a failure, such as frame loss or disconnection of the vehicle-mounted sensor, the data collection function of the vehicle is triggered. The collected data includes parameter data of various vehicle-mounted sensor, domain control, gateway, integrated navigation and other equipment, which fully reflects the status of the vehicle when it fails. The collected data is uploaded to the cloud database for storage for subsequent analysis and processing.
[0072] Through manual troubleshooting, the specific cause of the faulty vehicle is determined, which may include connector burnout, connector loosening, etc. The troubleshooting results are mapped with the parameter data of the vehicle stored in the cloud, and the cause of the fault is associated with the parameter data.
[0073] Based on the stored parameter data and its corresponding fault causes, a training data set is established, in which each data sample corresponds to a specific fault cause.
[0074] Step 302: Using the training data set, at least one initial graph attention network convolutional layer, an initial time-based neural network, and an initial multi-layer perceptron are trained to obtain a wide-body vehicle fault detection model.
[0075] In the specific implementation process, at least one initial graph attention network convolution layer, initial multi-layer perceptron, and initial time-series-based neural network can be trained in a variety of ways using a training data set to obtain a wide-body vehicle fault detection model, which is not limited by the description of this specification. For example, at least one initial graph attention network convolution layer, initial multi-layer perceptron, and initial time-series-based neural network can be trained end-to-end in a joint training manner to obtain a wide-body vehicle fault detection model.
[0076] During the training process, a loss function (e.g., a cross entropy loss function) can be constructed, and an optimization algorithm (e.g., a back-propagation algorithm) can be used to adjust the parameters of the initial graph attention network convolutional layer, the initial multi-layer perceptron, and the initial time-series-based neural network to reduce the value of the loss function until the model training end condition is reached (e.g., the loss function converges), thereby obtaining a wide-body vehicle fault detection model.
[0077] Figure 4 This is the second flow chart of the method for diagnosing faults of a perception sensor of an unmanned vehicle provided by the present invention.
[0078] Step 401: Obtain parameter data of multiple on-board devices of the faulty vehicle.
[0079] Collect parameter data of multiple onboard devices on the faulty vehicle, including but not limited to perception sensors such as cameras, millimeter-wave radars, and lidars, as well as real-time data of devices such as domain control, gateways, and integrated navigation (IMU+RTK). These data fully reflect the status of the vehicle when it fails, providing a basis for subsequent fault diagnosis.
[0080] Step 402: Diagnose the cause of the faulty vehicle based on the constructed diagnostic rule base.
[0081] For fault types with clear indicators or normal range empirical values, joint threshold rules can be designed based on expert knowledge and experience, and a diagnostic rule base can be built for vehicle-mounted perception sensors such as Lidar, Camera, and Radar. In the diagnostic rule base, in addition to the monitoring data based on the vehicle-mounted perception sensors themselves, comprehensive diagnosis can also be performed with the environmental data of domain control and gateways and the combined navigation IMU+RTK data.
[0082] During the specific implementation process, the diagnostic rule library can be used to monitor the collected parameter data in real time to determine the cause of the fault of the faulty vehicle.
[0083] As an example only, the following example illustrates the diagnosis process of a vehicle frame loss or disconnection failure.
[0084] Camera diagnostic rules: Monitor the domain controller decoder register (such as video pipe dec) to obtain the video quality and link status. However, monitoring this register alone cannot determine whether the video quality problem is caused by the video link or the camera module. Therefore, add a monitoring camera module register (such as pclk 0x55f). Combining the status of these two registers, when the video link is normal, it can be accurately determined whether the video quality problem is caused by the camera module or the link problem. At the same time, monitor the domain controller temperature, CPU occupancy, and memory usage to determine whether the domain controller is stuck when the frame is lost and the line is disconnected, and then determine whether the domain controller is stuck, causing the application layer driver abnormality and data acquisition problems. Combined with the acceleration and elevation information of the integrated navigation IMU, analyze whether the frame loss is caused by the loosening of the wiring harness connector caused by the acceleration vibration impact.
[0085] Millimeter wave radar diagnostic rules: By monitoring the CAN circuit load rate, voltage status, and temperature status, the cause of frame loss and disconnection faults can be determined. At the same time, combined with the domain controller temperature information, CPU load, and memory usage, it can be determined whether the frame loss and disconnection faults are caused by domain controller freezes or excessive process resource usage.
[0086] LiDAR diagnostic rules: Phy link down indicates that the Ethernet harness connection is abnormal. Check the radar harness. Ring buffer overflow indicates that the domain control buffer setting is unreasonable, causing frame loss. It is recommended to reconfigure. The packet loss rate of the gateway corresponding to the network port is large. It can be determined that the sensor frame loss and disconnection are caused by the gateway transfer link. It is recommended to check the gateway. The gateway voltage is abnormal and the CPU load memory usage is high, which may cause the gateway to be stuck and blocked, and then cause frame loss and disconnection. The domain control temperature is too high and the CPU load memory usage is high. It may be caused by the domain control program being stuck and blocked. It is recommended to check whether the air conditioning equipment is turned on to reduce the domain control temperature, and check the CPU load and memory usage. The radar speed is too high or too low, and the combined navigation acceleration and elevation change dramatically. The mechanical radar may be abnormal due to vibration and impact conditions. If the abnormality recovers quickly, no processing is required; if it persists, it is necessary to check whether the radar firmware version is suitable for the mining area, or consider returning the radar for repair.
[0087] Through the above steps, comprehensive and accurate fault diagnosis of faulty vehicles can be achieved, providing strong guarantee for the stable operation of sensors.
[0088] The perception sensor fault diagnosis device for an unmanned vehicle provided by the present invention is described below. The perception sensor fault diagnosis device for an unmanned vehicle described below and the perception sensor fault diagnosis method for an unmanned vehicle described above can be referenced to each other.
[0089] Fig.12Schematic diagram of the structure of the sensor fault diagnosis device for unmanned vehicles provided by the present invention. Fig.12 As shown, the device 1200 includes the following modules.
[0090] The acquisition module 1210 is used to acquire parameter data of multiple vehicle-mounted devices of the faulty vehicle at multiple times; wherein the multiple vehicle-mounted devices include: a vehicle-mounted sensing sensor and a vehicle-mounted device associated with the vehicle-mounted sensing sensor.
[0091] The construction module 1220 is used to construct a vehicle-mounted device hypergraph at each moment according to the parameter data of the multiple vehicle-mounted devices at each moment in the multiple moments, so as to obtain multiple vehicle-mounted device hypergraphs; wherein the vehicle-mounted device hypergraph includes multiple super nodes, each super node includes a subgraph, the subgraph corresponds to a vehicle-mounted device among the multiple vehicle-mounted devices, and the subgraph includes at least one node, and the value of the node is a parameter data of the vehicle-mounted device corresponding to the subgraph.
[0092] The prediction module 1230 is used to use the wide-body vehicle fault detection model to process the multiple vehicle-mounted equipment hypergraphs to obtain a fault prediction result of the faulty vehicle; wherein the wide-body vehicle fault detection model is based on parameter data of multiple vehicle-mounted equipment of historical faulty vehicles, and is obtained by training at least one graph attention network convolutional layer, a time-based neural network, and a multi-layer perceptron.
[0093] Fig.13 An example of a physical structure diagram of an electronic device is shown in FIG. Fig.13As shown, the electronic device may include: a processor (processor) 1310, a communication interface (Communications Interface) 1320, a memory (memory) 1330 and a communication bus 1340, wherein the processor 1310, the communication interface 1320, and the memory 1330 communicate with each other through the communication bus 1340. The processor 1310 can call the logic instructions in the memory 1330 to execute the perception sensor fault diagnosis method of the unmanned vehicle, the method comprising: obtaining parameter data of multiple on-board devices of the faulty vehicle at multiple times; wherein the multiple on-board devices include: an on-board perception sensor and an on-board device associated with the on-board perception sensor; according to the parameter data of the multiple on-board devices at each time in the multiple times, respectively constructing a hypergraph of the on-board devices at each time to obtain multiple hypergraphs of the on-board devices; wherein the on-board device hypergraph includes multiple super nodes, each super node includes a subgraph, the subgraph corresponds to a type of on-board device among the multiple on-board devices, the subgraph includes at least one node, and the value of the node is a type of parameter data of the on-board device corresponding to the subgraph; using a wide-body vehicle fault detection model, processing the multiple on-board device hypergraphs to obtain a fault prediction result of the faulty vehicle; wherein the wide-body vehicle fault detection model is obtained by training at least one graph attention network convolution layer, a time-series-based neural network, and a multi-layer perceptron based on the parameter data of multiple on-board devices of historical faulty vehicles.
[0094] In addition, the logic instructions in the above-mentioned memory 1330 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.
[0095] On the other hand, the present invention also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute the perception sensor fault diagnosis method for an unmanned vehicle provided by the above-mentioned methods, and the method includes: obtaining parameter data of multiple on-board devices of the faulty vehicle at multiple times; wherein the multiple on-board devices include: on-board perception sensors and on-board devices associated with the on-board perception sensors; and constructing the parameter data of each of the multiple on-board devices at each of the multiple times, respectively. The vehicle-mounted equipment hypergraph is engraved to obtain multiple vehicle-mounted equipment hypergraphs; wherein the vehicle-mounted equipment hypergraph includes multiple supernodes, each supernode includes a subgraph, the subgraph corresponds to one of the multiple vehicle-mounted equipment, and the subgraph includes at least one node, and the value of the node is a parameter data of the vehicle-mounted equipment corresponding to the subgraph; using the wide-body vehicle fault detection model, the multiple vehicle-mounted equipment hypergraphs are processed to obtain the fault prediction result of the faulty vehicle; wherein the wide-body vehicle fault detection model is obtained by training at least one graph attention network convolution layer, a time-based neural network and a multi-layer perceptron based on the parameter data of multiple vehicle-mounted equipment of historical faulty vehicles.
[0096] In another aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which is implemented when the computer program is executed by a processor to perform the above-mentioned unmanned vehicle perception sensor fault diagnosis method, the method comprising: obtaining parameter data of multiple vehicle-mounted devices of the faulty vehicle at multiple times; wherein the multiple vehicle-mounted devices include: a vehicle-mounted perception sensor and a vehicle-mounted device associated with the vehicle-mounted perception sensor; according to the parameter data of the multiple vehicle-mounted devices at each of the multiple times, respectively constructing a vehicle-mounted device hypergraph at each time, to obtain multiple vehicle-mounted device hypergraphs; Wherein, the vehicle-mounted equipment hypergraph includes multiple supernodes, each supernode includes a subgraph, the subgraph corresponds to one of the multiple vehicle-mounted equipment, and the subgraph includes at least one node, and the value of the node is a parameter data of the vehicle-mounted equipment corresponding to the subgraph; using the wide-body vehicle fault detection model, the multiple vehicle-mounted equipment hypergraphs are processed to obtain the fault prediction result of the faulty vehicle; wherein, the wide-body vehicle fault detection model is obtained by training at least one graph attention network convolution layer, a time-based neural network and a multi-layer perceptron based on the parameter data of multiple vehicle-mounted equipment of historical faulty vehicles.
[0097] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.
[0098] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.
[0099] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A method for diagnosing faults of a perception sensor of an unmanned vehicle, characterized in that: include: Acquire parameter data of multiple on-board devices of the faulty vehicle at multiple times; wherein the multiple on-board devices include: an on-board sensing sensor and an on-board device associated with the on-board sensing sensor; According to the parameter data of the multiple vehicle-mounted devices at each of the multiple moments, a vehicle-mounted device hypergraph at each moment is constructed respectively to obtain multiple vehicle-mounted device hypergraphs; wherein the vehicle-mounted device hypergraph includes multiple super nodes, each super node includes a subgraph, the subgraph corresponds to a vehicle-mounted device among the multiple vehicle-mounted devices, and the subgraph includes at least one node, and the value of the node is a parameter data of the vehicle-mounted device corresponding to the subgraph; The wide-body vehicle fault detection model is used to process the multiple vehicle-mounted equipment hypergraphs to obtain a fault prediction result of the faulty vehicle; wherein the wide-body vehicle fault detection model is obtained by training at least one graph attention network convolutional layer, a time-based neural network, and a multi-layer perceptron based on parameter data of multiple vehicle-mounted equipment of historical faulty vehicles.
2. The method for diagnosing faults of a perception sensor of an unmanned vehicle according to claim 1, characterized in that: The initial value of each supernode in the vehicle-mounted equipment hypergraph is obtained by the following method: For the parameter data of the plurality of vehicle-mounted devices at each time point, the following operations are performed to determine the initial value of each supernode in the vehicle-mounted device hypergraph at the time point: According to the parameter data of each of the vehicle-mounted devices, a node value of a subgraph corresponding to each of the vehicle-mounted devices in the vehicle-mounted device hypergraph is obtained; Using a preset algorithm, processing the node values of each subgraph in the vehicle-mounted equipment hypergraph to obtain the node feature vectors of each subgraph in the vehicle-mounted equipment hypergraph; The node feature vectors of each subgraph in the vehicle-mounted equipment hypergraph are integrated to obtain the initial value of each supernode in the vehicle-mounted equipment hypergraph.
3. The method for diagnosing faults of a perception sensor of an unmanned vehicle according to claim 2, characterized in that: The method of using the wide-body vehicle fault detection model to process the plurality of vehicle-mounted equipment hypergraphs to obtain a fault prediction result of the faulty vehicle includes: Using the at least one graph attention network convolutional layer, iteratively process each of the vehicle-mounted equipment hypergraphs to obtain a first fused feature vector of each supernode in each of the vehicle-mounted equipment hypergraphs; Based on the first fused feature vector of the super node corresponding to the vehicle-mounted perception sensor in each of the vehicle-mounted equipment hypergraphs, the state feature of the vehicle-mounted perception sensor at each moment is obtained; Using the time-series-based neural network, based on the state characteristics of the vehicle-mounted perception sensor at the multiple time instants, a time-series state feature vector of the vehicle-mounted perception sensor is obtained; The multi-layer perceptron is used to obtain a fault prediction result of the faulty vehicle according to the time series state feature vector of the vehicle-mounted perception sensor.
4. The method for diagnosing faults of a perception sensor of an unmanned vehicle according to claim 3, characterized in that: The obtaining of the state characteristics of the vehicle-mounted perception sensor at each moment based on the first fused feature vector of the super node corresponding to the vehicle-mounted perception sensor in each vehicle-mounted device hypergraph includes: For each of the super nodes corresponding to all the vehicle-mounted perception sensors in the vehicle-mounted device hypergraph, a graph convolution operation and a graph pooling operation are performed to obtain a second fused feature vector of a super node corresponding to each of the vehicle-mounted perception sensors in each of the vehicle-mounted device hypergraphs; The second fused feature vector of the super node corresponding to each of the vehicle-mounted perception sensors in each of the vehicle-mounted equipment hypergraphs is used as the state feature of the vehicle-mounted perception sensor at each moment.
5. The method for diagnosing faults of a perception sensor of an unmanned vehicle according to any one of claims 1 to 4, characterized in that: The wide-body vehicle fault detection model is trained by the following method: A training data set is obtained based on parameter data of multiple on-board devices of historical faulty vehicles; the training data set includes parameter data as sample data and fault causes of the historical faulty vehicles as labels; Using the training data set, at least one initial graph attention network convolutional layer, an initial multi-layer perceptron, and an initial time-series-based neural network are trained to obtain the wide-body vehicle fault detection model.
6. The method for diagnosing faults of a perception sensor of an unmanned vehicle according to claim 5, characterized in that: The training data set is obtained based on the parameter data of multiple on-board devices of historically faulty vehicles, including: In response to detecting that the historical fault vehicle has a fault, collecting parameter data of a plurality of on-board devices of the historical fault vehicle and using the parameter data as sample data; The fault cause of the historical fault wide body is determined, and the fault cause is used as a label of the corresponding sample data.
7. The method for diagnosing faults of a perception sensor of an unmanned vehicle according to claim 1, characterized in that: The multiple vehicle-mounted devices include: vehicle-mounted perception sensors, domain control devices, gateway devices, and integrated navigation devices; The step of constructing a hypergraph of the vehicle equipment at each moment according to the parameter data of the multiple vehicle equipment at each moment in the multiple moments, to obtain multiple hypergraphs of the vehicle equipment, comprises: According to the parameter data of the vehicle-mounted perception sensor, the domain control device, the gateway device and the combined navigation device at each moment, the correlation between the vehicle-mounted perception sensor, the domain control device, the gateway device and the combined navigation device is used as the edge value, and a vehicle-mounted device hypergraph at each moment is constructed to obtain multiple vehicle-mounted device hypergraphs.
8. A sensor fault diagnosis device for an unmanned vehicle, characterized in that: include: An acquisition module, used to acquire parameter data of multiple on-board devices of the faulty vehicle at multiple times; wherein the multiple on-board devices include: an on-board perception sensor and an on-board device associated with the on-board perception sensor; A construction module, used to construct a vehicle-mounted device hypergraph at each moment according to the parameter data of the multiple vehicle-mounted devices at each moment in the multiple moments, so as to obtain multiple vehicle-mounted device hypergraphs; wherein the vehicle-mounted device hypergraph includes multiple super nodes, each super node includes a subgraph, the subgraph corresponds to a vehicle-mounted device among the multiple vehicle-mounted devices, and the subgraph includes at least one node, the value of the node is a parameter data of the vehicle-mounted device corresponding to the subgraph; A prediction module is used to use a wide-body vehicle fault detection model to process the multiple vehicle-mounted equipment hypergraphs to obtain a fault prediction result of the faulty vehicle; wherein the wide-body vehicle fault detection model is based on parameter data of multiple vehicle-mounted equipment of historical faulty vehicles, and is obtained by training at least one graph attention network convolutional layer, a time-based neural network, and a multi-layer perceptron.
9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the steps of the method for diagnosing faults of a perception sensor of an unmanned vehicle as described in any one of claims 1 to 7 are implemented.
10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method for diagnosing faults of a perception sensor of an unmanned vehicle as claimed in any one of claims 1 to 7 are implemented.
