Unmanned vehicle fault management method and device and electronic equipment

Abstract

The application relates to an unmanned vehicle fault management method, device and electronic equipment, which comprises the following steps: acquiring abnormal state data of a target unmanned vehicle; processing the abnormal state data through a feature extraction network to extract key fault feature data from the abnormal state data; analyzing the key fault feature data through a fault analysis network to determine the fault cause of the target unmanned vehicle; determining fault guidance information matched with the fault cause in a preset atlas and sending the fault guidance information to a fault management terminal; receiving fault feedback data sent by the fault management terminal; the fault feedback data comprises one or more of an actual fault cause, component update information, processing time consumption and a vehicle running state after processing; and the feature extraction network, the fault analysis network and one or more of the preset atlas are updated and optimized according to the fault feedback data.

Description

Technical Field

[0001] This invention relates to the field of unmanned vehicle management technology, and more specifically, to an unmanned vehicle fault management method, device, and electronic device. Background Technology

[0002] With the widespread application of autonomous driving technology in mining areas, ports, and other scenarios, the monitoring of the operational status and fault management of unmanned vehicles are becoming increasingly important. Currently, fault management for unmanned vehicles mainly relies on fault code reporting or regular manual inspections. However, these methods suffer from problems such as insufficient data utilization, reliance on human experience for fault diagnosis, and a lack of intelligent data analysis and processing tools.

[0003] Therefore, there is an urgent need for a fault management method for unmanned vehicles that can automatically acquire abnormal data, intelligently analyze the causes of faults, and effectively manage them. Summary of the Invention

[0004] The purpose of this invention is to provide a method, device, and electronic device for managing the faults of unmanned vehicles, in order to solve the problems existing in the prior art.

[0005] Firstly, this disclosure provides a method for managing faults in unmanned vehicles, including: Acquire abnormal state data of the target unmanned vehicle; The abnormal state data is processed by a feature extraction network to extract key fault feature data from the abnormal state data. The key fault characteristic data are analyzed by a fault analysis network to determine the cause of the target unmanned vehicle's failure. Determine fault guidance information that matches the cause of the fault in the preset map, and send the fault guidance information to the fault management terminal; Receive fault feedback data sent by the fault management terminal; the fault feedback data includes one or more of the following: actual fault cause, component update information, processing time, and vehicle operating status after processing. Based on the fault feedback data, one or more of the feature extraction network, fault analysis network, and preset map are optimized and updated.

[0006] Optionally, acquiring the abnormal state data of the target unmanned vehicle includes: Obtain the original state data of the target unmanned vehicle; If the original state data is determined to meet the preset abnormal conditions, the original state data is standardized and cleaned to obtain the abnormal state data of the target unmanned vehicle.

[0007] Optionally, the data standardization and cleaning of the original state data includes one or more of the following: Instantaneous outliers are removed from the original state data based on a preset numerical range; The invalid scores of the original state data are calculated based on the isolated forest algorithm, and the original state data with invalid scores exceeding a preset score threshold are removed. The missing data segments in the original state data are supplemented by interpolation; The authenticity of the original state data is verified, and false state data is removed.

[0008] Optionally, the key fault characteristic data includes at least one of the following: Valid fault characteristic data; the valid fault characteristic data refers to fault characteristic data that exceeds the preset fault level; Key indicator feature data; the key indicator feature data refers to the preset indicator data collected when the target unmanned vehicle malfunctions; Model update feature data; the model update feature data refers to the feature data used to update the fault analysis network or the preset map.

[0009] Optionally, the key fault feature data includes time-series feature data, fault code feature data, and image feature data; the fault analysis network includes a feature fusion subnetwork, a graph neural subnetwork, a time-series causal inference subnetwork, and a spatiotemporal alignment subnetwork. The step of analyzing the key fault characteristic data through a fault analysis network to determine the cause of the target unmanned vehicle's failure includes: The time-series feature data, fault code feature data, and image feature data are fused through the feature fusion subnetwork to obtain multimodal fused feature data; The graph neural subnetwork is used to perform fault correlation mining on the multimodal fusion feature data to identify the transmission and coupling relationships between faults. The temporal causal inference subnetwork performs temporal causal inference on the multimodal fused feature data to determine the fault source object and the fault-related object; wherein, the temporal causal inference subnetwork includes a temporal convolutional layer and a causal attention layer; The spatiotemporal alignment sub-network is used to spatiotemporally align the multimodal fusion feature data with environmental monitoring data to determine the correlation between faults and the environment. Based on the fault propagation and coupling relationships, fault source objects and fault-related objects, as well as the relationship between faults and the environment, the fault causes of the target unmanned vehicle are generated.

[0010] Optionally, the fault propagation and coupling relationship includes fault propagation paths and multi-fault point coupling relationships; The step of performing fault correlation mining on the multimodal fused feature data through the graph neural subnetwork to identify the transmission and coupling relationships between faults includes: Construct a heterogeneous information network for the target unmanned vehicle; the nodes of the heterogeneous information network include mechanical component nodes, electrical unit nodes, communication link nodes, and software module nodes, and the edges between each node represent physical connection relationships or data interaction relationships; The graph neural subnetwork is used to infer the inter-node relationships in the heterogeneous information network and determine the degree of fault correlation of each node. Based on the degree of fault correlation and the edges between nodes, the fault propagation path and the coupling relationship of multiple fault points are determined.

[0011] Optionally, the step of performing time-series causal inference on the multimodal fused feature data through the time-series causal inference sub-network to determine the fault source object and fault-related objects includes: Temporal feature analysis is performed on the multimodal fusion feature data through the temporal convolutional layer and causal attention layer in the temporal causal inference subnetwork to determine the causal inference relationship chain; Based on the causal inference chain, the fault source object and the fault-related object are determined.

[0012] Optionally, determining the fault guidance information matching the fault cause in the preset map includes: Using the cause of the fault as a query condition, semantic retrieval is performed in the preset graph to determine multiple historical fault events associated with the cause of the fault and their corresponding historical processing information. Based on the cause of the fault, multiple historical fault events, and the historical processing information corresponding to each historical fault event, fault guidance information matching the cause of the fault is generated.

[0013] Secondly, an unmanned vehicle fault management device is provided, comprising: The status data acquisition module is used to acquire abnormal status data of the target unmanned vehicle; The feature data extraction module is used to process the abnormal state data through a feature extraction network to extract key fault feature data from the abnormal state data. The fault cause analysis module is used to analyze the key fault characteristic data through a fault analysis network to determine the fault cause of the target unmanned vehicle. The guidance information determination module is used to determine the fault guidance information that matches the cause of the fault in the preset map, and send the fault guidance information to the fault management terminal; The feedback receiving module is used to receive fault feedback data sent by the fault management terminal; the fault feedback data includes one or more of the following: actual fault cause, component update information, processing time, and vehicle operating status after processing. The model update module is used to optimize and update one or more of the feature extraction network, fault analysis network, and preset map based on the fault feedback data.

[0014] Thirdly, an electronic device is provided, comprising: A memory on which computer programs are stored; A processor for executing the computer program in the memory to implement the steps of the method described in any of the above embodiments.

[0015] The technical solutions provided by the embodiments of this disclosure may include the following beneficial effects: The system acquires abnormal state data of the target unmanned vehicle; processes this data using a feature extraction network to filter and refine key fault features; performs multi-dimensional analysis of these key fault features using a fault analysis network to accurately determine the cause of the target unmanned vehicle's fault; identifies fault guidance information matching the fault cause in a pre-defined fault map and sends this information to the fault management terminal; simultaneously, it receives fault feedback data uploaded by the fault management terminal and iteratively optimizes the network and fault map based on the feedback data. On one hand, by filtering redundant and interfering information and focusing on core fault features through the feature extraction network, and then collaborating with the fault analysis network to analyze the data, the system fully leverages the value of vehicle operation data, eliminating the reliance on human experience in traditional fault diagnosis. This allows for accurate extraction of key fault features and efficient tracing of fault causes, significantly improving the accuracy and efficiency of unmanned vehicle fault diagnosis. On the other hand, by generating targeted fault guidance information through pre-defined fault map matching and pushing it to the fault management terminal, the system replaces manual inspection and troubleshooting, providing intelligent decision support for unmanned vehicle fault maintenance and improving the targeting of fault handling and the efficiency of maintenance response. In addition, by continuously updating the network and fault map with maintenance feedback, the fault diagnosis system can be self-iteratively optimized, and the fault management capability in complex scenarios can be continuously improved in the long term.

[0016] Other features and advantages of this disclosure will be described in detail in the following detailed description section. Attached Figure Description

[0017] The accompanying drawings are provided to further illustrate the present disclosure and form part of the specification. They are used together with the following detailed description to explain the present disclosure, but do not constitute a limitation thereof. In the drawings: Figure 1 This is a flowchart illustrating an unmanned vehicle fault management method according to an exemplary embodiment of the present disclosure.

[0018] Figure 2 This is a flowchart illustrating an unmanned vehicle fault management method according to another exemplary embodiment of the present disclosure.

[0019] Figure 3 This is a structural block diagram of an unmanned vehicle fault management device according to an exemplary embodiment of the present disclosure.

[0020] Figure 4 This is a block diagram illustrating an electronic device according to an exemplary embodiment of the present disclosure. Detailed Implementation

[0021] The specific embodiments of this disclosure will be described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are for illustration and explanation only and are not intended to limit this disclosure.

[0022] In this disclosure, unless otherwise stated, directional terms such as "upper," "lower," "left," and "right" are used for ease of description based on the drawing orientations of the corresponding figures, while "inner" and "outer" are defined based on the contours of the corresponding components themselves. Terms such as "first" and "second" used in this disclosure are used to distinguish one element from another and do not have sequential or importance implications. Furthermore, when the following description refers to the figures, unless otherwise indicated, the same numbers in different figures represent the same or similar elements.

[0023] An exemplary embodiment of this disclosure first provides an unmanned vehicle fault management method. This method can be applied to an unmanned vehicle fault management system, which includes at least unmanned vehicles and a cloud platform. The unmanned vehicles can be unmanned mining trucks in open-pit mines, unmanned container trucks in ports, unmanned transport vehicles in logistics parks, unmanned buses, and other unmanned vehicles of different scenarios or types. There can be one or more unmanned vehicles, and any one of them can be the target unmanned vehicle. The cloud platform can be a computing platform that provides services such as data storage, model inference, and knowledge retrieval, such as a server consisting of one or more servers or a cloud server cluster. For example, the unmanned vehicle can collect abnormal state data, extract key fault feature data, and send the key fault feature data to the cloud platform for processing. The cloud platform then determines the cause of the target unmanned vehicle's fault and finally determines fault guidance information in a preset map based on the fault cause. The unmanned vehicle fault management method can also be applied to the cloud platform. For example, the unmanned vehicle can collect abnormal state data and send it to the cloud platform, where key fault feature data is extracted, and the cause of the target unmanned vehicle's fault is determined based on the key fault feature data. Finally, fault guidance information is determined in a preset map based on the fault cause.

[0024] Please see Figure 1Firstly, this disclosure provides a method for managing the faults of unmanned vehicles, including: Step S110: Obtain abnormal state data of the target unmanned vehicle.

[0025] The target unmanned vehicle refers to an unmanned vehicle that requires fault management, such as unmanned mining trucks in open-pit mines or unmanned container trucks in automated port terminals. Status data refers to data reflecting the operating status, working parameters, and environmental interaction information of various components of the unmanned vehicle. This data can be multi-source data of various types, such as engine coolant temperature, engine speed, oil pressure, fuel consumption rate, transmission oil temperature, hydraulic system pressure, brake pressure, suspension displacement, tire pressure, battery voltage, motor current, controller area network bus load rate, communication link latency, packet loss rate, GPS signal strength, inertial measurement unit data, LiDAR point cloud density, and camera image frame rate. Abnormal status data refers to data deviating from the normal operating range of the vehicle, as determined from the above status data. Examples include engine temperature exceeding the normal threshold, drastic fluctuations detected by vibration sensors, a sudden increase in communication link packet loss rate, or loss of positioning signals.

[0026] Obtaining abnormal state data of the target autonomous vehicle can be achieved in various ways. It can be done in real-time by data acquisition modules deployed on the vehicle, such as sensor data collected by lidar sensors, millimeter-wave radar sensors, and camera modules; operational data of vehicle control systems such as the engine, transmission, hydraulic system, and drive-by-wire chassis collected via the controller area network bus interface; link status data between the vehicle and the dispatch system, such as signal strength, packet loss rate, and communication latency, collected via the communication interface; historical abnormal state data read from a cloud database; or manual input or upload by management personnel via external devices. This exemplary embodiment can continuously collect the target autonomous vehicle's state data and preprocess and filter it to identify abnormal state data.

[0027] Step S120: The abnormal state data is processed by a feature extraction network to extract key fault feature data from the abnormal state data.

[0028] Feature extraction networks are neural network models used to extract representative feature data from raw data. They can be convolutional neural networks, recurrent neural networks, long short-term memory networks, or lightweight networks compressed using knowledge distillation techniques. Key fault feature data refers to low-dimensional data extracted from abnormal state data that characterizes the essential features of the fault, such as a 128-dimensional feature vector or a set of encoded fault feature parameters. After identifying abnormal state data, the feature extraction network can perform forward computation to map the high-dimensional abnormal state data to low-dimensional key fault feature data. Feature extraction networks can be configured within the target autonomous vehicle. For example, after the target autonomous vehicle collects abnormal state data, it processes the data using its own configured feature extraction network to extract key fault feature data, which can then be sent to a cloud platform. Alternatively, feature extraction networks can be configured on a cloud platform. For example, after the cloud platform acquires abnormal state data, it processes the data using its own configured feature extraction network to extract key fault feature data and continues with subsequent processes.

[0029] It should be noted that abnormal state data is not necessarily fault data. It may include non-fault anomalies caused by factors such as environmental interference, sensor noise, or communication abnormalities. However, the key fault feature data obtained after processing by the feature extraction network can effectively filter out non-fault components.

[0030] Step S130: Analyze the key fault feature data through a fault analysis network to determine the cause of the target unmanned vehicle's fault.

[0031] The fault analysis network refers to a deep learning model used to analyze feature data to determine the cause of a fault. It can be a multimodal model based on the Transformer architecture, or a large model combining graph neural networks, temporal convolutional networks, and other models. The cause of the fault refers to the reason why the vehicle malfunctions. This can include the faulty object, such as the engine injector, lidar, or communication module; the fault type, such as blockage, signal attenuation, or communication timeout; possible triggers for the fault, such as component aging, dust obstruction, or electromagnetic interference; and one or more other pieces of information, such as the confidence level of the cause. This exemplary embodiment can input key fault feature data into the fault analysis network, which processes the key fault feature data and outputs the cause of the target unmanned vehicle's malfunction. Specifically, this can be based on the fault analysis network performing multimodal fusion, correlation mining, and causal inference on the key fault feature data to obtain the cause of the fault.

[0032] Step S140: Determine the fault guidance information that matches the cause of the fault in the preset map, and send the fault guidance information to the fault management terminal.

[0033] The pre-built graph refers to a pre-constructed data structure that stores knowledge in graph form, such as an unmanned vehicle maintenance knowledge graph. Nodes in the pre-built graph can represent entities such as fault phenomena, fault causes, and maintenance operations, while edges can represent relationships between entities, such as causal relationships and sequential relationships. Fault guidance information refers to information used to guide managers in troubleshooting and maintenance, such as inspection items, troubleshooting sequence, maintenance steps, replacement parts lists, or safety precautions, or one or more of these. The fault management terminal refers to a terminal device used to receive fault guidance information and display it to managers, such as a centralized management terminal in a mining area dispatch center, or a mobile terminal such as a tablet or smartphone used by managers. This exemplary embodiment can use the fault cause as a query condition to search the pre-built graph, obtain matching fault guidance information, and then send the fault guidance information to the fault management terminal so that managers can manage the target unmanned vehicle by performing maintenance, inspection, and other tasks according to the fault guidance information.

[0034] Based on the above description, in this exemplary embodiment, abnormal state data of the target unmanned vehicle is acquired; the abnormal state data is processed by a feature extraction network to extract key fault feature data; the key fault feature data is analyzed by a fault analysis network to determine the cause of the target unmanned vehicle's fault; fault guidance information matching the cause of the fault is determined in a preset graph, and the fault guidance information is sent to the fault management terminal. On the one hand, by coordinating the processing of abnormal state data of the unmanned vehicle through the feature extraction network and the fault analysis network, key fault features can be accurately extracted and fault causes can be efficiently determined, significantly improving the accuracy and efficiency of unmanned vehicle fault diagnosis; on the other hand, by generating and pushing targeted fault guidance information to the fault management terminal through preset graph matching, precise and intelligent decision support is provided for unmanned vehicle fault operation and maintenance, greatly improving the targeting of fault handling and the efficiency of operation and maintenance response.

[0035] Step S150: Receive fault feedback data sent by the fault management terminal; the fault feedback data includes one or more of the following: actual fault cause, component update information, processing time, and vehicle operating status after processing. Step S160: Based on the fault feedback data, optimize and update one or more of the feature extraction network, fault analysis network, and preset map.

[0036] It is understood that, based on the above description, in this exemplary embodiment, abnormal state data of the target unmanned vehicle is acquired; the abnormal state data is processed by a feature extraction network to extract key fault feature data from the abnormal state data; the key fault feature data is analyzed by a fault analysis network to determine the cause of the target unmanned vehicle's fault; fault guidance information matching the fault cause is determined in a preset graph, and the fault guidance information is sent to the fault management terminal. On the one hand, by coordinating the abnormal state data of the unmanned vehicle through the feature extraction network and the fault analysis network, key fault features can be accurately extracted and fault causes can be efficiently determined, significantly improving the accuracy and efficiency of unmanned vehicle fault diagnosis; on the other hand, by generating and pushing targeted fault guidance information to the fault management terminal through preset graph matching, precise and intelligent decision support is provided for unmanned vehicle fault operation and maintenance, greatly improving the targeting of fault handling and the efficiency of operation and maintenance response.

[0037] In one embodiment, acquiring the abnormal state data of the target unmanned vehicle may include: Obtain the original state data of the target unmanned vehicle; If the original state data is determined to meet the preset abnormal conditions, the original state data is standardized and cleaned to obtain the abnormal state data of the target unmanned vehicle.

[0038] In one embodiment, the data standardization and cleaning of the original state data includes one or more of the following: Instantaneous outliers are removed from the original state data based on a preset numerical range; The invalid scores of the original state data are calculated based on the isolated forest algorithm, and the original state data with invalid scores exceeding a preset score threshold are removed. The missing data segments in the original state data are supplemented by interpolation; The authenticity of the original state data is verified, and false state data is removed.

[0039] Raw state data refers to vehicle state data acquired directly without processing, such as raw point cloud data from lidar, target echo data from millimeter-wave radar, raw image data from cameras, engine speed and temperature on the controller area network bus, hydraulic system pressure values, brake air pressure values ​​from the drive-by-wire chassis, or communication link state data between the vehicle and the dispatch system. Preset anomaly conditions refer to pre-defined conditions or rules used to determine whether data is abnormal. Different state data or indicators can correspond to different preset anomaly conditions. For example, for engine temperature, "temperature > 105℃" can be set as an anomaly condition; for vibration sensors, "vibration amplitude exceeding three times the normal value standard deviation" can be set as an anomaly condition; and for communication links, "packet loss rate > 10%" can be set as an anomaly condition. Data standardization and cleaning refers to the process of standardizing raw state data to remove outliers, supplement missing values, and eliminate noise. When it is determined that the raw state data meets the preset anomaly conditions, this exemplary embodiment can perform data standardization and cleaning on the raw state data to obtain abnormal state data.

[0040] Specifically, data standardization and cleaning of raw state data can include one or more of the following methods.

[0041] Method 1 involves removing transient outliers from the raw state data based on a preset numerical range. The preset numerical range refers to the normal fluctuation range of the target unmanned vehicle's state data, which can be determined based on the historical state data distribution of the target unmanned vehicle. A transient outlier refers to an abnormal data point in the time series data where the amplitude significantly deviates from the normal range at a single or a few consecutive time points, such as abnormal temperature readings caused by electromagnetic interference. When raw state data outside the preset numerical range is identified, it can be considered a transient outlier and removed. For example, for the engine speed data of an unmanned mining truck, if a reading of 1800 rpm is collected, this reading exceeds the preset numerical range corresponding to the engine speed, and can therefore be identified as a transient outlier and removed.

[0042] Method Two: In practical applications, some distorted or unusable invalid data may be collected. Therefore, the Isolation Forest algorithm can be used to calculate the invalidity score of the original state data, and the original state data with invalidity scores exceeding a preset score threshold can be removed. The Isolation Forest algorithm is an unsupervised anomaly detection algorithm based on a binary tree structure. It recursively divides the data space by randomly selecting features and split values. Invalid data is more easily isolated and has a shorter path length. The invalidity score is a numerical value used to measure the invalidity of data points, ranging from 0 to 1. The closer the score is to 1, the more likely the original state data is to be invalid. For example, for a set of vibration data containing 100 sampling points, 50 isolation trees are constructed, the invalidity score for each sampling point is calculated, and the invalidity score threshold is set to 0.7. Sampling points with an invalidity score of 0.92 are judged as invalid data and removed.

[0043] Method three involves interpolating to fill in missing data segments in the original state data. Interpolation refers to methods that estimate the value of missing data segments using known data points, such as linear interpolation, spline interpolation, or interpolation based on generative models. For example, when communication link instability causes the complete loss of engine oil pressure data within 500 milliseconds, interpolation methods can be used to fill in the missing segments.

[0044] Method four involves verifying the authenticity of the original state data and eliminating false state data. Authenticity verification refers to the process of checking the physical rationality and consistency of the original state data, such as checking whether the original state data remains constant over a long period and whether the outputs of multiple sensors of the same type are consistent. False state data refers to data that fails to reflect the true physical state of the vehicle due to sensor malfunctions, communication errors, or other reasons. Examples include constant output values ​​caused by sensor jamming, abnormal code values ​​caused by corrupted data packets, or externally injected forged signals. For instance, if the reading of a certain sensor remains constant for a period of time, such as engine oil pressure remaining unchanged, while actual operating conditions should have normal fluctuations, the sensor output is determined to be false state data and eliminated.

[0045] This exemplary embodiment can perform data standardization cleaning using one or more of the methods described above. For example, it can use only method one or method two, or perform data standardization cleaning sequentially in the order of method one, method two, method three, and method four. Considering that raw state data in the harsh environment of open-pit mines often contains outliers, noise, missing data, and spurious data, this exemplary embodiment can use a combination of multiple methods to perform data standardization cleaning on the raw state data. This can effectively improve data quality, provide reliable input for subsequent feature extraction and fault analysis, reduce false alarm rates, and improve the accuracy of fault diagnosis.

[0046] In an exemplary embodiment, the above-described processing of abnormal state data using a feature extraction network to extract key fault feature data from the abnormal state data may include: Abnormal state data is input into a pre-configured feature extraction network for forward computation, outputting a low-dimensional feature vector, which is then used as key fault feature data.

[0047] The feature extraction network can be a neural network model with fewer parameters and lower computational complexity compared to large deep learning models, such as lightweight convolutional neural networks like MobileNet and ShuffleNet, or a student network compressed from a large teacher network using knowledge distillation techniques, or a network processed by model compression techniques such as pruning and quantization. Forward computation refers to the process by which a neural network calculates its output layer by layer according to its network structure, given input data. This includes the input layer receiving data, the hidden layers performing weighted summation and activation function transformations, and the output layer generating the result. A low-dimensional feature vector is a set of low-dimensional feature representations extracted from abnormal state data. It can include one or more of time-series feature vectors, fault code feature vectors, and image feature vectors. For example, the original abnormal state data contains 1024-dimensional time-series data, 32-dimensional fault code data, and 256-dimensional image data. After processing by the feature extraction network, it can output a 128-dimensional time-series feature vector, a 16-dimensional fault code feature vector, and a 64-dimensional image feature vector.

[0048] Abnormal state data is input into a pre-configured feature extraction network for forward computation, outputting a low-dimensional feature vector. Specifically, this can be implemented, for example, by deploying a feature extraction network trained using knowledge distillation technology on the edge computing device of an unmanned mining truck. This network's input layer can receive 1024-dimensional abnormal state data, and the network contains three fully connected hidden layers (256 neurons per layer), with a 128-dimensional output layer. After forward computation, a 128-dimensional low-dimensional feature vector is obtained. Once the low-dimensional feature vector is determined, it can be directly used as key fault feature data. Considering the limited communication bandwidth in mining areas and the scarcity of computing resources on the vehicle, this exemplary embodiment compresses high-dimensional abnormal state data into low-dimensional key fault feature data through the feature extraction network. This reduces data transmission bandwidth consumption, lowers the computing load on the vehicle, and retains the critical information required for fault diagnosis.

[0049] In one embodiment, the key fault characteristic data includes at least one of the following: Valid fault characteristic data; the valid fault characteristic data refers to fault characteristic data that exceeds the preset fault level; Key indicator feature data; the key indicator feature data refers to the preset indicator data collected when the target unmanned vehicle malfunctions; Model update feature data; the model update feature data refers to the feature data used to update the fault analysis network or the preset map.

[0050] It is understood that valid fault characteristic data refers to fault characteristic data that exceeds a preset fault level. The preset fault level is a pre-defined threshold or level used to determine whether fault characteristic data has high fault characterization value; different fault characteristic data can correspond to different preset fault levels. When the value of fault characteristic data exceeds its corresponding preset fault level, the fault characteristic data can be identified as valid fault characteristic data. For example, for vibration characteristic data, the preset fault level can be set to 5 mm / s (millimeters per second); when the vibration characteristic value reaches 8 mm / s, the characteristic data is marked as valid fault characteristic data. For temperature characteristic data, the preset fault level can be set to 105℃; when the temperature characteristic value reaches 110℃, the characteristic data is included in the valid fault characteristic data.

[0051] Key performance indicator (KPI) characteristic data refers to preset KPI data collected when the target unmanned vehicle malfunctions. These preset KPI data are a set of data indicators that require focused attention, predefined according to the type of malfunction. For example, for an "engine overheating" malfunction, preset KPIs might include engine coolant temperature, ambient temperature, engine speed, engine load rate, fan speed, vehicle speed, current gradient, and air conditioning status. These data are automatically collected as KPI characteristic data when this malfunction occurs. Similarly, for a "positioning signal loss" malfunction, preset KPIs might include GPS (Global Positioning System) signal strength, vehicle speed, steering wheel angle, current coordinates, packet loss rate, and communication latency. These data are also automatically collected as KPI characteristic data when this malfunction occurs.

[0052] Model update feature data refers to the feature data used to update the fault analysis network or preset map. For example, when the field manager finds that the actual cause of the fault is "poor contact of the fuel injector circuit" but the cloud-based fault analysis network diagnoses it as "fuel injector blockage", the original feature data of the fault and the actual repair results are packaged as model update feature data; when the system encounters a completely new and unseen fault mode, all relevant feature data of the fault and field repair records are used as model update feature data.

[0053] In one embodiment, the key fault feature data includes time-series feature data, fault code feature data, and image feature data; the fault analysis network includes a feature fusion subnetwork, a graph neural network, a time-series causal inference subnetwork, and a spatiotemporal alignment subnetwork. The step of analyzing the key fault characteristic data through a fault analysis network to determine the cause of the target unmanned vehicle's failure includes: The time-series feature data, fault code feature data, and image feature data are fused through the feature fusion subnetwork to obtain multimodal fused feature data; The graph neural subnetwork is used to perform fault correlation mining on the multimodal fusion feature data to identify the transmission and coupling relationships between faults. The temporal causal inference subnetwork performs temporal causal inference on the multimodal fused feature data to determine the fault source object and the fault-related object; wherein, the temporal causal inference subnetwork includes a temporal convolutional layer and a causal attention layer; The spatiotemporal alignment sub-network is used to spatiotemporally align the multimodal fusion feature data with environmental monitoring data to determine the correlation between faults and the environment. Based on the fault propagation and coupling relationships, fault source objects and fault-related objects, as well as the relationship between faults and the environment, the fault causes of the target unmanned vehicle are generated.

[0054] It is understandable that the above-mentioned analysis of key fault characteristic data through a fault analysis network to determine the cause of failure in the target unmanned vehicle may include: Multimodal fusion feature data is obtained by fusing time-series feature data, fault code feature data, and image feature data through a feature fusion subnetwork. Fault correlation mining is performed on multimodal fusion feature data through graph neural subnetworks to identify the transmission and coupling relationships between faults; A temporal causal inference subnetwork is used to perform temporal causal inference on multimodal fused feature data to determine the fault source object and the fault-related object; the temporal causal inference subnetwork includes a temporal convolutional layer and a causal attention layer; By using a spatiotemporal alignment subnetwork, multimodal fusion feature data and environmental monitoring data are spatiotemporally aligned to determine the correlation between faults and the environment; Based on the transmission and coupling relationships between faults, the fault source object and the fault-related object, as well as the relationship between the fault and the environment, the fault cause of the target unmanned vehicle is generated.

[0055] Temporal feature data refers to feature data with time-series characteristics, such as the curve characteristics of engine temperature changing over time, the waveform characteristics of vibration signals, and the time-series parameters of vehicle speed changing over time. Fault code feature data can be fault diagnostic codes or identification features related to fault types; for example, fault code feature data P0118 indicates a high input to the engine coolant temperature sensor circuit. Image feature data refers to visual features extracted from image data, such as edge features and texture features extracted from images of mining roads taken by cameras, or high-level semantic features extracted through convolutional neural networks.

[0056] Feature fusion subnetworks are subnetworks used to fuse feature data from multiple modalities into a unified multimodal fusion feature vector, such as attention-based fusion modules. Graph neural subnetworks are subnetworks based on graph neural networks used for correlation reasoning on heterogeneous information networks in vehicle systems. Temporal causal inference subnetworks are subnetworks used to analyze time-series data and infer causal relationships in faults; they include temporal convolutional layers and causal attention layers. Spatiotemporal alignment subnetworks are subnetworks used to align fault feature data with environmental monitoring data in terms of time and space dimensions.

[0057] Multimodal fusion feature data refers to comprehensive feature data that integrates time-series feature data, fault code feature data, and image feature data. Fault propagation and coupling relationships refer to the correlations of fault transmission and mutual influence within a vehicle system. This can include the propagation path of a single fault, such as an engine cooling fan motor failure leading to a decrease in coolant flow and consequently an increase in engine temperature, or the coupling relationship between multiple faults, such as "sensor A failure" and "controller B false alarm" occurring simultaneously and influencing each other. The fault source object refers to the fault object corresponding to the root cause of the fault event, such as "cooling fan not turning" or "fuel injector blockage." The fault associated object refers to the fault object corresponding to the accompanying phenomena affected by the fault source, such as "overheating" caused by "cooling fan not turning," or "engine vibration" caused by "fuel injector blockage." Environmental monitoring data can include, for example, ambient temperature, ambient humidity, dust concentration, wind speed, light intensity, and road surface water depth.

[0058] In this exemplary embodiment, three types of feature data are first input into a feature fusion subnetwork for fusion. For example, for an engine malfunction event in an unmanned mining truck, the system collects 128-dimensional temporal feature data (such as engine temperature change curves), 32-dimensional fault code feature data, and 256-dimensional image feature data (such as engine compartment image features). The feature fusion subnetwork uses a multi-head attention mechanism to fuse the three, outputting a 256-dimensional multimodal fusion feature vector. Subsequently, a graph neural network performs fault correlation mining on the fused feature. For example, it constructs a heterogeneous information network containing nodes such as the engine, transmission, hydraulic system, lidar, and communication module to identify the path of fault propagation from the "engine control unit" to the "transmission control unit," as well as the coupling relationship between "engine high temperature" and "air conditioning off." Next, a temporal causal inference subnetwork performs temporal causal inference on the fused feature. For example, it analyzes the data before and after the fault through temporal convolutional layers and causal attention layers to determine that "fan speed decrease" is the fault source object and "engine temperature increase" is the fault association object. Then, the spatiotemporal alignment sub-network aligns the fusion features with environmental monitoring data spatiotemporally, such as obtaining the ambient temperature, dust concentration, and vehicle location at the time of the fault occurrence, to determine whether the fault is related to environmental factors (such as high temperature and high dust). Finally, the above results are combined to generate the cause of the fault, for example, a diagnosis of "cooling fan motor failure leading to insufficient heat dissipation, which in turn causes engine overheating, with a confidence level of 89%".

[0059] Considering the high complexity of the sources of failure in unmanned vehicles in open-pit mines (intertwined with multiple systems such as mechanical, electrical, and communication), this exemplary embodiment can uncover the correlation and root causes between different failure sources through multimodal fusion and collaborative analysis of multiple sub-networks, effectively distinguishing between the root cause and the accompanying symptoms, and improving the accuracy and interpretability of fault diagnosis.

[0060] In one embodiment, the fault propagation and coupling relationship includes fault propagation paths and multi-fault point coupling relationships; The step of performing fault correlation mining on the multimodal fused feature data through the graph neural subnetwork to identify the transmission and coupling relationships between faults includes: Construct a heterogeneous information network for the target unmanned vehicle; the nodes of the heterogeneous information network include mechanical component nodes, electrical unit nodes, communication link nodes, and software module nodes, and the edges between each node represent physical connection relationships or data interaction relationships; The graph neural subnetwork is used to infer the inter-node relationships in the heterogeneous information network and determine the degree of fault correlation of each node. Based on the degree of fault correlation and the edges between nodes, the fault propagation path and the coupling relationship of multiple fault points are determined.

[0061] It is understandable that a heterogeneous information network refers to a graph structure network containing multiple types of nodes and edges. Mechanical component nodes refer to the nodes corresponding to components in a vehicle that perform mechanical functions, such as engine nodes, transmission nodes, and braking system nodes. Electrical unit nodes refer to the nodes corresponding to units in a vehicle that perform electrical control functions, such as battery management unit nodes and motor controller nodes. Communication link nodes refer to the nodes corresponding to links in a vehicle that perform data transmission functions, such as CAN (Controller Area Network) bus nodes and Ethernet nodes. Software module nodes refer to the nodes corresponding to software modules in a vehicle that perform algorithmic or logical functions, such as perception algorithm module nodes and planning and control module nodes. Physical connection relationships refer to the connections established between nodes through physical media (such as mechanical drive shafts and wiring harnesses), such as the mechanical transmission connection between engine nodes and transmission nodes, and the wiring harness connection between electrical unit nodes. Data interaction relationships refer to the relationships where nodes exchange data through communication networks.

[0062] Fault correlation refers to the strength of the correlation between each node and the current fault event, which can be represented by probability values ​​or scores. For example, a fault correlation of 0.92 for a node indicates that the node is highly likely to be a fault-related node. Fault propagation path refers to the path a fault travels between nodes, such as the path from the "engine node" to the "transmission node" and then to the "vehicle controller node". Multi-fault coupling refers to the mutual influence and correlation between multiple fault nodes, such as "sensor A failure" and "controller B false alarm" occurring simultaneously and influencing each other.

[0063] In this exemplary embodiment, a heterogeneous information network for the target unmanned vehicle is first constructed. For example, for a certain type of unmanned mining truck, a heterogeneous information network is constructed, including engine nodes, transmission nodes, hydraulic pump nodes, lidar nodes, millimeter-wave radar nodes, camera nodes, CAN bus nodes, perception algorithm nodes, and planning and control nodes. Engine nodes and transmission nodes are connected via mechanical transmission edges, and lidar nodes and perception algorithm nodes are connected via data interaction edges. Then, a graph neural network is used to infer the relationships between nodes. For example, a graph convolutional network is used to aggregate neighbor node information and calculate the fault correlation degree of each node under the current fault event. For example, the correlation degree of the engine node is 0.92, the correlation degree of the transmission node is 0.45, and the correlation degree of the lidar node is 0.13. Finally, the fault propagation path and multi-fault point coupling relationship are determined based on the fault correlation degree and the edges between nodes. For example, the fault propagation path is determined as "engine node - cooling system node - temperature sensor node" based on the nodes with high correlation degree (engine node, cooling system node) and their connecting edges. When multiple nodes with high correlation degree exist simultaneously and there is no clear unidirectional propagation path between them, it is identified as a multi-fault point coupling relationship.

[0064] Considering the tight coupling between the various systems of the unmanned vehicles in the mining area, this exemplary embodiment uses graph neural networks to perform inter-node correlation reasoning, which can automatically discover the propagation path and coupling relationship of faults inside the vehicle, providing managers with clear clues for fault tracing and reducing the time spent on blind investigation.

[0065] In one embodiment, the step of performing temporal causal inference on the multimodal fused feature data through the temporal causal inference subnetwork to determine the fault source object and fault-related objects includes: Temporal feature analysis is performed on the multimodal fusion feature data through the temporal convolutional layer and causal attention layer in the temporal causal inference subnetwork to determine the causal inference relationship chain; Based on the causal inference chain, the fault source object and the fault-related object are determined.

[0066] It is understandable that a temporal convolutional layer refers to a convolutional neural network layer used to extract features at different time scales from time-series data, such as using multiple convolutional kernels of different sizes (e.g., 3, 5, 7) to perform convolution operations on time-series data. A causal attention layer refers to an attention mechanism layer used to calculate the weights of the causal influence between different variables at different time delays. A causal inference chain refers to a chain-like structure describing the order of causal transmission between variables, for example, variable A (fan speed decreases) at time t causes variable B (engine temperature increases) at time t+Δt. A fault source object refers to the fault object corresponding to the root cause node in the causal chain; fault-related objects can be the fault objects corresponding to intermediate and terminal nodes in the causal chain.

[0067] In this exemplary embodiment, the temporal causal inference subnetwork first processes the temporal feature components in the multimodal fusion feature data. For example, it extracts temporal feature data of multiple variables such as engine temperature, fan speed, vehicle speed, and ambient temperature within 30 seconds before and after the fault occurs. The temporal convolutional layer extracts the temporal dependency features of each variable at different time scales through multi-scale convolution operations. The causal attention layer calculates the causal attention weights between different variables, such as the causal weight of fan speed on engine temperature being 0.87 and the causal weight of ambient temperature on engine temperature being 0.12. Based on these weights, a causal inference chain is constructed, for example, determining the causal inference chain of "fan speed decrease (t=0s) - coolant flow decrease (t=2s) - engine temperature increase (t=5s) - power decrease (t=8s)". Then, based on this causal inference chain, the fault source object is determined to be "fan motor" or "fan control circuit", and the fault-related objects are "engine high temperature" and "power decrease".

[0068] Considering the complex nonlinear temporal relationship between the malfunction phenomenon and the root cause of unmanned vehicles in the mining area, this exemplary embodiment uses temporal convolution and causal attention mechanisms to perform temporal causal inference, which can accurately distinguish between the root cause and the accompanying phenomena, and avoid misjudging the accompanying phenomena as the root cause of the malfunction.

[0069] In one embodiment, determining the fault guidance information matching the fault cause in the preset map includes: Using the cause of the fault as a query condition, semantic retrieval is performed in the preset graph to determine multiple historical fault events associated with the cause of the fault and their corresponding historical processing information. Based on the cause of the fault, multiple historical fault events, and the historical processing information corresponding to each historical fault event, fault guidance information matching the cause of the fault is generated.

[0070] Semantic retrieval refers to a retrieval method based on semantic similarity matching. Historical fault events refer to past fault instances recorded in a pre-defined graph, such as a previously recorded event of "fuel injector blockage causing misfire in the third cylinder." Historical processing information refers to maintenance records associated with historical fault events, specifically including maintenance operations, replaced parts, and test results. Maintenance operations refer to actions performed on the fault, such as "removing the fuel injector," "ultrasonic cleaning," or "reinstalling and testing." Replaced parts refer to parts replaced during maintenance, such as "fuel injector seals" or "spark plugs." Test results refer to the inspection conclusions verifying whether the fault has been eliminated after maintenance, such as "no abnormalities after 30 minutes of test run" or "engine vibration disappeared."

[0071] In this exemplary embodiment, the cause of the fault, such as "minor blockage of the fuel injector but not complete failure," can be used as a query condition to perform semantic retrieval in the maintenance knowledge graph. This recalls multiple historical fault events with semantically similar causes and their corresponding historical processing information. For example, historical cases of "severe blockage of the fuel injector" and their processing information (including disassembly, cleaning, and replacement of the seal ring) can be recalled. Then, by combining the specific characteristics of the current fault cause (such as a minor blockage and a short vehicle operating time) with the recalled historical cases, customized fault guidance information is generated through retrieval enhancement generation technology. This fault guidance information may include: inspection items, such as "check the fuel injector circuit resistance" or "check the fuel injector spray pattern"; troubleshooting order, such as "electrical inspection first, then mechanical disassembly"; repair steps, such as "disassemble the fuel injector - ultrasonic cleaning - replace the seal ring - reinstall and test," and suggestions on whether cleaning is sufficient and replacement is not necessary can be made based on the degree of blockage; a list of replacement parts, such as "fuel injector seal ring × 4," which may not need to be replaced if the wear is minor; and safety precautions, such as "wait for the engine to cool down before operating."

[0072] Considering the potential differences between various fault cases, this exemplary embodiment recalls similar cases through semantic retrieval and generates customized guidance information by combining current fault characteristics, thereby improving the relevance and practicality of maintenance guidance.

[0073] In an exemplary embodiment, the aforementioned preset graph includes a maintenance knowledge graph, which can be constructed through the following steps: Entities and relationships are extracted from vehicle design manuals, historical repair work orders, fault cause databases, and expert experience documents. Entities include fault phenomena, fault codes, root causes of faults, repair operations, replaced parts, and test results. Relationships include causal relationships, sequential relationships, ownership relationships, and tool-related relationships. Entities and relationships are stored in a graph database in a graph structure to form a maintenance knowledge graph.

[0074] Among these, a maintenance knowledge graph refers to a knowledge base that stores maintenance-related knowledge in a graph structure, where nodes represent entities and edges represent relationships between entities. Vehicle design manuals refer to technical documents provided by the vehicle manufacturer, such as engine maintenance manuals, transmission technical manuals, and vehicle electrical schematics. Historical maintenance work orders refer to documents formed from past maintenance records, such as maintenance records for all unmanned mining trucks within the past three years. A fault causality database refers to a verified database of fault causal analysis results, such as a causal chain recording "fuel injector blockage" leading to "third cylinder misfire." Expert experience documents refer to troubleshooting guides or experience summaries written by senior technicians.

[0075] Entities are nodes in a maintenance knowledge graph, such as fault symptoms (e.g., "engine vibration," "overheating"), fault codes, root causes (e.g., "clogged fuel injectors," "cooling fan not turning"), maintenance operations (e.g., "cleaning fuel injectors," "replacing fan motor"), replacement parts (e.g., "fuel injector seals," "fan motor"), and test results (e.g., "no abnormalities after 30 minutes of trial run"). Relationships are edges in a maintenance knowledge graph, used to connect entities. Causal relationships indicate that one entity causes another entity to occur, for example, "clogged fuel injectors" causes "engine vibration." Sequential relationships indicate the order in which maintenance operations are performed, for example, "removing fuel injectors" precedes "cleaning fuel injectors." Belonging relationships indicate the affiliation between entities, for example, "fuel injectors" belong to "engine fuel system." Tool-use relationships indicate the tools required for maintenance operations, for example, "replacing fuel injectors" requires a "socket wrench." Graph databases refer to databases specifically designed for storing and querying graph-structured data, such as Neo4j and JanusGraph.

[0076] In this exemplary embodiment, entities and relationships are first extracted from the aforementioned multi-source documents using natural language processing techniques. For example, component entities such as "fuel injector" and "sealing ring" are extracted from the engine repair manual, as well as operation entities such as "disassembly" and "installation." The causal relationship between the fault symptom "overheating" and the repair operation "replacing the thermostat" is extracted from historical repair work orders. The tool relationship between "checking the fuel injector circuit resistance" and "using a multimeter" is extracted from expert experience documents. Then, the extracted entities are stored as nodes and the relationships as edges in a graph database to construct a complete repair knowledge graph.

[0077] Considering that the maintenance of unmanned vehicles in mining areas involves complex electromechanical and software integrated systems, this exemplary embodiment constructs a maintenance knowledge graph to structurally store maintenance knowledge scattered in multiple source documents, providing a knowledge foundation for the rapid retrieval and generation of subsequent fault guidance information.

[0078] In an exemplary embodiment, after sending the fault guidance information to the fault management terminal, the method may further include: Receive fault feedback data sent by the fault management terminal; the fault feedback data includes one or more of the following: actual fault cause, component update information, processing time, and vehicle operating status after processing. Based on the fault feedback data, one or more of the feature extraction network, fault analysis network, and preset graphs are optimized and updated.

[0079] The fault feedback data refers to the data reported by management personnel after the fault has been resolved. The actual fault cause refers to the real cause of the fault confirmed on-site by management personnel, such as "poor contact in the fuel injector circuit" or "cooling fan motor jamming." Component update information refers to records of components replaced or repaired during the repair process, such as "replaced the fuel injector seal" or "cleaned the lidar lens." Processing time refers to the time taken from the start to the completion of the process, such as "45 minutes." Post-processing vehicle operating status refers to the status data of the vehicle when it is put into trial operation or normal use after the repair, such as "no abnormalities after 2 hours of trial operation" or "engine temperature has returned to normal."

[0080] Optimization and updates refer to adjusting and improving feature extraction networks, fault analysis networks, or preset maps using fault feedback data. For example, when the actual fault cause is inconsistent with the diagnostic results output by the fault analysis network, the feature data of the fault and the actual fault cause are used as new training samples to incrementally train the fault analysis network, enabling it to correctly identify similar faults in the future. Or, when fault data reported by management personnel indicates that the features extracted by the vehicle-side lightweight feature extraction network have deviations, the fault feedback data can be used for incremental training of the feature extraction network to optimize its feature extraction capabilities. Alternatively, when the preset map lacks maintenance knowledge for a certain fault, the actual fault cause and corresponding fault handling information are added to the preset map as new entities and relationships.

[0081] In this exemplary embodiment, the fault management terminal sends back the fault feedback data filled in by the management personnel. After receiving this data, the system analyzes it and optimizes and updates one or more of the feature extraction network, fault analysis network, and preset map. For example, if the system previously diagnosed "fuel injector blockage" as the cause of the fault, but the management personnel reported that the actual cause of the fault was "poor contact in the fuel injector circuit," then the system will match the feature data of the fault with the actual cause for subsequent optimization of the fault analysis network. Or, if the management personnel reported "dirty lidar lens causing point cloud anomalies," and the feature extraction network failed to effectively extract the relevant features, then the feedback data can be used to adjust the parameters of the feature extraction network so that it can better identify the feature patterns of lens dirt. Alternatively, if the system finds that there is no relevant knowledge about the fault in the preset map, then the inspection items and cleaning steps reported by the management personnel will be added to the preset map to enrich the knowledge base.

[0082] Considering that fault feedback data is an important resource for optimizing fault diagnosis models and knowledge graphs, this exemplary embodiment achieves a closed loop from fault diagnosis to maintenance verification by receiving fault feedback data and sending it to the system, enabling the feature extraction network, fault analysis network, and preset knowledge graph to continuously learn and optimize from real-world cases.

[0083] In an exemplary embodiment, the aforementioned fault feedback data can be collected in the following manner: Managers record fault feedback data through a fault management terminal using one or more of the following methods: structured forms, voice input, and image upload. The fault feedback data includes at least one of the following: maintenance execution status, actual fault cause, information on replaced parts, and vehicle operating status after maintenance. Establish a unique association between the recorded fault feedback data and the cause of the fault.

[0084] The fault feedback data refers to the data provided by management personnel after handling a fault. Structured forms are spreadsheets containing pre-defined fields, such as "actual fault cause," "replaced parts," and "repair time." Voice input refers to management personnel describing the fault handling process via voice input, such as saying to the fault management terminal, "The fault cause is poor contact in the fuel injector circuit; the seal has been replaced." Image upload refers to management personnel taking and uploading photos of the faulty parts or the repair site, such as photos of worn fuel injector parts or before-and-after cleaning comparisons. Repair execution status is a brief description of the fault handling process, such as "The fuel injector was disassembled, ultrasonic cleaning was performed, and the seal was replaced." Actual fault cause refers to the actual cause of the fault confirmed by management personnel on-site, such as "poor contact in the fuel injector circuit." Replaced parts information refers to the name and quantity of parts replaced during the repair process, such as "4 fuel injector seals." Post-repair vehicle operating status can be the state of the vehicle's first start or test run after repair, such as "Starting normally, engine vibration disappeared." Unique association identifier is a unique code used to bind the fault feedback data to the previously diagnosed fault cause. The purpose of establishing a unique associated identifier is to ensure that the corresponding fault case can be accurately traced during subsequent optimization and updates.

[0085] In this exemplary embodiment, after on-site management personnel complete fault handling, they can use a fault management terminal, such as a tablet or smartphone, to fill in various items using a structured form, or directly record the repair process via voice, or upload photos of the faulty components. The system automatically generates a unique identifier, associates the fault feedback data with the corresponding fault cause, and then transmits it back to the cloud platform. For example, if a management personnel finds that a previously diagnosed "fuel injector blockage" fault is actually caused by "poor circuit contact," they will fill in the actual cause in the form, add a voice explanation that "the plug oxidation caused poor contact," and upload a photo of the plug. The system packages these three types of information and stores them in association with the unique identifier of the fault cause.

[0086] To ensure ease of operation for on-site management personnel and completeness of data recording, this exemplary embodiment provides multiple data entry methods, reducing the burden of data entry. At the same time, the unique associated identifier ensures the traceability of fault feedback data and diagnostic results, providing a reliable data foundation for subsequent model optimization.

[0087] In an exemplary embodiment, the system can associate and store key data during the fault occurrence and handling process to form a traceable and complete case record. Specifically, after acquiring abnormal state data of the target unmanned vehicle, the system sequentially performs feature extraction and fault analysis to generate fault causes, and then obtains fault guidance information through preset graph matching and sends it to the fault management terminal. After completing the repair according to the guidance information, the management personnel upload fault feedback data (including actual fault causes, information on replaced parts, processing time, and vehicle operating status after repair) through the fault management terminal. The system associates the fault causes, fault guidance information, fault feedback data uploaded by the management personnel, and subsequent tracked vehicle operating status data in the above process, assigns them the same case identifier, and stores them in the case library. For example, a complete case record of a fault may include the fault phenomenon as "engine vibration, fault code 0303", the diagnosis result as "fuel injector blockage", the repair measures as "disassembling the fuel injector, ultrasonic cleaning, and replacing the sealing ring", and the repair effect as "no abnormalities after 30 minutes of trial operation, vibration disappears". This information is stored in a structured form to facilitate subsequent fault statistics, root cause analysis, model training, and knowledge graph updates.

[0088] By forming a complete case data chain, the system can accumulate reusable knowledge from individual fault handling, providing real and reliable data support for the incremental learning of the fault analysis network and the continuous expansion of the preset map. At the same time, it supports managers to retrieve historical cases by time, vehicle, fault type and other conditions, improving the overall efficiency of fault management.

[0089] Figure 2 A flowchart illustrating another autonomous vehicle fault management method is provided, applied to an autonomous vehicle fault management system. This system may include an autonomous vehicle terminal 210, a cloud platform 220, and a fault management terminal 230. The autonomous vehicle terminal 210 can execute: Step S211: Obtain the original state data of the target unmanned vehicle.

[0090] Step S212: If the original state data meets the preset abnormal conditions, perform data standardization and cleaning on the original state data to obtain the abnormal state data of the target unmanned vehicle.

[0091] Step S213: Process the abnormal state data through a feature extraction network to extract key fault feature data from the abnormal state data.

[0092] Step S214: Determine the data type of the key fault characteristic data and process it using the processing method corresponding to the data type.

[0093] The key fault feature data can be categorized into key feature vectors, key event trigger data, and steady-state operation summaries. Key event trigger data refers to the raw data packets automatically collected by the system before and after a preset key event (such as fault code triggering, sensor exceeding threshold, or communication interruption). Steady-state operation summaries refer to the statistical characteristics and summary information of each data channel when the autonomous vehicle is operating normally. In this exemplary embodiment, valid fault feature data can be uploaded in the form of key feature vectors, key indicator feature data can be uploaded in the form of key event trigger data, and model update feature data can be uploaded in the form of steady-state operation summaries.

[0094] Specifically, for feature vectors, a real-time compression upload strategy can be adopted to ensure the real-time transmission of core fault features, ensuring that the cloud can obtain key information for fault analysis as soon as possible; for key event-triggered data, an event-driven upload strategy can be adopted to upload the original data packets before and after the event to the cloud completely for subsequent fault backtracking analysis and deep root cause mining; for steady-state operation summaries, a statistical feature upload strategy can be adopted to upload only the statistical summary of the data instead of all the original data, thereby significantly reducing the amount of data transmitted during non-fault periods and reducing communication bandwidth usage and storage costs.

[0095] Step S215: Send the key fault characteristic data to the cloud platform.

[0096] The cloud platform 220 is configured with a fault analysis network, which includes a feature fusion subnetwork, a graph neural subnetwork, a temporal causal inference subnetwork, and a spatiotemporal alignment subnetwork.

[0097] Cloud platform 220 can execute: Step S221: The time-series feature data, fault code feature data and image feature data are fused through the feature fusion sub-network to obtain multimodal fused feature data.

[0098] Step S222: Fault correlation mining is performed on the multimodal fusion feature data through graph neural subnetwork to identify the transmission and coupling relationships between faults.

[0099] Step S223: Perform temporal causal inference on the multimodal fusion feature data through the temporal causal inference sub-network to determine the fault source object and the fault-related object; wherein, the temporal causal inference sub-network includes a temporal convolutional layer and a causal attention layer.

[0100] Step S224: The multimodal fusion feature data and environmental monitoring data are spatiotemporally aligned through a spatiotemporal alignment sub-network to determine the correlation between the fault and the environment.

[0101] Step S225: Based on the fault propagation and coupling relationship, fault source object and fault associated object, and the relationship between fault and environment, generate the fault cause of the target unmanned vehicle.

[0102] Step S226: Determine whether the target unmanned vehicle has a substantial fault, i.e., a real fault, rather than a false fault, based on the cause of the fault. For example, if the cause of the fault can be attributed to a real problem in the vehicle's mechanical, electrical, or software systems, then it is a substantial fault; if the cause of the fault is caused by external environmental interference (such as dust obscuring the lidar, or instantaneous vibration causing sensor false alarms) or communication abnormalities (such as packet loss or delay), and there is no damage to the vehicle's own components, then it is a false fault.

[0103] If it is a substantive fault, then step S227 is executed, a preset map is called, fault guidance information matching the cause of the fault is determined in the preset map, and the fault guidance information is sent to the fault management terminal.

[0104] If the fault is non-substantial, then proceed to step S228 to mark the fault as non-substantial and not trigger the maintenance management process.

[0105] Fault management terminal 230 can perform the following: Step S231: Receive fault guidance information so that managers can perform maintenance management on the target unmanned vehicle according to the fault guidance information.

[0106] Step S232: Receive fault feedback data uploaded by the user; the fault feedback data includes one or more of the following: actual fault cause, component update information, processing time, and vehicle operating status after processing.

[0107] Step S233: Return the fault feedback data to the cloud platform so that the cloud platform can optimize and update the feature extraction network and fault analysis network based on the fault feedback data.

[0108] In this exemplary embodiment, based on fault feedback data, incremental iterations of the fault analysis network and preset maps can be achieved, forming a self-evolving closed loop of "diagnosis-feedback-optimization". Specifically, the system can use real repair cases returned by management personnel as the data source to drive model iteration. Cases with successful repairs and accurate diagnoses are designated as positive samples. For example, if the system diagnoses a fault as "fuel injector blockage," but management personnel report the actual cause as "fuel injector blockage," and the fault is resolved after repair, this case is marked as a positive sample. The system strengthens the fault analysis network's ability to identify effective fault features through comparative learning, enabling the model to output diagnostic results faster and more accurately when facing similar faults. Conversely, cases with incorrect diagnoses or misjudgments are designated as negative samples. For example, if the system diagnoses "fuel injector blockage," but management personnel report the actual cause as "poor contact in the fuel injector circuit," this case is marked as a negative sample. The system adjusts model parameters through a backpropagation algorithm to correct diagnostic biases and reduce the misjudgment rate of subsequent similar faults. During model parameter updates, a flexible weight fixation technique can be employed to ensure that while introducing new knowledge, the model does not forget previously learned historical knowledge. For example, when the model learns a new fault mode, "poor contact in the fuel injector circuit," it retains its ability to identify older fault modes such as "fuel injector blockage" and "cooling fan failure." This approach allows for the updating of parameters in both lightweight networks and fault analysis networks.

[0109] This exemplary embodiment introduces positive and negative sample comparison learning, backpropagation adjustment, and elastic weight solidification techniques to achieve incremental iteration and full-link deployment updates of the network, enabling the system to continuously learn and evolve from actual maintenance cases and adapt to complex and ever-changing fault scenarios in mining areas.

[0110] Secondly, please refer to Figure 3 A fault management device for unmanned vehicles is provided, comprising: The status data acquisition module is used to acquire abnormal status data of the target unmanned vehicle; The feature data extraction module is used to process the abnormal state data through a feature extraction network to extract key fault feature data from the abnormal state data. The fault cause analysis module is used to analyze the key fault characteristic data through a fault analysis network to determine the fault cause of the target unmanned vehicle. The guidance information determination module is used to determine the fault guidance information that matches the cause of the fault in the preset map, and send the fault guidance information to the fault management terminal; The feedback receiving module is used to receive fault feedback data sent by the fault management terminal; the fault feedback data includes one or more of the following: actual fault cause, component update information, processing time, and vehicle operating status after processing. The model update module is used to optimize and update one or more of the feature extraction network, fault analysis network, and preset map based on the fault feedback data.

[0111] In one exemplary embodiment, the state data acquisition module includes: a raw data acquisition unit for acquiring raw state data of the target unmanned vehicle; and a data standardization unit for performing data standardization and cleaning on the raw state data to obtain abnormal state data of the target unmanned vehicle when it is determined that the raw state data meets preset anomaly conditions. The data standardization and cleaning of the raw state data includes one or more of the following: removing instantaneous outliers from the raw state data based on a preset numerical range; calculating invalid scores for the raw state data based on the isolated forest algorithm and removing raw state data with invalid scores exceeding a preset score threshold; interpolating and supplementing missing data segments in the raw state data; and verifying the authenticity of the raw state data and removing false state data.

[0112] In an exemplary embodiment, the key fault feature data includes at least one of the following: valid fault feature data; valid fault feature data refers to fault feature data that exceeds a preset fault level; key indicator feature data; key indicator feature data refers to preset indicator data collected when the target unmanned vehicle malfunctions; model update feature data; model update feature data refers to feature data used to update the fault analysis network or preset map.

[0113] In an exemplary embodiment, key fault feature data includes time-series feature data, fault code feature data, and image feature data; the fault analysis network includes a feature fusion subnetwork, a graph neural network, a time-series causal inference subnetwork, and a spatiotemporal alignment subnetwork; the fault cause analysis module includes: a feature fusion unit, used to fuse the time-series feature data, fault code feature data, and image feature data through the feature fusion subnetwork to obtain multimodal fused feature data; and an association mining unit, used to perform fault association mining on the multimodal fused feature data through the graph neural network to identify the transmission and coupling relationships between faults. The system comprises: a causal inference unit, used to perform temporal causal inference on multimodal fusion feature data through a temporal causal inference subnetwork to determine the fault source object and fault-related objects; wherein the temporal causal inference subnetwork includes a temporal convolutional layer and a causal attention layer; a spatiotemporal alignment unit, used to perform spatiotemporal alignment of multimodal fusion feature data and environmental monitoring data through a spatiotemporal alignment subnetwork to determine the correlation between faults and the environment; and a fault cause generation unit, used to generate the fault cause of the target unmanned vehicle based on the transmission and coupling relationship between faults, the fault source object and the fault-related objects, and the correlation between faults and the environment.

[0114] In an exemplary embodiment, the fault propagation and coupling relationships include fault propagation paths and multi-fault point coupling relationships; the association mining unit includes: a heterogeneous information network construction subunit, used to construct a heterogeneous information network of the target unmanned vehicle; the nodes of the heterogeneous information network include mechanical component nodes, electrical unit nodes, communication link nodes, and software module nodes, and the edges between each node represent physical connection relationships or data interaction relationships; a node reasoning subunit, used to perform inter-node association relationship reasoning on the heterogeneous information network through a graph neural network to determine the degree of fault association of each node; and a coupling relationship determination subunit, used to determine the fault propagation path and multi-fault point coupling relationships based on the degree of fault association and the edges between nodes.

[0115] In an exemplary embodiment, the causal inference unit includes: a relation chain determination subunit, configured to perform temporal feature analysis on multimodal fusion feature data through temporal convolutional layers and causal attention layers in the temporal causal inference subnetwork to determine the causal inference relation chain; and an object determination subunit, configured to determine the fault source object and the fault associated object based on the causal inference relation chain.

[0116] In an exemplary embodiment, the guidance information determination module includes: a semantic retrieval unit, used to perform semantic retrieval in a preset graph using the cause of the fault as a query condition, to determine multiple historical fault events associated with the cause of the fault and their corresponding historical processing information; and an information generation unit, used to generate fault guidance information matching the cause of the fault based on the cause of the fault, multiple historical fault events and the historical processing information corresponding to each historical fault event.

[0117] In an exemplary embodiment, after sending fault guidance information to the fault management terminal, the device further includes: a feedback data receiving module, used to receive fault feedback data sent by the fault management terminal; the fault feedback data includes one or more of the following: actual fault cause, component update information, processing time, and vehicle operating status after processing; and a network optimization module, used to optimize and update one or more of the feature extraction network, fault analysis network, and preset map based on the fault feedback data.

[0118] The specific details of each part of the above-mentioned device have been described in detail in the method section of the implementation plan. For any undisclosed details, please refer to the implementation plan of the method section, and therefore will not be repeated here.

[0119] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to exemplary embodiments of this disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0120] Thirdly, an electronic device is provided, which is exemplarily described in the form of a general-purpose computing device. It should be understood that... Figure 4 The electronic device 400 shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein. The electronic device 400 may include: A memory on which computer programs are stored; A processor for executing the computer program in the memory to implement the steps of the method described in any of the above embodiments.

[0121] Memory 420 may include volatile memory, such as RAM 421 and cache unit 422, and may also include non-volatile memory, such as ROM 423. Memory 420 may also include one or more program modules 424, including but not limited to: operating system, one or more application programs, other program modules, and program data. Each or some combination of these examples may include an implementation of a network environment. For example, program module 424 may include the modules in the above-described apparatus.

[0122] The processor 410 may include one or more processing units, such as an AP (Application Processor), a modem processor, a GPU (Graphics Processing Unit), an ISP (Image Signal Processor), a controller, an encoder, a decoder, a DSP (Digital Signal Processor), a baseband processor, and / or an NPU (Neural-Network Processing Unit).

[0123] The processor 410 can be used to execute executable instructions stored in the memory 420, such as the above-mentioned unmanned vehicle fault management method.

[0124] Bus 430 is used to connect different components of electronic device 400 and may include a data bus, an address bus and a control bus.

[0125] Electronic device 400 can communicate with one or more external devices 500 (such as keyboard, mouse, external controller, etc.) through I / O interface 440.

[0126] Electronic device 400 can communicate with one or more networks via network adapter 450. For example, network adapter 450 can provide mobile communication solutions such as 3G / 4G / 5G, or wireless communication solutions such as wireless LAN, Bluetooth, and near-field communication. Network adapter 450 can communicate with other modules of electronic device 400 via bus 430.

[0127] Electronic device 400 can display a graphical user interface via display 460.

[0128] although Figure 4 Other hardware and / or software modules, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, may also be configured in the electronic device 400.

[0129] As can be seen from the above, the technical solutions disclosed herein can be implemented as methods, devices, electronic devices, etc. Those skilled in the art will understand that various aspects of this disclosure can be specifically implemented in the following forms: a completely hardware implementation, a completely software implementation (including firmware, microcode, etc.), or an implementation combining hardware and software aspects, which may be referred to as "circuit", "module" or "system" respectively.

[0130] Exemplary embodiments of this disclosure also provide a computer program product. The computer program product includes a computer program that, when executed by a processor, implements the aforementioned unmanned vehicle fault management method.

[0131] In one implementation, the computer program product can be a tangible product containing a computer program, such as a computer-readable storage medium storing the computer program. The readable storage medium can be a storage medium based on electrical, magnetic, optical, electromagnetic, infrared, or other signals, including but not limited to: random access memory (RAM), read-only memory (ROM), magnetic tape, floppy disk, flash memory, hard disk drive (HDD), solid-state drive (SSD), etc. For example, the computer program product can be implemented as a non-volatile storage medium storing a computer program, such as read-only memory, NAND flash memory, etc.

[0132] In one implementation, the computer program product can be an intangible product containing a computer program. For example, the computer program product can be implemented as a virtual digital product, such as an executable file, installation package, or other digital file storing the computer program.

[0133] Computer program code can be written in one or more programming languages. Examples of programming languages ​​include C, Java, and C++. Program code can execute entirely on the user's computing device, partially on the user's computing device, or as a standalone software package. It can also execute partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, such as a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via an internet connection provided by a mobile network operator).

[0134] Computer programs can be carried or transmitted via signals such as electricity, magnetism, light, electromagnetic radiation, and infrared radiation. Electronic devices can convert the signals carrying computer programs into digital signals, thereby running the computer programs. When a computer program runs on an electronic device, its code is used to cause the electronic device to execute (more specifically, to execute by the processor of the electronic device) the method steps of various exemplary embodiments of this disclosure, such as the aforementioned unmanned vehicle fault management method.

[0135] Exemplary embodiments of this disclosure also provide an electronic device. The electronic device may include a processor and a memory. The memory stores executable instructions of the processor, such as program code. The processor executes the executable instructions to perform the methods of this exemplary embodiment. Furthermore, the electronic device may also include a display for displaying a graphical user interface.

[0136] The preferred embodiments of this disclosure have been described in detail above with reference to the accompanying drawings. However, this disclosure is not limited to the specific details of the above embodiments. Within the scope of the technical concept of this disclosure, various simple modifications can be made to the technical solutions of this disclosure, and these simple modifications all fall within the protection scope of this disclosure.

[0137] It should also be noted that the various specific technical features described in the above specific embodiments can be combined in any suitable manner without contradiction. In order to avoid unnecessary repetition, this disclosure will not describe the various possible combinations separately.

[0138] Furthermore, various different embodiments of this disclosure can be combined in any way, as long as they do not violate the spirit of this disclosure, they should also be regarded as the content disclosed in this disclosure.

Claims

1. A fault management method for unmanned vehicles, characterized in that, include: Acquire abnormal state data of the target unmanned vehicle; The abnormal state data is processed by a feature extraction network to extract key fault feature data from the abnormal state data. The key fault characteristic data are analyzed by a fault analysis network to determine the cause of the target unmanned vehicle's failure. Determine fault guidance information that matches the cause of the fault in the preset map, and send the fault guidance information to the fault management terminal; Receive fault feedback data sent by the fault management terminal; The fault feedback data includes one or more of the following: actual fault cause, component update information, processing time, and vehicle operating status after processing. Based on the fault feedback data, one or more of the feature extraction network, fault analysis network, and preset map are optimized and updated.

2. The unmanned vehicle fault management method according to claim 1, characterized in that, The acquisition of abnormal state data of the target unmanned vehicle includes: Obtain the original state data of the target unmanned vehicle; If the original state data is determined to meet the preset abnormal conditions, the original state data is standardized and cleaned to obtain the abnormal state data of the target unmanned vehicle.

3. The unmanned vehicle fault management method according to claim 2, characterized in that, The data standardization and cleaning of the original state data includes one or more of the following: Instantaneous outliers are removed from the original state data based on a preset numerical range; The invalid scores of the original state data are calculated based on the isolated forest algorithm, and the original state data with invalid scores exceeding a preset score threshold are removed. The missing data segments in the original state data are supplemented by interpolation; The authenticity of the original state data is verified, and false state data is removed.

4. The unmanned vehicle fault management method according to claim 1, characterized in that, The key fault characteristic data includes at least one of the following: Valid fault feature data: The valid fault feature data refers to fault feature data that exceeds the preset fault level; Key indicator feature data: The key indicator feature data refers to the preset indicator data collected when the target unmanned vehicle malfunctions; Model update feature data: The model update feature data refers to the feature data used to update the fault analysis network or the preset map.

5. The unmanned vehicle fault management method according to claim 4, characterized in that, The key fault feature data includes time-series feature data, fault code feature data, and image feature data; the fault analysis network includes a feature fusion subnetwork, a graph neural subnetwork, a time-series causal inference subnetwork, and a spatiotemporal alignment subnetwork. The step of analyzing the key fault characteristic data through a fault analysis network to determine the cause of the target unmanned vehicle's failure includes: The time-series feature data, fault code feature data, and image feature data are fused through the feature fusion subnetwork to obtain multimodal fused feature data; The graph neural subnetwork is used to perform fault correlation mining on the multimodal fusion feature data to identify the transmission and coupling relationships between faults. The temporal causal inference subnetwork performs temporal causal inference on the multimodal fused feature data to determine the fault source object and the fault-related object; wherein, the temporal causal inference subnetwork includes a temporal convolutional layer and a causal attention layer; The spatiotemporal alignment sub-network is used to spatiotemporally align the multimodal fusion feature data with environmental monitoring data to determine the correlation between faults and the environment. Based on the fault propagation and coupling relationships, fault source objects and fault-related objects, as well as the relationship between faults and the environment, the fault causes of the target unmanned vehicle are generated.

6. The unmanned vehicle fault management method according to claim 5, characterized in that, The fault propagation and coupling relationships include fault propagation paths and multi-fault point coupling relationships. The step of performing fault correlation mining on the multimodal fused feature data through the graph neural subnetwork to identify the transmission and coupling relationships between faults includes: Construct a heterogeneous information network for the target unmanned vehicle; the nodes of the heterogeneous information network include mechanical component nodes, electrical unit nodes, communication link nodes, and software module nodes, and the edges between each node represent physical connection relationships or data interaction relationships; The graph neural subnetwork is used to infer the inter-node relationships in the heterogeneous information network and determine the degree of fault correlation of each node. Based on the degree of fault correlation and the edges between nodes, the fault propagation path and the coupling relationship of multiple fault points are determined.

7. The unmanned vehicle fault management method according to claim 5, characterized in that, The step of performing time-series causal inference on the multimodal fused feature data through the time-series causal inference subnetwork to determine the fault source object and fault-related objects includes: Temporal feature analysis is performed on the multimodal fusion feature data through the temporal convolutional layer and causal attention layer in the temporal causal inference subnetwork to determine the causal inference relationship chain; Based on the causal inference chain, the fault source object and the fault-related object are determined.

8. The unmanned vehicle fault management method according to claim 1, characterized in that, The step of determining the fault guidance information that matches the cause of the fault in the preset map includes: Using the cause of the fault as a query condition, semantic retrieval is performed in the preset graph to determine multiple historical fault events associated with the cause of the fault and their corresponding historical processing information. Based on the cause of the fault, multiple historical fault events, and the historical processing information corresponding to each historical fault event, fault guidance information matching the cause of the fault is generated.

9. A fault management device for unmanned vehicles, characterized in that, include: The status data acquisition module is used to acquire abnormal status data of the target unmanned vehicle; The feature data extraction module is used to process the abnormal state data through a feature extraction network to extract key fault feature data from the abnormal state data. The fault cause analysis module is used to analyze the key fault characteristic data through a fault analysis network to determine the fault cause of the target unmanned vehicle. The guidance information determination module is used to determine the fault guidance information that matches the cause of the fault in the preset map, and send the fault guidance information to the fault management terminal; The feedback receiving module is used to receive fault feedback data sent by the fault management terminal; The fault feedback data includes one or more of the following: actual fault cause, component update information, processing time, and vehicle operating status after processing. The model update module is used to optimize and update one or more of the feature extraction network, fault analysis network, and preset map based on the fault feedback data.

10. An electronic device, characterized in that, include: A memory on which computer programs are stored; A processor for executing the computer program in the memory to implement the steps of the method according to any one of claims 1-8.

Images