Vehicle fault detection method, device and equipment, storage medium and vehicle

Abstract

The invention discloses a vehicle fault detection method, device and equipment, a storage medium and a vehicle, and relates to the technical field of vehicles. The method comprises the following steps: acquiring multi-modal data of a target vehicle; obtaining a whole vehicle three-dimensional structure model of the target vehicle; the multi-modal data and the whole vehicle three-dimensional structure model serve as input of a trained fault detection model, and a fault detection result is obtained; the fault detection model is obtained by training a fault case library and is used for carrying out correlation analysis on fault causes according to the multi-modal data and the whole vehicle three-dimensional structure model; the fault detection result comprises a fault reason of the fault point and position information of the fault point in the whole vehicle three-dimensional structure model. Compared with the prior art, the method has the advantages that the model is pre-trained through the fault case library, so that the fault detection model is obtained, meanwhile, the fault reason is integrally analyzed based on the multi-modal data of the fault case in different dimensions and the three-dimensional structure of the whole vehicle in the training, the detection efficiency is improved, and meanwhile, the accuracy is also improved compared with a manual judgment mode.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of vehicles, in particular to a vehicle fault detection method, device, equipment, storage medium and vehicle. BACKGROUND

[0002] In the development and manufacturing process of commercial vehicles, electrical problems often bring great distress to engineers and maintenance personnel. Currently, the traditional fault diagnosis of the electrical system of commercial vehicles mainly relies on maintenance personnel to obtain the fault code of the ECU (Electronic Control Unit) through the OBD (On-Board Diagnostics) interface, and to analyze the wire harness terminals and detect the fault codes one by one according to the wire harness principle. In addition, the analysis of the problem also depends a lot on the past experience of the maintenance personnel. This way takes a long time to detect, is low in efficiency, and is not accurate. SUMMARY

[0003] Therefore, the present application provides a vehicle fault detection method, device, equipment, storage medium and vehicle, which mainly aims to improve the low efficiency and low accuracy of the current vehicle fault detection.

[0004] In a first aspect, the present application provides a vehicle fault detection method, comprising:

[0005] Obtaining multi-modal data of a target vehicle, wherein the multi-modal data comprises data collected in different dimensions under a fault condition of the target vehicle;

[0006] Obtaining a three-dimensional structure model of the target vehicle;

[0007] Taking the multi-modal data and the three-dimensional structure model of the vehicle as inputs of a trained fault detection model to obtain a fault detection result; the fault detection model is obtained by training a fault case library, and is used for associating and analyzing the fault reason according to the multi-modal data and the three-dimensional structure model of the vehicle;

[0008] The fault detection result comprises a fault reason of a fault point and position information of the fault point in the three-dimensional structure model of the vehicle.

[0009] Optionally, the fault detection result further comprises a probability corresponding to the fault reason; after obtaining the fault detection result and the probability corresponding to the fault reason, the method further comprises: comparing a maximum value of the probability corresponding to the fault reason with a preset threshold value, and in the case that the maximum value of the probability corresponding to the fault reason is less than the preset threshold value, marking the current fault detection result and triggering an artificial review process.

[0010] Optionally, the method further comprises: in a case where the maximum value of the failure cause corresponding probability is less than the preset threshold, updating the current multi-modal data and the whole vehicle three-dimensional structure model to the failure detection model.

[0011] Optionally, the training step of the failure detection model comprises: loading a failure case library, the failure case library comprising a failure cause of a failure case, a whole vehicle three-dimensional structure model of a failure vehicle, position information of the failure point in the whole vehicle three-dimensional structure model, and multi-modal data of the failure vehicle; correlatively training the failure cause and the corresponding multi-modal data by cross-modal contrast learning alignment data representation; and obtaining the failure detection model based on the correlatively training of the failure cause, the multi-modal data, and the position information of the failure point in the whole vehicle three-dimensional structure model.

[0012] Optionally, the multi-modal data comprises: vehicle-mounted basic data, including vehicle-mounted sensor real-time data and vehicle bus data; and / or text data, including vehicle maintenance text records; and / or image data, including images containing the failure point in the whole vehicle three-dimensional structure model.

[0013] Optionally, the correlatively training the failure cause and the corresponding multi-modal data by cross-modal contrast learning alignment data representation comprises: extracting time domain features from the vehicle bus data by a time sequence convolution network to capture signal mutation conditions; extracting maintenance information from the vehicle maintenance text records by a BERT model, the maintenance information comprising maintenance actions and maintenance components; and correlatively training the failure cause based on the signal mutation conditions, the maintenance information, and the images containing the failure point in the whole vehicle three-dimensional structure model.

[0014] In a second aspect, the present application provides a vehicle failure detection device, comprising:

[0015] A first acquisition unit configured to acquire multi-modal data of a target vehicle, the multi-modal data comprising data collected in different dimensions under a failure condition of the target vehicle;

[0016] A second acquisition unit configured to acquire a whole vehicle three-dimensional structure model of the target vehicle;

[0017] A processing unit configured to take the multi-modal data and the whole vehicle three-dimensional structure model as inputs of a trained failure detection model to obtain a failure detection result; the failure detection model is obtained by training a failure case library, and is used for correlatively analyzing a failure cause according to the multi-modal data and the whole vehicle three-dimensional structure model;

[0018] The fault detection result includes fault reasons of the fault points and position information of the fault points in the three-dimensional structure model of the whole vehicle.

[0019] In a third aspect, the present application provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the vehicle fault detection method in the first aspect.

[0020] In a fourth aspect, the present application provides an electronic device, which includes a storage medium, a processor, and a computer program stored in the storage medium and executable on the processor, and the processor executes the computer program to implement the vehicle fault detection method in the first aspect.

[0021] In a fifth aspect, the present application provides a vehicle, which includes the vehicle fault detection device in the second aspect or the electronic device in the fourth aspect.

[0022] According to the technical solutions, the vehicle fault detection method, device, equipment, storage medium and vehicle provided by the present application first acquire multi-modal data of a target vehicle, wherein the multi-modal data includes data collected in different dimensions under a fault condition of the target vehicle. Then, a three-dimensional structure model of the whole vehicle is acquired, and the multi-modal data and the three-dimensional structure model of the whole vehicle are taken as inputs of a trained fault detection model to obtain a fault detection result. The fault detection model is obtained by training a fault case library, and is used to analyze fault reasons according to the multi-modal data and the three-dimensional structure model of the whole vehicle. The fault detection result includes fault reasons of fault points and position information of the fault points in the three-dimensional structure model of the whole vehicle. Compared with related technologies, the model is pre-trained by a large commercial vehicle fault case library, so that the fault detection model used for analyzing vehicle faults is obtained. Meanwhile, the fault reasons are analyzed based on the multi-modal data of the fault case in different dimensions and the three-dimensional structure of the whole vehicle in the training, so that the detection efficiency is improved, and the accuracy is also improved compared with the manual judgment mode.

[0023] The above description is only a summary of the technical solutions of the present application. In order to enable a clearer understanding of the technical means of the present application, the specific embodiments of the present application can be implemented according to the content of the description, and in order to enable the above and other purposes, features and advantages of the present application to be more obvious and easy to understand, the following specific embodiments of the present application are described. BRIEF DESCRIPTION OF DRAWINGS

[0024] The accompanying drawings, which are incorporated into and form part of the specification, illustrate embodiments consistent with the present application and, together with the specification, serve to explain the principles of the present application.

[0025] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, for those skilled in the art, other drawings can also be obtained based on these drawings without any creative effort.

[0026] Figure 1 A flowchart of a vehicle fault detection method provided by an embodiment of the present application is shown;

[0027] Figure 2 A flowchart of a fault detection model training step provided by an embodiment of the present application is shown;

[0028] Figure 3 A system architecture diagram of a vehicle fault detection method provided by an embodiment of the present application is shown;

[0029] Figure 4 A structural diagram of a vehicle fault detection device provided by an embodiment of the present application is shown. DETAILED DESCRIPTION

[0030] In order to more clearly illustrate the above-mentioned purposes, features and advantages of the present application, the solutions of the present application will be further described below. It should be noted that the embodiments of the present application and the features in the embodiments can be combined with each other without conflict.

[0031] A vehicle fault detection method provided by the present embodiment is applied to a vehicle fault detection device or electronic equipment, which can be installed or integrated in some data processing system. These data processing systems can be a commercial vehicle fault diagnosis system for vehicle fault detection. When running, any of the vehicle fault detection methods mentioned below can be executed.

[0032] In the above background, in the process of troubleshooting the electrical problems of the current commercial vehicle, there are the following technical pain points: on the one hand, maintenance personnel use specific tools to obtain CAN bus data through OBD interface, which includes fault codes of each ECU. It should be noted that fault codes are standardized error identifiers generated by ECU (Electronic Control Unit) through preset self-diagnosis rules, which are used to record abnormal system states. However, not all fault codes represent hardware damage, and they may also be signal abnormalities, performance deviations or logic conflicts, such as sensor signal exceeding reasonable range, communication timeout or communication anomaly, etc. More obviously, it may be a "false fault" caused by transient fluctuation. Therefore, this situation may cause a heavy workload for maintenance personnel, resulting in low efficiency of manual detection, which is easily overlooked by conventional technical solutions. On the other hand, specific tools such as CANOE, ZLG, PCAN are expensive, and multiple tools need to be used according to different scenarios, which is costly.

[0033] To improve the above technical problems, the embodiment proposes a vehicle fault detection method, as shown in the figure, the method comprises: Figure 1

[0034] S101, acquiring multimodal data of a target vehicle;

[0035] Multimodal data refers to information collected by multiple different types of data sources or sensors that describe the same object or scene. For example, the multimodal data of the target vehicle includes vehicle-mounted basic data, text data and image data. Further, the vehicle-mounted basic data can include data parameters such as real-time monitoring data of vehicle-mounted sensors, environmental temperature, vibration level, and vehicle CAN bus data, wherein the CAN bus data contains fault codes. The text data includes text records of vehicle maintenance, and the image data includes images of fault points for more intuitive representation of the situation of the fault points, and images of the fault points in the three-dimensional structure model of the whole vehicle for subsequent representation of the specific position of the fault points in the three-dimensional structure of the vehicle. In the above multimodal data, the vehicle-mounted basic data, the text data and the image data are necessary types, but in the specific data selection, for example, which sensors' data can be selected for the vehicle-mounted basic data can be selected according to the actual application situation. These data modalities usually have different forms of expression and characteristics, but are associated or complementary with each other, and can provide more comprehensive and accurate information together.

[0036] S102, acquiring a three-dimensional structure model of the whole vehicle;

[0037] The three-dimensional structure model (3D Structural Model) refers to a three-dimensional geometric representation of the target vehicle constructed by digital technology, which can accurately describe the size of the appearance, the internal structure, the layout of the parts and components, the assembly relationship and the spatial geometric characteristics of the vehicle. It not only contains visual information of the appearance, but also covers internal mechanical structure, system components and connection relationship, which is the core basis for vehicle design, engineering analysis, simulation test and manufacturing production. Further, the three-dimensional structure model of the whole vehicle also includes the specific architecture system of the in-vehicle current voltage harness, and by acquiring the three-dimensional structure model of the whole vehicle, the specific position of the fault point in the three-dimensional structure is determined, which is also convenient for the maintenance personnel to understand the surrounding harness architecture (circuit system current voltage guide) while determining the position of the fault point, so as to further analyze and verify.

[0038] S103, taking the multimodal data and the three-dimensional structure model of the whole vehicle as inputs of a trained fault detection model to obtain a fault detection result.

[0039] ​The pre-trained fault detection model is a model constructed based on machine learning, deep learning or data-driven algorithm, and the core goal is to identify whether the target vehicle has a fault by analyzing the input data (multimodal data and the three-dimensional structure model of the whole vehicle), and further locate the fault cause or type. The model can automatically determine whether the current state of the vehicle is abnormal by learning the data patterns in historical fault cases and normal states, and output the detection result. The multimodal data fusion capability of the fault detection model can process multimodal data from different sensors or data sources and extract key features therefrom; the three-dimensional structure model of the whole vehicle is used to associate the fault phenomenon with the specific parts or system positions of the vehicle, and can also identify abnormal patterns in the data and classify them into specific fault types.

[0040] The fault detection result includes the fault cause of the fault point and the position information of the fault point in the three-dimensional structure model of the whole vehicle. The fault cause of the fault point is a conclusion drawn by the model from the analysis of the fault cause of the fault point in the current situation based on the training of the past fault case library. Specifically, the conclusion can include multiple types, and each fault cause model can analyze the corresponding probability. In addition, the model can also output a solution to the fault cause. The position information of the fault point in the three-dimensional structure model of the whole vehicle refers to the specific position where the fault occurs, and this position is accurately represented by the three-dimensional structure model of the whole vehicle. As a feasible embodiment, a coordinate system can be introduced to represent the specific position coordinates of the fault point.

[0041] The traditional troubleshooting of commercial vehicle electrical system mainly relies on the fault codes of OBD, the analysis of wire harness terminals according to the wire harness principle, and the detection of fault codes one by one, in addition, the analysis of the problem also depends on the past experience of the maintenance personnel. This way takes a long time to detect, is low in efficiency, and is not high in accuracy. In this embodiment, first, the multimodal data of the target vehicle is obtained, wherein the multimodal data includes data collected in different dimensions under the fault condition of the target vehicle. Then, the three-dimensional structure model of the whole vehicle is obtained, and the multimodal data and the three-dimensional structure model of the whole vehicle are taken as the input of the trained fault detection model to obtain the fault detection result. The fault detection model is trained by the fault case library, and is used to analyze the fault cause based on the correlation analysis of the multimodal data and the three-dimensional structure model of the whole vehicle. The fault detection result includes the fault cause of the fault point and the position information of the fault point in the three-dimensional structure model of the whole vehicle. Compared with related technologies, the model is pre-trained based on a large commercial vehicle fault case library, thereby obtaining a fault detection model for analyzing vehicle faults, and in the training, the fault cause is analyzed based on the multimodal data in different dimensions and the overall analysis of the three-dimensional structure of the whole vehicle, thereby improving the detection efficiency and also improving the accuracy compared with the manual judgment method.

[0042] Optionally, the fault detection result further comprises a probability corresponding to the fault cause; after obtaining the fault detection result and the probability corresponding to the fault cause, the method further comprises: comparing a maximum value of the probability corresponding to the fault cause with a preset threshold, and in a case where the maximum value of the probability corresponding to the fault cause is less than the preset threshold, marking the current fault detection result and triggering an artificial review process.

[0043] In this embodiment, the fault detection result output by the model not only contains the fault cause, but also outputs a corresponding probability value for each possible cause. The system selects the maximum value from the probabilities of all fault causes and compares it with a preset threshold. For example, the threshold is set to 70%, if the maximum probability is less than the threshold: the system will mark the current fault detection result and automatically trigger the artificial review process, and the technician will intervene for further analysis. When further reviewed by the technician, the specific location of the fault point in the three-dimensional structure of the vehicle can be utilized, thereby facilitating the verification and judgment of the cause according to the surrounding current and voltage orientation. If the maximum probability is greater than or equal to the threshold, the system directly outputs the result.

[0044] Optionally, the method further comprises: in a case where the maximum value of the probability corresponding to the fault cause is less than the preset threshold, updating the current multi-modal data and the three-dimensional structure model of the vehicle to the fault detection model.

[0045] In this embodiment, the model also has an online incremental learning mechanism. When a new fault mode is detected, real-time vehicle data is collected, and the fault information and vehicle data record model are trained and recorded to increase the case library.

[0046] If the maximum probability value is less than the preset threshold, it means that the model is not determined enough for the current fault. The current multi-modal data (such as vehicle-mounted sensor data, vehicle maintenance text records, fault component image features, etc.) and the three-dimensional structure model (such as fault location annotation) are added to the training data set of the model as new samples. These new samples may contain fault modes that the model has not learned before, or supplementary information for existing modes. The updated data set can be used to retrain the model (such as incremental learning) or fine-tune the model parameters to improve the detection ability of similar faults. The model will gradually cover more fault scenarios and improve the generalization ability.

[0047] Optionally, the training step of the fault detection model comprises: loading a fault case library, the fault case library comprising fault causes of fault cases, a whole vehicle three-dimensional structure model of a fault vehicle, position information of a fault point in the whole vehicle three-dimensional structure model, and multi-modal data of the fault vehicle; aligning data representations through cross-modal contrastive learning to associate and train the fault causes with corresponding multi-modal data; and obtaining the fault detection model based on the associated training of the fault causes, the multi-modal data, and the position information of the fault point in the whole vehicle three-dimensional structure model.

[0048] In the embodiment, the process of model training is described. First, the initial model is loaded into a fault case library, and the content of the fault case library comprises: a fault cause, i.e., a clearly labeled fault type; a whole vehicle three-dimensional structure model, i.e., a digital three-dimensional model of a fault vehicle, containing the geometric shapes, assembly relationships, and spatial positions of all vehicle parts, in addition to the position structure, including circuit system current voltage guidance, whole vehicle three-dimensional wire harness high voltage and low voltage architecture; fault point position information, i.e., the specific position of the fault occurrence marked in the three-dimensional model; and multi-modal data, i.e., multi-dimensional information such as integrated vehicle CAN bus data (sampling frequency 100 Hz), vehicle-mounted sensor real-time data, vehicle maintenance work order text records, fault component image features, etc.

[0049] Further, the initial model is aligned through cross-modal contrastive learning to align data representations, and the fault causes are associated and trained with corresponding multi-modal data. Cross-modal contrastive learning is a multi-modal learning technique, and the purpose is to enable the model to understand the relevance between different modal data, and the core idea is to map different modal data (such as sensor data and corresponding images) of the same fault case into the same feature space, so that their distances in the feature space are as close as possible. At the same time, the data of different fault cases are pulled apart in the feature space to enhance the distinguishing ability of the model. Aligned data representation refers to mapping different modal data (such as images, texts, sensor data, etc.) or data from different sources into a unified feature space, so that they have comparability and relevance in semantics or structure. The core purpose of alignment is to enable the model to understand the internal relationship between different data, thereby improving the performance of cross-modal tasks (such as fault detection, image-text matching, multi-modal classification, etc.).

[0050] The purpose of the correlation training is to combine the fault cause, multi-modal data and the location information of the fault point in the three-dimensional model, and train the model to learn the mapping relationship among them. The three-dimensional structure model provides the spatial location information of the fault point, helping the model to understand the specific physical location where the fault occurs. The multi-modal data can provide multi-dimensional representation of the fault, helping the model to understand the fault from different angles. Specifically, the input multi-modal data (such as sensor data, image) and the three-dimensional structure model (containing fault point location label) are compared with the difference between the predicted result and the real label (fault cause and location) to calculate the loss function, and the model parameters are optimized to make the predicted result as close to the real label as possible, and the predicted fault cause and fault point location are output.

[0051] As a feasible embodiment: discretize the whole vehicle beam CAD model into a voxel grid; adopt a 3D convolutional neural network to predict the voxel coordinates of the fault point; and convert the voxel coordinates into real three-dimensional space coordinates through a back projection algorithm. Further, the correlation training is performed with the fault cause.

[0052] Optionally, the multi-modal data includes: vehicle-mounted basic data, including vehicle-mounted sensor real-time data and vehicle bus data; and / or text data, including vehicle maintenance text records; and / or image data, including images containing fault points in the whole vehicle three-dimensional structure model.

[0053] In the embodiment, the vehicle-mounted basic data comes from the real-time or historical data of the vehicle's own system, reflecting the running state of the vehicle. The vehicle-mounted sensor real-time data includes the numerical data collected by temperature sensors (such as water tank temperature), pressure sensors (such as tire air pressure), vibration sensors (such as engine vibration), etc.; the vehicle bus data includes the data transmitted through the vehicle internal communication bus (such as CAN bus), containing the interaction information between ECUs (Electronic Control Units), and can also obtain fault codes. The vehicle-mounted basic data provides quantitative indicators of the vehicle's running state, and is the basic data source for fault detection.

[0054] The text data is vehicle-related information recorded in text form, usually including historical maintenance records, diagnosis reports or user feedback. The vehicle maintenance text records include maintenance logs, fault descriptions, maintenance measures and other unstructured text data, providing the historical background and maintenance experience of the fault, helping the model to understand the fault mode and maintenance scheme.

[0055] The image data is visual data related to vehicle faults, usually containing visual information of fault points. The images containing fault points in the whole vehicle three-dimensional structure model: images taken by cameras, endoscopes and other devices, or fault point visualization data extracted from three-dimensional models. Image data provides an intuitive representation of the fault, helping the model to detect and locate faults in combination with visual features.

[0056] The vehicle-mounted sensor data in the multi-modal data includes fault codes, which can be used to monitor the faults of each ECU in real time (such as temperature anomaly detection); the text data is used to mine historical maintenance records; the image data is used to further understand the image information of the fault point (such as detecting brake disc wear through image recognition) and locate the fault point from the three-dimensional structural model. Combining sensor data and image data can improve the accuracy of fault detection, and combining text data and three-dimensional models can assist in fault location and maintenance guidance.

[0057] Optionally, the fault cause is associated with the corresponding multi-modal data through cross-modal contrast learning to align the data representation, including: extracting time domain features of vehicle bus data through a time convolution network to capture signal mutation conditions; extracting maintenance information from vehicle maintenance text records through a BERT model, the maintenance information including maintenance actions and maintenance components; and associating the fault cause based on the signal mutation conditions, the maintenance information, and the image containing the fault point in the three-dimensional structural model of the vehicle.

[0058] In this embodiment, cross-modal contrast learning is a technique that maps data of different modalities (such as text, image, sensor data) to a unified feature space, so that different modal data of the same fault are close in the feature space, and data of different faults are far apart. The purpose of alignment is to enable the model to understand the association between different modal data (such as the association between sensor data mutation and fault action described in the maintenance text). The time domain feature extraction of vehicle bus data can use TCN (Temporal Convolutional Network) to process vehicle bus data. The TCN is a convolutional neural network specially designed for processing time series data, which captures local and long-term dependencies in time series through one-dimensional convolution operations. The time domain feature extraction of vehicle bus data can be used to capture the time sequence changes and mutation conditions of the signal (such as sudden temperature rise, abnormal pressure fluctuation, etc.). The maintenance information extraction of vehicle maintenance text records is the natural language processing (NLP) of vehicle maintenance text records using a BERT model. The BERT is a language model that learns general language representations (word embeddings) through large-scale unsupervised text data, and then adapts to specific downstream tasks (such as text classification, named entity recognition, etc.) through fine-tuning. Using the BERT model to extract key maintenance information from the text includes: maintenance actions (such as “replace a certain component” “clean the radiator”) and maintenance components (such as “brake disc”). The visual feature extraction of the fault point image uses a convolutional neural network or other image processing model to extract features from the image containing the fault point. The visual features in the image (such as the expansion area, crack shape, etc.) are extracted.

[0059] Specifically, the steps of the association training of the multi-modal features and the fault cause are as follows:

[0060] 1. Feature alignment: Map the time-domain features extracted by TCN, the maintenance information features extracted by BERT, and the image features extracted by CNN to the same feature space (through cross-modal contrastive learning). 2. Fault cause association: Train the aligned multi-modal features with the fault causes. Specifically, supervised learning can be used, taking the multi-modal features as input and the fault causes as labels, and predicting the fault causes through a classifier; or contrastive learning can be used to make the multi-modal features of the same fault closer in the feature space. 3. By optimizing the loss function, adjust the model parameters to make the association between the multi-modal features and the fault causes more accurate.

[0061] Further, the model's judgment on false faults has improved the accuracy of artificial judgment. Specifically, when the vehicle bus data is extracted by TCN to extract time-domain features (such as mean, variance, maximum, minimum, etc.), there are abnormal components. For example, using the automobile brake system, using TCN to process the brake system bus data to obtain the mean, variance, maximum, minimum, etc. of the brake pressure sensor, when the brake pressure mean is normal but the change rate is abnormally large, it is likely that the signal is caused by transient fluctuations, such as electromagnetic interference, sensor transient failure, etc. This point is also recorded in the fault case library as a false fault that exists in historical cases, so that when the same data situation is encountered again, the probability of false fault judgment will be improved. On the other hand, through association training, multiple related features are associated and combined, which can also further analyze the fault cause. For example, the engine speed and intake pressure are combined to analyze their correlation, and if the correlation changes abnormally, it may indicate that the system has a logical conflict or performance deviation. Such features are associated with false causes in the fault cause, and the model learns a large amount of data to establish a mapping between features and fault types.

[0062] Further, the vehicle fault detection method applied in the embodiment is introduced, as shown in Figure 2 The flowchart of another vehicle fault detection method proposed in the embodiment is shown.

[0063] S201, load the fault case library;

[0064] The pre-collected and sorted fault case library is loaded into the model. The fault case library usually contains detailed information of various vehicle faults, such as fault phenomenon description, vehicle state data when the fault occurs, fault cause, maintenance record, etc. These cases are important basis for the model to learn fault patterns and rules.

[0065] S202, extract time-domain features from vehicle bus data through time convolution network to capture signal mutation;

[0066] The vehicle bus is a channel for data communication between various electronic control units inside the vehicle. The bus data contains the working status and operating parameters of various components of the vehicle, such as sensor data, actuator status, etc. The time convolution network has been introduced above and will not be repeated here. The vehicle bus data is input into the TCN, which performs convolution operation on the data to extract features in the time dimension, such as signal fluctuation trend, periodic change, etc. When the vehicle fails, some signals in the bus data may suddenly change, such as sudden increase, decrease or abnormal fluctuation. TCN can sensitively capture these signal mutation characteristics, which are of great significance for fault diagnosis.

[0067] S203, extracting repair information from the vehicle repair text record by the BERT model;

[0068] The vehicle repair text record contains information such as the repair process of the faulty vehicle, the replaced parts, and the repair measures taken. These text information exists in the form of natural language, which is difficult to be directly processed and analyzed by computer. BERT is a pre-trained language model, which has been introduced above and will not be repeated here. The vehicle repair text record is input into the BERT model, which encodes and understands the text and extracts key information such as fault type, repair method, and replaced part name. These extracted repair information can be fused with other data such as vehicle bus data to provide a more comprehensive basis for fault diagnosis.

[0069] S204, associating and training the fault reason based on the signal mutation, repair information, and image containing the fault point in the three-dimensional structure model of the whole vehicle;

[0070] The signal mutation extracted by TCN reflects the abnormal change of vehicle bus data when the fault occurs. The repair information is extracted from the repair text record by the BERT model. The three-dimensional structure model of the whole vehicle can intuitively display the various components and structures of the vehicle. By marking the location of the fault point in the model and generating the corresponding image, spatial information can be provided for fault analysis. The signal mutation, repair information, and fault point image are used as input features, and the corresponding fault reason is used as a label to construct a training data set. Then use machine learning or deep learning algorithm (such as neural network) to train the data set, so that the model can learn the mapping relationship between the input features and the fault reason. By continuously adjusting the parameters of the model, the prediction accuracy of the model for the fault reason is improved.

[0071] S205, associating and training based on the fault reason, multi-modal data, and location information of the fault point in the three-dimensional structure model of the whole vehicle to obtain a fault detection model.

[0072] Multi-modal data: including vehicle bus data (time domain features after TCN processing), maintenance information (key information extracted by BERT model), and other possible related data such as vehicle operating environment data, etc. These multi-modal data reflect the vehicle's state and fault condition from different angles. The location information of the fault point in the three-dimensional structure model of the whole vehicle provides the spatial position of the fault occurrence, which helps to more accurately locate the fault and perform maintenance. Integrating fault causes, multi-modal data and fault point location information together, a more comprehensive training data set is constructed. Using more complex model architecture (such as deep neural network, ensemble learning model, etc.) to train the data set, further optimize the parameters of the model, so that the model can comprehensively consider various data information, accurately detect and diagnose vehicle faults. After multiple iterations of training and verification, a well-performing fault detection model is finally obtained.

[0073] In the above embodiment, loading the fault case library provides a rich data basis for model training and enhances the model's generalization ability. The time series convolution network processes vehicle bus data, which can accurately capture the key fault feature of signal mutation and effectively extract time domain information, providing a strong basis for fault diagnosis. The BERT model efficiently extracts maintenance information from maintenance text records, converting text knowledge into structured data and enriching the fault analysis dimension. Based on the correlation training of signal mutation, maintenance information, fault point image and fault cause, and the subsequent fusion of multi-modal data and fault point location information, the complementarity of different types of data is fully utilized, enabling the model to comprehensively understand the fault from multiple angles. Through this multi-stage, multi-modal correlation training method, an accurate and robust fault detection model can be constructed, achieving precise detection and diagnosis of vehicle faults, providing reliable support for vehicle maintenance and repair, reducing maintenance costs, saving time and manpower, and building a model platform that greatly facilitates the repair system.

[0074] Further, the system architecture diagram of the vehicle fault detection method applied in the embodiment is introduced, as shown in Figure 3 The system architecture diagram of another vehicle fault detection method proposed in the embodiment is shown.

[0075] 1. Basic physical layer structure

[0076] Vehicle data connection tool: one end connects the vehicle OBD with traditional 16-pin terminal, and the other end connects the computer with USB interface. The computer is installed with the completed training model host computer, and the connection tool is built-in with vehicle network data acquisition module, non-intrusive sensor array, deployment of current and voltage ripple detection module (precision ±0.5), insulation impedance monitoring unit, etc.

[0077] 2. Model intelligent perception

[0078] Through the connection tool, multi-protocol adaptation: support commercial vehicle communication protocols such as J1939, ISO11898, ISO14229, etc., input data for each physical layer module, combine intelligent sensing algorithm, and exclude basic line short circuit and open circuit faults in priority.

[0079] 3. AI diagnosis engine

[0080] (1) Fault feature: use machine neural network to process data signal, cooperate with central analysis algorithm, and perform deep analysis and penetration processing on collected data;

[0081] (2) Fault reasoning model: build a neural network structure with causal reasoning ability, each controller node corresponds to an ECU control unit, through collected data and input fault principles, analyze and locate fault points and accuracy;

[0082] (3) Three-dimensional fault location algorithm: based on the three-dimensional AI model of the whole vehicle, real-time simulation of input information, combined with positioning confirmation algorithm to analyze the source of electrical problems;

[0083] (4) Credibility evaluation module: generate credibility score when the model outputs the diagnosis result, start expert consultation process when the score is below the threshold, and manually input data analysis.

[0084] 4. Knowledge and experience management system

[0085] (1) Fault case library: fault definition interface input, storage structured diagnosis record (including bus message information, voltage and current, whole vehicle wire harness connection, sensor detection data, environment temperature, vibration level data parameters), through fault case library and correlation training based on multi-modal data, so that the model has analysis ability based on the fault case library.

[0086] (2) Self-evolution knowledge graph: node fault case library is dynamically updated with diagnosis results, and supports custom diagnosis database case increase.

[0087] 5. Maintenance scheme generator

[0088] Based on learning algorithm, fault case library, maintenance path planning, etc., determine the source of electrical problems, automatically match the correct maintenance scheme, and reduce the average maintenance time.

[0089] Further, as a specific implementation of the method shown in Figures 1 to 3 , the embodiment provides a vehicle fault detection device, as shown in Figure 4 , the device comprises a first acquisition unit 401, a second acquisition unit 402 and a processing unit 401.

[0090] The first acquisition unit 401 is configured to acquire multi-modal data of a target vehicle, wherein the multi-modal data comprises data collected in different dimensions under a fault condition of the target vehicle.

[0091] The second acquisition unit 402 is configured to acquire a three-dimensional structure model of the target vehicle.

[0092] The processing unit 403 is configured to take the multi-modal data and the three-dimensional structure model of the vehicle as inputs of a trained fault detection model to obtain a fault detection result, wherein the fault detection model is obtained by training a fault case library and is used for correlating and analyzing fault causes according to the multi-modal data and the three-dimensional structure model of the vehicle.

[0093] The fault detection result comprises a fault cause of a fault point and position information of the fault point in the three-dimensional structure model of the vehicle.

[0094] In a specific application scenario, the processing unit 403 is specifically configured to compare a maximum value of a probability corresponding to the fault cause with a preset threshold value, and in a case where the maximum value of the probability corresponding to the fault cause is less than the preset threshold value, mark the current fault detection result and trigger an artificial review process.

[0095] In a specific application scenario, the processing unit 403 is specifically further configured to, in a case where the maximum value of the probability corresponding to the fault cause is less than the preset threshold value, update the current multi-modal data and the three-dimensional structure model of the vehicle to the fault detection model.

[0096] In a specific application scenario, the processing unit 403 is specifically further configured to load a fault case library, wherein the fault case library comprises a fault cause of a fault case, a three-dimensional structure model of a fault vehicle, position information of a fault point in the three-dimensional structure model of the vehicle, and multi-modal data of the fault vehicle; correlate and train the fault cause and the corresponding multi-modal data by cross-modal comparison learning to align data representation; and obtain the fault detection model based on the correlation and training of the fault cause, the multi-modal data, and the position information of the fault point in the three-dimensional structure model of the vehicle.

[0097] In a specific application scenario, the first acquisition unit 401 is specifically further configured to acquire vehicle-mounted basic data, including real-time data of a vehicle-mounted sensor, vehicle bus data; and / or text data, including vehicle maintenance text records; and / or image data, including images containing the fault point in the three-dimensional structure model of the vehicle.

[0098] In a specific application scenario, the processing unit 403 is specifically further configured to extract time domain features from the vehicle bus data through a time sequence convolution network to capture signal mutation conditions; extract maintenance information from the vehicle maintenance text record through a BERT model, the maintenance information including maintenance actions and maintenance components; and perform association training on the fault cause based on the signal mutation conditions, the maintenance information, and the image containing the fault point in the whole vehicle three-dimensional structure model.

[0099] It should be noted that other corresponding descriptions of the functions of the vehicle fault detection device provided in the embodiment can be referred to the corresponding descriptions in the Figures 1 to 3 , which will not be described here.

[0100] Based on the method as shown in Figures 1 to 3 , correspondingly, the embodiment further provides a computer readable storage medium having a computer program stored thereon, and the computer program is executed by a processor to implement the method as shown in Figures 1 to 3 .

[0101] Based on the method as shown in Figures 1 to 3 , correspondingly, the embodiment further provides a computer program product having a computer program stored thereon, and the computer program is executed by a processor to implement the method as shown in Figures 1 to 3 .

[0102] Based on such understanding, the technical solution of the present application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a U disk, a mobile hard disk, etc.), and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods of various implementation scenarios of the present application.

[0103] Based on the method as shown in Figures 1 to 3 , and the virtual device embodiment as shown in Figure 4 , in order to achieve the above purpose, the embodiment of the present application further provides an electronic device which can be configured on a computer side, etc., the device includes a storage medium and a processor; the storage medium is used to store a computer program; the processor is used to execute the computer program to implement the method as shown in Figures 1 to 3 .

[0104] Based on the method as shown in Figures 1 to 3 , and the virtual device embodiment as shown in Figure 4To achieve the above object, the virtual device embodiment shown also provides a chip comprising one or more interface circuits and one or more processors; the interface circuit is used to receive a signal from the memory of an electronic device and send the signal to the processor, the signal comprising computer instructions stored in the memory; when the processor executes the computer instructions, the electronic device performs the above-mentioned Figures 1 to 3 The method shown.

[0105] Optionally, the above-mentioned entity device can also include a user interface, a network interface, a camera, a radio frequency (RF) circuit, a sensor, an audio circuit, a WI-FI module, etc. The user interface can include a display screen, an input unit such as a keyboard, etc. The optional user interface can also include a USB interface, a card reader interface, etc. The network interface can optionally include a standard wired interface, a wireless interface (such as a WI-FI interface), etc.

[0106] Those skilled in the art can understand that the above-mentioned entity device structure provided by the embodiment does not constitute a limitation on the entity device, and can include more or fewer components, or combine certain components, or different component arrangements.

[0107] The storage medium can also include an operating system and a network communication module. The operating system is a program that manages the hardware and software resources of the above-mentioned entity device and supports the running of information processing programs and other software and / or programs. The network communication module is used to realize communication between the components inside the storage medium and communication with other hardware and software in the information processing entity device.

[0108] Through the above description of the embodiments, those skilled in the art can clearly understand that the present application can be implemented by means of software and necessary general hardware platforms, or by hardware. By applying the scheme of the embodiment, compared with the related art, the model is pre-trained through a large commercial vehicle fault case library, so as to obtain a fault detection model for analyzing vehicle faults, and at the same time, the fault causes are analyzed based on the multi-modal data of the fault case in different dimensions and the overall analysis of the three-dimensional structure of the whole vehicle in the training, thereby improving the detection efficiency. Compared with the manual judgment method, the accuracy is also improved.

[0109] It has to be noted that, in the present document, relational terms are intended only to convey a possible relationship between elements or

[0110] The above description is merely that of a specific implementation to enable a person skilled in the art to understand or implement the application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein can be applied to other embodiments without departing from the spirit or scope of the application. Thus, the present application is not intended to be limited to the embodiments described herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A vehicle malfunction detection method characterized by, The method comprises the following steps: acquiring multi-modal data of a target vehicle, the multi-modal data comprising data collected in different dimensions under a fault condition of the target vehicle; acquiring a three-dimensional structure model of the target vehicle; inputting the multi-modal data and the three-dimensional structure model into a trained fault detection model to obtain a fault detection result, the fault detection model being trained by a fault case library and used for correlating and analyzing fault causes according to the multi-modal data and the three-dimensional structure model; the fault detection result comprising a fault cause of a fault point and position information of the fault point in the three-dimensional structure model.

2. The method of claim 1, wherein, the fault detection result further comprising a probability corresponding to the fault cause; after obtaining the fault detection result and the probability corresponding to the fault cause, the method further comprises: comparing a maximum value of the probability corresponding to the fault cause with a preset threshold, and in the case that the maximum value of the probability corresponding to the fault cause is less than the preset threshold, marking the current fault detection result and triggering an artificial review process.

3. The method of claim 2, wherein, the method further comprises: in the case that the maximum value of the probability corresponding to the fault cause is less than the preset threshold, updating the current multi-modal data and the three-dimensional structure model into the fault detection model.

4. The method of claim 1, wherein, the training steps of the fault detection model comprise: loading a fault case library, the fault case library comprising a fault cause of a fault case, a three-dimensional structure model of a fault vehicle, position information of a fault point in the three-dimensional structure model, and multi-modal data of the fault vehicle; correlating and training the fault cause with corresponding multi-modal data by cross-modal contrastive learning of data representation; based on the fault cause, the multi-modal data, and the position information of the fault point in the three-dimensional structure model, correlating and training to obtain the fault detection model.

5. The method of claim 4, wherein, the multi-modal data comprises: vehicle-mounted basic data, including real-time vehicle-mounted sensor data and vehicle bus data; and / or, text data, including vehicle maintenance text records; and / or, image data, including images containing the fault point in the three-dimensional structure model.

6. The method of claim 5, wherein, correlating and training the fault cause with corresponding multi-modal data by cross-modal contrastive learning of data representation comprises: extracting time domain features from the vehicle bus data by a time convolution network to capture signal mutation conditions; extracting maintenance information from the vehicle maintenance text records by a BERT model, the maintenance information comprising maintenance actions and maintenance components; based on the signal mutation conditions, the maintenance information, and the images containing the fault point in the three-dimensional structure model, correlating and training with the fault cause.

7. A vehicle malfunction detection apparatus characterized by comprising: The method comprises the following steps: a first acquisition unit configured to acquire multi-modal data of a target vehicle, the multi-modal data comprising data collected in different dimensions under a fault condition of the target vehicle; a second acquisition unit configured to acquire a three-dimensional structure model of the target vehicle; The processing unit is configured to take the multi-modal data and the three-dimensional structure model of the whole vehicle as inputs of a trained fault detection model to obtain a fault detection result; the fault detection model is obtained by training a fault case library and is used for correlatively analyzing fault causes according to the multi-modal data and the three-dimensional structure model of the whole vehicle. The fault detection result includes fault causes of fault points and position information of the fault points in the three-dimensional structure model of the whole vehicle.

8. An electronic device comprising a storage medium, a processor, and a computer program stored on the storage medium and executable on the processor, characterized in that, The processor executes the computer program to implement the method in any one of claims 1 to 7.

9. A computer readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the method in any one of claims 1 to 7.

10. A vehicle characterized by comprising: The vehicle fault detection device according to claim 7 or the electronic device according to claim 8.

Images