A vehicle fault diagnosis model and its training and use method, device and medium

Abstract

The present application discloses a vehicle fault diagnosis model and its training and use methods, devices and media. By acquiring fault signals, fault description texts and actual fault types to obtain the time characteristics and spatial characteristics of the faults and the correlation between different types of faults, various types of fault diagnosis can be achieved, and the fault signals are converted into fault images to reduce the noise signals in the fault signals. The fault images and fault description texts are fused based on the time characteristics and spatial characteristics to obtain the image features of the fault images and the text features of the fault description texts. The predicted fault types are obtained based on the image features and the text features. The diagnosis accuracy is improved by fusing multimodal features, and the image features and text features are matched from the two dimensions of time and space to further improve the diagnosis accuracy and precision of the model.

Description

Technical Field

[0001] The present application relates to the technical field of vehicle fault diagnosis, and in particular to a vehicle fault diagnosis model and a training and use method, device and medium thereof. Background Art

[0002] As people's living standards continue to improve, more and more families buy cars as a means of transportation. However, various faults may occur in the process of using the car, especially the faults that occur during the driving process of the car need to be discovered and handled in time to ensure driving safety.

[0003] With the continuous development of new energy vehicle technology, more and more cars can be connected to the cloud platform. When a vehicle fails, the fault feedback mechanism will upload the fault signal and fault data to the cloud platform for operation and maintenance personnel to view and analyze the fault. At present, the cloud platform also uses machine learning to analyze fault information and fault data to identify the fault type and solution. However, this machine learning method can only diagnose a single type of fault, and the recognition accuracy of other types of faults is low. For different types of faults, the model needs to be rebuilt and trained, which will result in greater time and economic costs.

[0004] Therefore, there is an urgent need to build a model that can efficiently and accurately identify and diagnose various types of faults. Summary of the invention

[0005] In order to solve the above technical problems, the present application is proposed. The embodiments of the present application provide a vehicle fault diagnosis model and a training and use method, device and medium thereof.

[0006] According to one aspect of the present application, a training method for a vehicle fault diagnosis model is provided, comprising: acquiring fault data; wherein the fault data comprises a fault signal, a fault description text and a real fault type, and the fault signal and the fault description text both comprise time features and space features; obtaining a fault picture according to the fault signal; wherein the fault picture is obtained by transforming the fault signal into the time-frequency domain; obtaining a predicted fault type corresponding to the fault picture and the fault description text according to the fault picture, the fault description text and an initial model; wherein the initial model comprises a knowledge fusion module and a classification module, the knowledge fusion module being used to fuse the fault picture and the fault description text based on the time features and the space features to obtain the image features of the fault picture and the text features of the fault description text, and the classification module being used to obtain the predicted fault type based on the image features and the text features; adjusting the model parameters of the initial model to reduce the loss between the real fault type and the predicted fault type, and obtaining the vehicle fault diagnosis model.

[0007] In one embodiment, the model parameters of the initial model are adjusted to reduce the loss between the real fault type and the predicted fault type to obtain the vehicle fault diagnosis model, including: calculating the spatial loss function of the initial model based on the fault picture and the fault description text; calculating the time loss function of the initial model based on the fault picture and the fault description text; calculating the loss function of the initial model based on the spatial loss function and the time loss function; based on the loss function of the initial model, adjusting the model parameters of the initial model to achieve that the loss between the real fault type and the predicted fault type is less than a preset loss threshold, to obtain the vehicle fault diagnosis model.

[0008] In one embodiment, the spatial loss function of the initial model is calculated based on the fault image and the fault description text, including: the knowledge fusion module includes multiple forward feedback layers, and the multiple forward feedback layers are used to perform feature recognition on the fault image and the fault description text respectively to obtain multiple feature pairs; wherein each feature pair includes image features and text features corresponding to the same fault; based on the multiple feature pairs, the spatial loss function is calculated.

[0009] In one embodiment, the use of multiple forward feedback layers to perform feature recognition on the fault image and the fault description text respectively to obtain multiple feature pairs includes: calculating the probability of each forward feedback layer recognizing the fault image and the fault description text; selecting a predetermined number of the forward feedback layers from large to small according to the probability to perform feature recognition on the fault image and the fault description text to obtain multiple feature pairs.

[0010] In one embodiment, the spatial loss function is calculated based on the multiple feature pairs, including: calculating the global loss function and the local loss function of the fault image and the fault description text in each feature pair; and calculating the spatial loss function based on the global loss function and the local loss function corresponding to all the feature pairs.

[0011] In one embodiment, the time loss function of the initial model is calculated based on the fault picture and the fault description text, including: calculating the deviation between the fault picture and the fault description text in the time dimension; calculating the time loss function based on the deviation between the fault picture and the fault description text in the time dimension.

[0012] In one embodiment, the fusing of the fault picture and the fault description text based on the time feature and the spatial feature to obtain the image feature of the fault picture and the text feature of the fault description text includes: calculating the time deviation between the fault picture and the fault description text, and the spatial position deviation between the fault picture and the fault description text; fusing the fault picture and the fault description text whose time deviation is less than a preset time threshold and whose spatial position deviation is less than a preset distance threshold to obtain the image feature and the text feature.

[0013] According to another aspect of the present application, a vehicle fault diagnosis model is provided, including: a knowledge fusion module and a classification module, the knowledge fusion module is used to fuse the fault picture and the fault description text based on the time feature and the space feature to obtain the image feature of the fault picture and the text feature of the fault description text, and the classification module is used to obtain the vehicle fault type based on the image feature and the text feature; wherein the vehicle fault diagnosis model is obtained by using any of the above-mentioned vehicle fault diagnosis model training methods.

[0014] In one embodiment, the knowledge fusion module includes multiple forward feedback layers, and the multiple forward feedback layers respectively perform feature recognition on the fault image and the fault description text to obtain multiple feature pairs; wherein each feature pair includes image features and text features corresponding to the same fault.

[0015] According to another aspect of the present application, a method for using a vehicle fault diagnosis model is provided, comprising: obtaining fault data of a target vehicle; wherein the fault data of the target vehicle comprises a fault signal and a fault description text of the target vehicle; inputting the fault signal and the fault description text of the target vehicle into a vehicle fault diagnosis model to obtain a fault type of the target vehicle; wherein the vehicle fault diagnosis model is obtained using any of the vehicle fault diagnosis model training methods described above.

[0016] According to another aspect of the present application, a training device for a vehicle fault diagnosis model is provided, comprising: a fault data acquisition module, for acquiring fault data; wherein the fault data comprises a fault signal, a fault description text and a real fault type, and the fault signal and the fault description text both comprise a time feature and a space feature; a fault picture acquisition module, for acquiring a fault picture according to the fault signal; wherein the fault picture is acquired by transforming the fault signal into a time-frequency domain; a prediction type identification module, for acquiring a predicted fault type corresponding to the fault picture and the fault description text according to the fault picture, the fault description text and an initial model, wherein the initial model comprises a knowledge fusion module and a classification module, the knowledge fusion module is used to fuse the fault picture and the fault description text based on the time feature and the space feature to obtain the picture feature of the fault picture and the text feature of the fault description text, and the classification module is used to obtain the predicted fault type based on the picture feature and the text feature; a model parameter adjustment module, for adjusting the model parameters of the initial model to reduce the loss between the real fault type and the predicted fault type, and to obtain the vehicle fault diagnosis model.

[0017] According to another aspect of the present application, a computer-readable storage medium is provided, characterized in that the storage medium stores a computer program, and the computer program is used to execute any of the above-mentioned vehicle fault diagnosis model training methods and / or the above-mentioned vehicle fault diagnosis model use methods.

[0018] The present application provides a vehicle fault diagnosis model and its training and use methods, devices and media, which obtain fault data; wherein the fault data includes a fault signal, a fault description text and a real fault type, and the fault signal and the fault description text both include time features and space features; according to the fault signal, a fault picture is obtained; wherein the fault picture is obtained by transforming the fault signal into time and frequency domains; according to the fault picture, the fault description text and the initial model, a predicted fault type corresponding to the fault picture and the fault description text is obtained; wherein the initial model includes a knowledge fusion module and a classification module, the knowledge fusion module is used to fuse the fault picture and the fault description text based on time features and space features to obtain the image features of the fault picture and the text features of the fault description text, and the classification module is used to obtain the predicted fault type based on the image features and the text features Fault type; adjust the model parameters of the initial model to reduce the loss between the real fault type and the predicted fault type, and obtain a vehicle fault diagnosis model; that is, by obtaining the fault signal, the fault description text and the real fault type to know the time characteristics and spatial characteristics of the fault and the correlation between different types of faults, various types of fault diagnosis can be achieved, and the fault signal is converted into a fault picture to reduce the noise signal in the fault signal, the fault picture and the fault description text are fused based on the time characteristics and the spatial characteristics to obtain the image features of the fault picture and the text features of the fault description text, and the predicted fault type is obtained based on the image features and the text features, the diagnosis accuracy is improved by fusing multimodal features, and the image features and the text features are matched from the two dimensions of time and space to further improve the diagnosis accuracy and precision of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] By describing the embodiments of the present application in more detail in conjunction with the accompanying drawings, the above and other purposes, features and advantages of the present application will become more apparent. The accompanying drawings are used to provide a further understanding of the embodiments of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the present application and do not constitute a limitation of the present application. In the accompanying drawings, the same reference numerals generally represent the same components or steps.

[0020] Figure 1 It is a flowchart of a method for training a vehicle fault diagnosis model provided by an exemplary embodiment of the present application.

[0021] Figure 2 It is a structural diagram of a vehicle fault diagnosis model provided by an exemplary embodiment of the present application.

[0022] Figure 3 It is a structural schematic diagram of a vehicle fault diagnosis model training device provided by an exemplary embodiment of the present application.

[0023] Figure 4 It is a structural diagram of an electronic device provided by an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0024] Below, the exemplary embodiments according to the present application will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application, and it should be understood that the present application is not limited to the exemplary embodiments described here.

[0025] Figure 1 FIG. 1 is a flow chart of a method for training a vehicle fault diagnosis model provided by an exemplary embodiment of the present application. Figure 1 As shown, the training method of the vehicle fault diagnosis model includes the following steps:

[0026] Step 110: Obtain fault data.

[0027] Among them, fault data includes fault signals, fault description text and actual fault type. Both fault signals and fault description text include time characteristics and spatial characteristics. This application can extract information from the vehicle's fault manual, production record report and related literature to reflect the time characteristics and spatial characteristics of the fault, as well as the correlation between different types of faults, thereby improving the generalization ability and efficiency of fault diagnosis. Optionally, the fault data in this application can also include fault time, fault cause and operation and maintenance method, such as Figure 2 As shown, the present application can construct a spatiotemporal knowledge graph based on fault data, which contains relevant knowledge such as fault signals, causes, fault descriptions, operation and maintenance methods, fault locations and times. Its multimodal knowledge can fully reflect the temporal and spatial characteristics of faults, reveal the correlation between different types of faults, and help improve the accuracy and efficiency of fault diagnosis.

[0028] Step 120: Obtain a fault picture according to the fault signal.

[0029] Among them, the fault picture is obtained by transforming the fault signal into the time-frequency domain. Since the fault signal may contain noise, the present application transforms the fault signal into the time-frequency domain to obtain the fault picture to reduce the noise. Specifically, the present application uses wavelet transform to convert the fault signal into a fault picture. Among them, wavelet transform is a new transform analysis method, which inherits and develops the localization idea of ​​short-time Fourier transform, while overcoming the shortcomings such as the window size does not change with frequency, and can provide a "time-frequency" window that changes with frequency. It is an ideal tool for signal time-frequency analysis and processing. Its main feature is that it can fully highlight the characteristics of certain aspects of the problem through transformation, can perform localized analysis of time and frequency, and gradually refine the signal at multiple scales through telescopic translation operations, and finally achieve time subdivision at high frequencies and frequency subdivision at low frequencies. It can automatically adapt to the requirements of time-frequency signal analysis, so as to focus on any details of the signal.

[0030] All information extracted from the vehicle's fault manual, production record report and related literature is recorded as a collection ,in, represents the number of defects, Indicates i A fault, if the fault There are four corresponding fault signals, namely ,in, Indicates from t =1 to t = The fault image is obtained by wavelet transforming four fault signals within a period of time. Indicates i The time when a failure occurs, Indicates a fault In time t The fully connected layer is used to divide each image into multiple image blocks, and flatten and connect the feature vectors of multiple image blocks to obtain fully connected features. For a sample S, the fault image obtained by wavelet transform , divide each fault image into several small images and flatten them through the fully connected layer to obtain a feature vector set with dimension D ,in, For the f Picture No. j features and add a learnable embedding vector and the positional encoding vector , Represents a vector dimension of The vector space is used to retain the position information of each feature vector in the original fault image, and then the obtained feature vectors are concatenated to form the input feature of the fault image. ,in:

[0031] ,

[0032] in, Indicates a connection operation. , is a feature vector in a fault image The number of Together with the original fault image, it is imported into the model as a sample for learning.

[0033] Step 130: According to the fault image, the fault description text and the initial model, obtain the predicted fault type corresponding to the fault image and the fault description text.

[0034] like Figure 2As shown, the initial model includes a knowledge fusion module and a classification module. The knowledge fusion module is used to fuse the fault image and the fault description text based on the temporal features and spatial features to obtain the image features of the fault image and the text features of the fault description text. The classification module is used to obtain the predicted fault type based on the image features and the text features.

[0035] This application obtains the corresponding fault image and fault description text from the spatiotemporal knowledge graph through the visual encoder and word embedding layer respectively. After obtaining the fault image, the fault image and fault description text (specifically including fault description, fault location and time, etc.) are imported into the initial model together to obtain the output of the initial model, that is, the predicted fault type corresponding to the fault image and fault description text, and the initial model is trained in combination with the actual fault type. Among them, the initial model includes a knowledge fusion module and a classification module. The knowledge fusion module is used to fuse the fault image (fault signal feature) and the fault description text (fault semantic feature) based on time features and spatial features to obtain the corresponding image features and text features to improve the expression ability of the initial model, and the classification module is used to classify the image features and text features to obtain the fault category to improve the precision and accuracy of fault diagnosis and operation and maintenance decisions.

[0036] Step 140: Adjust the model parameters of the initial model to reduce the loss between the real fault type and the predicted fault type, and obtain a vehicle fault diagnosis model.

[0037] After obtaining the predicted fault type output by the initial model, the present application calculates the loss between the actual fault type and the predicted fault type. If the value of the target loss function is large (for example, greater than a preset value), the model parameters of the initial model are adjusted to reduce the loss between the actual fault type and the predicted fault type. By iteratively updating the model parameters of the initial model, model parameters with smaller losses (less than the preset values) are finally obtained, i.e., a vehicle fault diagnosis model is obtained.

[0038] The present application provides a method for training a vehicle fault diagnosis model, which obtains fault data; wherein the fault data includes a fault signal, a fault description text and a real fault type, and both the fault signal and the fault description text include time features and space features; according to the fault signal, a fault picture is obtained; wherein the fault picture is obtained by transforming the fault signal into the time-frequency domain; according to the fault picture, the fault description text and the initial model, a predicted fault type corresponding to the fault picture and the fault description text is obtained; wherein the initial model includes a knowledge fusion module and a classification module, the knowledge fusion module is used to fuse the fault picture and the fault description text based on the time features and the space features to obtain the image features of the fault picture and the text features of the fault description text, and the classification module is used to obtain the predicted fault type based on the image features and the text features; adjust The model parameters of the initial model are used to reduce the loss between the real fault type and the predicted fault type, and obtain the vehicle fault diagnosis model; that is, by obtaining the fault signal, the fault description text and the real fault type to obtain the time characteristics and spatial characteristics of the fault and the correlation between different types of faults, various types of fault diagnosis can be realized, and the fault signal is converted into a fault image to reduce the noise signal in the fault signal, and the fault image and the fault description text are fused based on the time characteristics and spatial characteristics to obtain the image features of the fault image and the text features of the fault description text, and the predicted fault type is obtained based on the corresponding image features and text features, so as to improve the diagnosis accuracy by fusing multimodal features, and match the image features and text features from the two dimensions of time and space to further improve the diagnosis accuracy and accuracy of the model.

[0039] In one embodiment, the specific implementation method of the above step 140 can be: based on the fault picture and the fault description text, calculate the spatial loss function of the initial model; based on the fault picture and the fault description text, calculate the time loss function of the initial model; based on the spatial loss function and the time loss function, calculate the loss function of the initial model; based on the loss function of the initial model, adjust the model parameters of the initial model to achieve that the loss between the actual fault type and the predicted fault type is less than a preset loss threshold, and obtain a vehicle fault diagnosis model.

[0040] This application inputs the fault image and fault description text (fault description, fault location and time) converted from the fault signal into the initial model, calculates the spatial loss function and the time loss function, calculates the loss function of the initial model by combining the spatial loss function and the time loss function, and adjusts the model parameters of the initial model based on the loss function of the initial model to achieve a loss between the actual fault type and the predicted fault type that is less than a preset loss threshold, thereby obtaining a vehicle fault diagnosis model.

[0041] In one embodiment, a specific implementation method for calculating the spatial loss function of the initial model based on the fault image and the fault description text may be: the knowledge fusion module includes multiple forward feedback layers, and multiple forward feedback layers are used to perform feature recognition on the fault image and the fault description text respectively to obtain multiple feature pairs; wherein each feature pair includes image features and text features corresponding to the same fault; based on multiple feature pairs, the spatial loss function is calculated.

[0042] like Figure 2 As shown, the knowledge fusion module in this application includes multiple stacked multi-head attention layers, forward feedback layers, regularization layers, and residual connection layers, wherein multiple forward feedback layers (experts) are randomly activated during training to enhance the fusion between the forward feedback layers, thereby improving the generalization ability of the model. The process of feature recognition of fault images by the knowledge fusion module can be expressed as ,in, is the global image feature, is the local image token sequence, is the length of the local image token sequence, is the feature vector space, d is the dimension of the feature vector space, represents the image feature recognition function. The process of feature recognition of fault description text by knowledge fusion module can be expressed as ,in represents the global text feature, Represents a local text token sequence, W Indicates the length of the local text token sequence. is the starting text data, Represents the text feature recognition function. For each input fault S, the image features and text features extracted by the knowledge fusion module can be expressed as feature pairs: ,in, represents the occurrence time of fault S, express t The global image features at the moment, express t The local image features at the moment, express t Local text features at the moment, express t The global text features at the moment, and based on all feature pairs, calculate the spatial loss function.

[0043] In one embodiment, multiple forward feedback layers are used to perform feature recognition on fault images and fault description texts respectively, and the specific implementation method for obtaining multiple feature pairs may be: calculating the probability of each forward feedback layer recognizing the fault image and the fault description text; selecting a predetermined number of forward feedback layers according to the probability from large to small to perform feature recognition on the fault image and the fault description text to obtain multiple feature pairs.

[0044] Specifically, the process of multiple forward feedback layers identifying fault images and fault description texts can be described as follows:

[0045] ,

[0046] ,

[0047] ,

[0048] ;

[0049] Among them, MSA represents the process of mining features by the multi-head attention mechanism, and CoE represents the process of mining features by the knowledge fusion module. LN represents the regularization function, It is the feature vector formed by combining the image feature vector and the text feature vector in the first layer of knowledge fusion module. is the feature vector mined by the multi-head attention mechanism, is the feature vector mined through multiple forward feedback layers, The feature vector obtained by fusing the image feature vector and the text feature vector in the L-th layer knowledge fusion module. y is the output of a layer of knowledge fusion module. v 1. v 2. ...is the visual feature vector, t 1. t 2. ..., t N is the text feature vector, L is the number of feature vectors. The set of multiple forward feedback layers in the knowledge fusion module can be expressed as , e 1. e 2. ... e E There are multiple forward feedback layers, and the probability of a token being assigned to each forward feedback layer can be calculated using the linear layer:

[0050] ,

[0051] in x represents a linear layer, represents the weight parameter of the linear layer, softmax express softmax Activation function. Each token is represented by a probability GATE(x) Highest k The forward feedback layer mines features to obtain multiple feature pairs. k is a preset positive integer, and is softmax The function obtains the probability of the final output:

[0052] ;

[0053] in, For the i A feedback-forward layer.

[0054] In one embodiment, based on multiple feature pairs, the specific implementation method of calculating the spatial loss function can be: calculating the global loss function and local loss function of the fault image and fault description text in each feature pair; calculating the spatial loss function based on the global loss function and local loss function corresponding to all feature pairs.

[0055] For fault S i The first t Feature pairs , extract global text features through text encoder and token feature sequence By comparing and learning the spatial distance between the global features of the image and the global features of the text, the fault image and the fault description text in the feature pair are matched. Specifically, the spatial distance between the matched fault image and the fault description text is close, while the spatial distance between the unmatched image data and the text data is far. The matching process can be expressed as:

[0056] ,

[0057] ;

[0058] in, represents the first spatial distance of the global text feature relative to all global image features at each moment, represents the first spatial distance of the global image feature relative to all global text features at each moment, B is the batchsieze size, represents the cosine similarity of two vectors, represents the scaling factor, |S k | Represents the total number of feature pairs for each fault sequence.

[0059] Then the global loss function can be expressed as:

[0060] .

[0061] in, is a global loss function, targeting local features of the image and local features of text , this application calculates No. j feature vector and The cosine similarity of all image feature vectors in , and the softmax function is applied to all cosine similarities to obtain the visual text representation . Using contrastive learning to align and The process can be expressed as:

[0062] ,

[0063] ;

[0064] in, is the scaling factor.

[0065] For each image-text feature pair , multimodal feature fusion is achieved by multiplying corresponding elements. The process can be expressed as . Eigenvector Represents the relationship and importance of different dimensions and forms a matrix , and obtain the weights of different image-text feature pairs :

[0066] ;

[0067] in, , Q and K is the parameter matrix that can be learned during model training, d is the dimension of the matrix, T Represents the transpose of a matrix.

[0068] Will As a weight, we get the local alignment loss of image features :

[0069] ;

[0070] in, |S t | is the number of text feature vectors, and |D| is the number of image feature vectors.

[0071] Similarly, the text feature local alignment loss can be calculated based on the cosine similarity between a single feature vector in the image local feature and all text feature vectors in the text local feature. :

[0072] .

[0073] Therefore, the overall local alignment loss can be expressed as:

[0074] .

[0075] After the global loss function and the local loss function are calculated, the spatial loss function is calculated based on the global loss function and the local loss function. Specifically, the spatial loss function is calculated based on the weighted sum of the global loss function and the local loss function.

[0076] In one embodiment, the specific implementation method of calculating the time loss function of the initial model based on the fault picture and the fault description text may be: calculating the deviation between the fault picture and the fault description text in the time dimension; calculating the time loss function based on the deviation between the fault picture and the fault description text in the time dimension.

[0077] Considering that there are also time series features in the fault pictures and fault description texts, this application uses time series features to further shorten the distance between the global feature vectors of the text and pictures of the same fault in the high-dimensional space, and increase the distance between the global feature vectors of the text and pictures of different faults. and The distance between all eigenvectors in The closest eigenvector for:

[0078] ;

[0079] in, Represents the image feature vector and the image feature vector The cosine similarity of is the weight of each feature vector, Represents a sequence of image features The image feature vector in , is the text feature sequence The text feature vector of t max For time t The maximum value of .

[0080] In order to improve the model's ability to capture temporal features, this application introduces classification loss and one-hot encodes the prediction results to convert global image features into With global text features Correspondingly, we get the loss:

[0081] ;

[0082] in , F represents the linear layer and softmax operation, Indicates that the prediction result is marked with a one-hot encoding vector.

[0083] Therefore, for sequence S, the objective function of forward propagation can be expressed as:

[0084] ;

[0085] in, To transform the global image features With global text features The corresponding loss value.

[0086] In the back propagation process, we first get and The similarity distribution of all feature vectors in :

[0087] ;

[0088] Since the farther the distance from time t is, the smaller the similarity is, the similarity distribution Approximate Gaussian distribution. In order to make the similarity distribution Approaching Gaussian distribution, in the process of back propagation Apply a Gaussian prior probability and then maximize , thus ensuring that a similar distribution peak occurs at time t, even if The value approaches 0, where is the mean, is the variance. Then introduce the regularization term To reduce the impact of abnormal variables on the optimization results, the objective function of back propagation is obtained:

[0089] ;

[0090] in, , represents the regularization weight, Represents a hyperparameter. Combine the objective function of the forward propagation and the objective function of the back propagation to obtain the objective function starting from the image feature vector .

[0091] Similarly, we can calculate the objective function starting from the text ,

[0092] , , ;

[0093] ,

[0094] in, , , z is a variable, represents the regularization weight, and All represent hyperparameters, , which is also the probability value of the defect type output by the prediction model, Represents the operation of the second fully connected layer.

[0095] Then the objective function considering the time series characteristics can be expressed as:

[0096] ;

[0097] The final target loss function of the vehicle fault diagnosis model in this application It can be expressed as:

[0098] ;

[0099] in and represents the hyperparameter that controls the weight of the loss function.

[0100] In order to better train the model, this application designs the training into three stages: understanding training stage, multimodal training stage and initialization stage. In the understanding training stage, this application inputs the wavelet transform image, fault description, fault location and time into the vehicle fault diagnosis model, and each forward feedback layer extracts the image features and text features respectively, so that the vehicle fault diagnosis model can capture the fault features and identify the fault type. In the multimodal training stage, all forward feedback layers are used to extract image features and text features, and by fine-tuning the multimodal data to enhance the recognition ability and controllability of the vehicle fault diagnosis model, the vehicle fault diagnosis model can also understand the multimodal data. At the same time, the weight of this stage is also used as the initial weight of the initialization stage to alleviate the problem of sparse model learning difficulties. In the initialization stage, the image token and the text token are first input into the router module. The router module calculates the probability of each forward feedback layer processing each token, and specifies the top k forward feedback layers with the largest probability to mine the features of a token. The router module consists of multiple linear layers, nonlinear activation functions, and normalization layers. When the vehicle fault diagnosis model mines the features of a token, only the top k forward feedback layers with the largest probability are activated by the router module, and the remaining forward feedback layers remain inactive, thereby reducing computing resources and increasing information exchange between forward feedback layers, thereby improving the learning ability of the vehicle fault diagnosis model.

[0101] In one embodiment, a specific implementation method of fusing a fault picture and a fault description text based on time features and spatial features to obtain image features of the fault picture and text features of the fault description text may be: calculating the time deviation between the fault picture and the fault description text, and the spatial position deviation between the fault picture and the fault description text; matching fault pictures and fault description texts whose time deviation is less than a preset time threshold and whose spatial position deviation is less than a preset distance threshold to obtain image features and text features.

[0102] This application uses the knowledge fusion module to align and match fault images and fault description texts from two dimensions: spatial features and temporal features, so as to form a feature pair of text features and image features corresponding to the same fault, so as to make full use of the mutual fusion of text features and image features corresponding to the same fault, thereby improving the diagnostic precision and accuracy of the model.

[0103] The present application also provides a vehicle fault diagnosis model, including: a knowledge fusion module and a classification module, the knowledge fusion module is used to fuse the fault picture and the fault description text based on the time feature and the space feature to obtain the image feature of the fault picture and the text feature of the fault description text, and the classification module is used to obtain the vehicle fault type based on the image feature and the text feature; wherein the vehicle fault diagnosis model is obtained by using any of the above-mentioned vehicle fault diagnosis model training methods.

[0104] The present application provides a vehicle fault diagnosis model, which obtains predicted fault types corresponding to fault pictures and fault description texts according to fault pictures, fault description texts and an initial model; wherein the vehicle fault diagnosis model includes a knowledge fusion module and a classification module, the knowledge fusion module is used to fuse the fault pictures and fault description texts based on time features and space features to obtain image features of the fault pictures and text features of the fault description texts, the classification module is used to obtain predicted fault types based on image features and text features, thereby realizing multiple types of fault diagnosis, and converting fault signals into fault pictures to reduce noise signals in fault signals, fusing the fault pictures and fault description texts based on time features and space features to obtain image features of the fault pictures and text features of the fault description texts, and obtaining predicted fault types based on corresponding image features and text features, so as to improve the diagnosis accuracy by fusing multimodal features, and matching image features and text features from two dimensions of time and space, so as to further improve the diagnosis accuracy and precision of the model.

[0105] In one embodiment, the knowledge fusion module includes multiple forward feedback layers, and the multiple forward feedback layers perform feature recognition on the fault image and the fault description text respectively to obtain multiple feature pairs; wherein each feature pair includes image features and text features corresponding to the same fault.

[0106] The present application utilizes multiple feedforward layers to be randomly activated during training to enhance the inter-integration between the feedforward layers, so as to fully utilize the mutual integration of text features and image features corresponding to the same fault, thereby improving the generalization ability of the model.

[0107] The present application also provides a method for using a vehicle fault diagnosis model, including: obtaining fault data of a target vehicle; wherein the fault data of the target vehicle includes a fault signal and a fault description text of the target vehicle; inputting the fault signal and the fault description text of the target vehicle into the vehicle fault diagnosis model to obtain the fault type of the target vehicle; wherein the vehicle fault diagnosis model is obtained using any of the above-mentioned vehicle fault diagnosis model training methods.

[0108] The present application provides a method for using a vehicle fault diagnosis model, which obtains fault data of a target vehicle; wherein the fault data of the target vehicle includes a fault signal and a fault description text of the target vehicle; the fault signal and the fault description text of the target vehicle are input into the vehicle fault diagnosis model to obtain the fault type of the target vehicle; wherein the vehicle fault diagnosis model includes a knowledge fusion module and a classification module, the knowledge fusion module is used to fuse the fault image and the fault description text based on the time feature and the space feature to obtain the image feature of the fault image and the text feature of the fault description text, and the classification module is used to obtain the predicted fault type based on the image feature and the text feature; that is, by obtaining the fault data to obtain the time feature and the space feature of the fault and the correlation between different types of faults, multiple types of fault diagnosis can be achieved, and the fault image and the fault description text are fused based on the time feature and the space feature to obtain the image feature of the fault image and the text feature of the fault description text, and the fault type is obtained based on the corresponding image feature and the text feature, so as to improve the diagnosis accuracy by fusing multimodal features, and match the image feature and the text feature from the two dimensions of time and space, so as to further improve the diagnosis accuracy and accuracy of the model.

[0109] Figure 3 FIG. 1 is a schematic diagram of a vehicle fault diagnosis model training device provided by an exemplary embodiment of the present application. Figure 3 As shown, the training device 20 of the vehicle fault diagnosis model includes: a fault data acquisition module 21, which is used to acquire fault data; wherein the fault data includes a fault signal, a fault description text and a real fault type, and the fault signal and the fault description text both include time features and space features; a fault picture acquisition module 22, which is used to obtain a fault picture according to the fault signal; wherein the fault picture is obtained by transforming the fault signal into the time-frequency domain; a prediction type identification module 23, which is used to obtain a predicted fault type corresponding to the fault picture and the fault description text according to the fault picture, the fault description text and the initial model, wherein the initial model includes a knowledge fusion module and a classification module, the knowledge fusion module is used to fuse the fault picture and the fault description text based on the time features and the space features to obtain the image features of the fault picture and the text features of the fault description text, and the classification module is used to obtain the predicted fault type based on the image features and the text features; a model parameter adjustment module 24, which is used to adjust the model parameters of the initial model to reduce the loss between the real fault type and the predicted fault type, and obtain the vehicle fault diagnosis model.

[0110] The present application provides a training device for a vehicle fault diagnosis model, which acquires fault data through a fault data acquisition module 21; wherein the fault data includes a fault signal, a fault description text and a real fault type, and both the fault signal and the fault description text include time features and space features; a fault picture acquisition module 22 acquires a fault picture according to the fault signal; wherein the fault picture is acquired by transforming the fault signal into the time-frequency domain; a prediction type recognition module 23 acquires a predicted fault type corresponding to the fault picture and the fault description text according to the fault picture, the fault description text and the initial model; wherein the initial model includes a knowledge fusion module and a classification module, the knowledge fusion module is used to fuse the fault picture and the fault description text based on the time features and the space features to obtain the image features of the fault picture and the text features of the fault description text, and the classification module is used to acquire the predicted fault type corresponding to the fault picture and the fault description text based on the image features and the text features. The predicted fault type is obtained; the model parameter adjustment module 24 adjusts the model parameters of the initial model to reduce the loss between the real fault type and the predicted fault type, and obtains the vehicle fault diagnosis model; that is, by obtaining the fault signal, the fault description text and the real fault type to obtain the time characteristics and spatial characteristics of the fault and the correlation between different types of faults, various types of fault diagnosis can be achieved, and the fault signal is converted into a fault image to reduce the noise signal in the fault signal, the fault image and the fault description text are fused based on the time characteristics and the space characteristics to obtain the image features of the fault image and the text features of the fault description text, and the predicted fault type is obtained based on the corresponding image features and text features, so as to improve the diagnostic accuracy by fusing multimodal features, and match the image features and text features from the two dimensions of time and space to further improve the diagnostic accuracy and precision of the model.

[0111] In one embodiment, the above-mentioned model parameter adjustment module 24 can be further configured as: based on the fault picture and the fault description text, calculating the spatial loss function of the initial model; based on the fault picture and the fault description text, calculating the time loss function of the initial model; based on the spatial loss function and the time loss function, calculating the loss function of the initial model; based on the loss function of the initial model, adjusting the model parameters of the initial model to achieve that the loss between the actual fault type and the predicted fault type is less than a preset loss threshold, thereby obtaining a vehicle fault diagnosis model.

[0112] In one embodiment, the above-mentioned model parameter adjustment module 24 can be further configured as: the knowledge fusion module includes multiple forward feedback layers, and multiple forward feedback layers are used to perform feature recognition on the fault image and the fault description text respectively to obtain multiple feature pairs; wherein each feature pair includes the image feature and text feature corresponding to the same fault; based on multiple feature pairs, the spatial loss function is calculated.

[0113] In one embodiment, the above-mentioned model parameter adjustment module 24 can be further configured to: calculate the probability of each forward feedback layer identifying the fault picture and the fault description text; select a predetermined number of forward feedback layers from large to small according to the probability to perform feature recognition on the fault picture and the fault description text to obtain multiple feature pairs.

[0114] In one embodiment, the above-mentioned model parameter adjustment module 24 can be further configured to: calculate the global loss function and local loss function of the fault image and fault description text in each feature pair; calculate the spatial loss function based on the global loss function and local loss function corresponding to all feature pairs.

[0115] In one embodiment, the model parameter adjustment module 24 may be further configured to: calculate the deviation between the fault image and the fault description text in the time dimension; and calculate the time loss function based on the deviation between the fault image and the fault description text in the time dimension.

[0116] In one embodiment, the above-mentioned model parameter adjustment module 24 can be further configured to: calculate the time deviation between the fault picture and the fault description text, and the spatial position deviation between the fault picture and the fault description text; match the fault picture and the fault description text whose time deviation is less than a preset time threshold and whose spatial position deviation is less than a preset distance threshold to obtain image features and text features.

[0117] Below, reference Figure 4 The electronic device according to the embodiment of the present application is described. The electronic device may be any one or both of the first device and the second device, or a stand-alone device independent of them, and the stand-alone device may communicate with the first device and the second device to receive the collected input signal from them.

[0118] Figure 4 A block diagram of an electronic device according to an embodiment of the present application is illustrated.

[0119] like Figure 4 As shown, the electronic device 10 includes one or more processors 11 and a memory 12 .

[0120] The processor 11 may be a central processing unit (CPU) or other forms of processing units having data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 10 to perform desired functions.

[0121] The memory 12 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, a random access memory (RAM) and / or a cache memory (cache), etc. The non-volatile memory may include, for example, a read-only memory (ROM), a hard disk, a flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 11 may run the program instructions to implement the methods of the various embodiments of the present application described above and / or other desired functions. Various contents such as input signals, signal components, noise components, etc. may also be stored in the computer-readable storage medium.

[0122] In one example, the electronic device 10 may further include: an input device 13 and an output device 14 , and these components are interconnected via a bus system and / or other forms of connection mechanisms (not shown).

[0123] When the electronic device is a stand-alone device, the input device 13 may be a communication network connector, which is used to receive the collected input signals from the first device and the second device.

[0124] In addition, the input device 13 may also include, for example, a keyboard, a mouse, and the like.

[0125] The output device 14 can output various information to the outside, including the determined distance information, direction information, etc. The output device 14 can include, for example, a display, a speaker, a printer, a communication network and a remote output device connected thereto, and the like.

[0126] Of course, to simplify, Figure 4 Only some of the components related to the present application in the electronic device 10 are shown, and components such as a bus, an input / output interface, etc. are omitted. In addition, according to specific application situations, the electronic device 10 may also include any other appropriate components.

[0127] In addition to the above-mentioned methods and devices, an embodiment of the present application may also be a computer program product, which includes computer program instructions, which, when executed by a processor, enable the processor to execute the steps of the method according to various embodiments of the present application described in the above-mentioned "Exemplary Method" section of this specification.

[0128] The computer program product may be written in any combination of one or more programming languages ​​to write program codes for performing the operations of the embodiments of the present application, including object-oriented programming languages, such as Java, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as an independent software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0129] In addition, an embodiment of the present application may also be a computer-readable storage medium on which computer program instructions are stored. When the computer program instructions are executed by a processor, the processor executes the steps of the method according to various embodiments of the present application described in the above "Exemplary Method" section of this specification.

[0130] The computer readable storage medium can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can include, for example, but is not limited to, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0131] The basic principles of the present application are described above in conjunction with specific embodiments. However, it should be noted that the advantages, strengths, effects, etc. mentioned in the present application are only examples and not limitations, and it cannot be considered that these advantages, strengths, effects, etc. are required by each embodiment of the present application. In addition, the specific details disclosed above are only for the purpose of illustration and ease of understanding, not for limitation, and the above details do not limit the present application to being implemented by adopting the above specific details.

[0132] The block diagrams of the devices, apparatuses, equipment, and systems involved in this application are only illustrative examples and are not intended to require or imply that they must be connected, arranged, and configured in the manner shown in the block diagram. As will be appreciated by those skilled in the art, these devices, apparatuses, equipment, and systems can be connected, arranged, and configured in any manner. Words such as "including", "comprising", "having", etc. are open words, referring to "including but not limited to", and can be used interchangeably with them. The words "or" and "and" used here refer to the words "and / or" and can be used interchangeably with them, unless the context clearly indicates otherwise. The words "such as" used here refer to the phrase "such as but not limited to", and can be used interchangeably with them.

[0133] It should also be noted that in the apparatus, device and method of the present application, each component or each step can be decomposed and / or recombined. Such decomposition and / or recombination should be regarded as equivalent solutions of the present application.

[0134] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present application. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of the present application. Therefore, the present application is not intended to be limited to the aspects shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

[0135] The above description has been given for the purpose of illustration and description. In addition, this description is not intended to limit the embodiments of the present application to the forms disclosed herein. Although multiple example aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, changes, additions and sub-combinations thereof.

Claims

1. A training method for a vehicle fault diagnosis model, characterized in that: include: Acquire fault data; wherein the fault data includes a fault signal, a fault description text and a real fault type, and both the fault signal and the fault description text include time characteristics and space characteristics; According to the fault signal, a fault picture is obtained; wherein the fault picture is obtained by performing time-frequency domain transformation on the fault signal; According to the fault picture, the fault description text and the initial model, a predicted fault type corresponding to the fault picture and the fault description text is obtained; wherein the initial model includes a knowledge fusion module and a classification module, the knowledge fusion module is used to fuse the fault picture and the fault description text based on the time feature and the spatial feature to obtain the image feature of the fault picture and the text feature of the fault description text, and the classification module is used to obtain the predicted fault type based on the image feature and the text feature; Adjusting the model parameters of the initial model to reduce the loss between the real fault type and the predicted fault type, and obtaining the vehicle fault diagnosis model; The adjusting the model parameters of the initial model to reduce the loss between the real fault type and the predicted fault type to obtain the vehicle fault diagnosis model includes: Calculating the spatial loss function of the initial model based on the fault picture and the fault description text; Calculating the time loss function of the initial model based on the fault picture and the fault description text; Calculating the loss function of the initial model based on the spatial loss function and the temporal loss function; Based on the loss function of the initial model, the model parameters of the initial model are adjusted to achieve that the loss between the real fault type and the predicted fault type is less than a preset loss threshold, thereby obtaining the vehicle fault diagnosis model.

2. The vehicle fault diagnosis model training method according to claim 1, characterized in that: The calculating the spatial loss function of the initial model based on the fault picture and the fault description text includes: The knowledge fusion module includes a plurality of forward feedback layers, and the plurality of forward feedback layers are used to perform feature recognition on the fault image and the fault description text respectively to obtain a plurality of feature pairs; wherein each feature pair includes a picture feature and a text feature corresponding to the same fault; The spatial loss function is calculated based on the plurality of feature pairs.

3. The training method of the vehicle fault diagnosis model according to claim 2, characterized in that: The plurality of forward feedback layers are used to perform feature recognition on the fault image and the fault description text respectively to obtain a plurality of feature pairs, including: Calculating the probability of each of the forward feedback layers identifying the fault image and the fault description text; A predetermined number of the forward feedback layers are selected from large to small according to the probability to perform feature recognition on the fault image and the fault description text to obtain a plurality of feature pairs.

4. The vehicle fault diagnosis model training method according to claim 2, characterized in that: The calculating the spatial loss function based on the plurality of feature pairs comprises: Calculating a global loss function and a local loss function of the fault image and the fault description text in each feature pair; The spatial loss function is calculated based on the global loss function and the local loss function corresponding to all the feature pairs.

5. The vehicle fault diagnosis model training method according to claim 1, characterized in that: The calculating the time loss function of the initial model based on the fault picture and the fault description text includes: Calculating the deviation between the fault image and the fault description text in the time dimension; The time loss function is calculated based on the deviation between the fault image and the fault description text in the time dimension.

6. The vehicle fault diagnosis model training method according to claim 1, characterized in that: The fusing the fault image and the fault description text based on the time feature and the spatial feature to obtain the image feature of the fault image and the text feature of the fault description text includes: Calculating the time deviation between the fault image and the fault description text, and the spatial position deviation between the fault image and the fault description text; The fault picture and the fault description text whose time deviation is less than a preset time threshold and whose spatial position deviation is less than a preset distance threshold are fused to obtain the picture feature and the text feature.

7. A vehicle fault diagnosis model, characterized in that: include: A knowledge fusion module and a classification module, wherein the knowledge fusion module is used to fuse the fault picture and the fault description text based on time features and space features to obtain the image features of the fault picture and the text features of the fault description text, and the classification module is used to obtain the vehicle fault type based on the image features and the text features; wherein the vehicle fault diagnosis model is obtained using the vehicle fault diagnosis model training method described in any one of claims 1 to 6.

8. The vehicle fault diagnosis model according to claim 7, characterized in that: The knowledge fusion module includes multiple forward feedback layers, and the multiple forward feedback layers respectively perform feature recognition on the fault image and the fault description text to obtain multiple feature pairs; wherein each feature pair includes image features and text features corresponding to the same fault.

9. A method for using a vehicle fault diagnosis model, characterized in that: include: Acquire fault data of a target vehicle; wherein the fault data of the target vehicle includes a fault signal and a fault description text of the target vehicle; The fault signal and fault description text of the target vehicle are input into a vehicle fault diagnosis model to obtain the fault type of the target vehicle; wherein the vehicle fault diagnosis model is obtained by using the vehicle fault diagnosis model training method described in any one of claims 1-6.

10. A training device for a vehicle fault diagnosis model, characterized in that: include: A fault data acquisition module, used to acquire fault data; wherein the fault data includes a fault signal, a fault description text and a real fault type, and the fault signal and the fault description text both include time characteristics and space characteristics; A fault image acquisition module is used to obtain a fault image according to the fault signal; wherein the fault image is obtained by performing time-frequency domain transformation on the fault signal; A prediction type identification module, used for obtaining the predicted fault type corresponding to the fault picture and the fault description text according to the fault picture, the fault description text and the initial model, wherein the initial model includes a knowledge fusion module and a classification module, the knowledge fusion module is used for fusing the fault picture and the fault description text based on the time feature and the spatial feature to obtain the image feature of the fault picture and the text feature of the fault description text, and the classification module is used for obtaining the predicted fault type based on the image feature and the text feature; A model parameter adjustment module, used for adjusting the model parameters of the initial model to reduce the loss between the real fault type and the predicted fault type, and obtaining the vehicle fault diagnosis model; The model parameter adjustment module is further configured as follows: Calculating the spatial loss function of the initial model based on the fault picture and the fault description text; Calculating the time loss function of the initial model based on the fault picture and the fault description text; Calculating the loss function of the initial model based on the spatial loss function and the temporal loss function; Based on the loss function of the initial model, the model parameters of the initial model are adjusted to achieve that the loss between the real fault type and the predicted fault type is less than a preset loss threshold, thereby obtaining the vehicle fault diagnosis model.

11. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and the computer program is used to execute the training method of the vehicle fault diagnosis model described in any one of claims 1 to 6 and / or the method for using the vehicle fault diagnosis model described in claim 9.

Images