Engine fault diagnosis method and device, electronic equipment and storage medium

Abstract

The invention provides an engine fault diagnosis method and device, electronic equipment and a storage medium. The method comprises the steps that audio data of an engine on a to-be-diagnosed vehicle and voice data of a driver are acquired; performing information extraction on the transliteration text corresponding to the voice data to obtain keywords and semantic information; keywords and semantic information are taken as references to guide audio data to perform feature extraction, so that the feature extraction process pays more attention to audio clips related to faults, irrelevant information is reduced, and accurate and effective audio features are obtained; according to the invention, the fault diagnosis is carried out based on the audio characteristics, the fault diagnosis result of the engine is obtained, comprehensive and accurate engine fault diagnosis is realized, the diagnosis efficiency is improved, the diagnosis range is widened, the dependence on professional equipment and personnel is reduced, and the vehicle can carry out self-diagnosis more conveniently, so that the diagnosis cost is reduced, and the driving safety is improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to an engine fault diagnosis method, device, electronic equipment and storage medium. Background Art

[0002] In the context of the rapid development of today's automobile industry, the engine is the core component of the car, and its operating status is directly related to the overall performance, fuel efficiency and driving safety of the vehicle. However, as a complex mechanical system, the engine will inevitably have various faults during operation. If these faults are not discovered and eliminated in time, they will seriously threaten the driving safety of the vehicle.

[0003] Traditional engine fault diagnosis methods usually rely on professional diagnostic equipment and experienced technicians to identify faults through physical testing, vibration analysis, and reading fault codes from the on-board diagnostic system. Although these methods are effective to a certain extent, they generally have problems such as high cost, low efficiency, limited diagnostic range, and poor accuracy. In particular, the ability to identify non-mechanical faults such as abnormal combustion is weak, and they are easily disturbed by external factors such as environmental noise, resulting in increased inaccuracy and uncertainty in the diagnostic results. Summary of the invention

[0004] The present invention provides an engine fault diagnosis method, device, electronic device and storage medium, which are used to solve the problems in the prior art that engine fault diagnosis relies on professional diagnostic equipment and personnel, is costly, inefficient, has a limited diagnostic range, and is susceptible to environmental interference, so as to achieve accurate and reliable fault diagnosis, improve diagnostic efficiency, and reduce diagnostic costs and thresholds.

[0005] The present invention provides an engine fault diagnosis method, comprising: Acquire audio data of the engine on the vehicle to be diagnosed and voice data of the driver; Extracting information from the transcribed text corresponding to the voice data to obtain keywords and semantic information; Taking the keywords and the semantic information as reference, guiding the audio data to perform feature extraction to obtain audio features; Fault diagnosis is performed based on the audio features to obtain a fault diagnosis result of the engine.

[0006] According to an engine fault diagnosis method provided by the present invention, the keyword and the semantic information are used as references to guide the audio data to perform feature extraction to obtain audio features, including: Extracting features from the audio data to obtain features of each audio segment in the audio data; Based on the keywords and the semantic information, the features of the audio clips are fused to obtain audio features.

[0007] According to an engine fault diagnosis method provided by the present invention, the feature fusion of the features of the audio clips based on the keywords and the semantic information to obtain the audio features includes: Determining a weight of each audio segment in the audio data based on the keyword and the semantic information; Based on the weight of each audio segment in the audio data, feature fusion is performed on the features of each audio segment to obtain audio features.

[0008] According to an engine fault diagnosis method provided by the present invention, the fault diagnosis is performed based on the audio feature to obtain the engine fault diagnosis result, including: Deeply fusing the audio features and the semantic features of the transcribed text to obtain fused features; Fault diagnosis is performed based on the fusion features to obtain a fault diagnosis result of the engine.

[0009] According to an engine fault diagnosis method provided by the present invention, the deep fusion of the audio features and the semantic features of the transcribed text to obtain the fused features includes: Performing feature mapping on the audio features and the semantic features of the transcribed text to obtain audio embedding and semantic embedding in the same feature space; The audio embedding and the semantic embedding are deeply fused to obtain a fusion feature.

[0010] According to an engine fault diagnosis method provided by the present invention, the deep fusion of the audio embedding and the semantic embedding to obtain the fusion feature includes: Determining a cross dimension based on a correlation between the audio embedding and the semantic embedding; Based on the cross dimension, feature crossover is performed on the audio embedding and the semantic embedding to obtain a fusion feature.

[0011] According to an engine fault diagnosis method provided by the present invention, the fault diagnosis is performed based on the fusion feature to obtain the engine fault diagnosis result, including: Perform fault diagnosis based on the fusion features to obtain fault keywords of the engine; Determine a fault diagnosis result of the engine based on the fault keywords and the fault knowledge graph; The fault knowledge graph includes multiple keywords and their corresponding fault information, and the fault information includes at least one of a fault component, a fault type, a fault cause, and a fault solution.

[0012] The present invention also provides an engine fault diagnosis device, comprising: A data acquisition unit, used to acquire audio data of the engine on the vehicle to be diagnosed and voice data of the driver; An information extraction unit, used to extract information from the transcribed text corresponding to the voice data to obtain keywords and semantic information; A feature extraction unit, used to guide the audio data to perform feature extraction based on the keywords and the semantic information to obtain audio features; A fault diagnosis unit is used to perform fault diagnosis based on the audio feature to obtain a fault diagnosis result of the engine.

[0013] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements any of the above-described engine fault diagnosis methods when executing the computer program.

[0014] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the engine fault diagnosis method as described in any one of the above is implemented.

[0015] The engine fault diagnosis method, device, electronic device and storage medium provided by the present invention extract information from the transcribed text corresponding to the driver's voice data to obtain keywords and semantic information; use the keywords and semantic information as a reference to guide the engine's audio data to perform feature extraction, so that the feature extraction process pays more attention to audio clips related to the fault, reduces irrelevant information, and obtains accurate and effective audio features; performs fault diagnosis based on the audio features to obtain the engine's fault diagnosis results, thereby achieving comprehensive and accurate engine fault diagnosis, improving diagnostic efficiency, broadening the diagnostic scope, reducing dependence on professional equipment and personnel, enabling vehicles to perform self-diagnosis more conveniently, reducing diagnostic costs, and improving driving safety. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the present invention or the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0017] Figure 1It is a flow chart of the engine fault diagnosis method provided by the present invention; Figure 2 It is an overall flow chart of the engine fault diagnosis method provided by the present invention; Figure 3 It is a structural schematic diagram of the engine fault diagnosis device provided by the present invention; Figure 4 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0018] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0019] At present, engine NVH (Noise, Vibration, Harshness) fault diagnosis is highly dependent on professional diagnostic equipment and experienced technicians, and fault identification is mostly performed through sound analysis, vibration analysis, reading fault codes from the on-board diagnostics system (OBD), etc. Although these methods are effective to a certain extent, they generally have the following problems: First, it is costly and inefficient. For example, fault detection based on vibration analysis and OBD data requires specific diagnostic equipment and professional technicians to operate and analyze, which is costly and inconvenient for car owners to diagnose themselves.

[0020] Second, the diagnostic scope is limited. For example, the solution based on vibration analysis is mainly used to identify mechanical fault models, and has weak recognition capabilities for non-mechanical faults such as combustion abnormalities. The solution based on OBD data relies on preset fault codes and is difficult to identify faults without preset codes.

[0021] Third, it is susceptible to environmental interference. For example, solutions based on sound analysis are easily disturbed by environmental noise, which leads to increased inaccuracy and uncertainty in diagnostic results.

[0022] In this regard, the present invention provides an engine fault diagnosis method, which aims to improve the intelligence and interactivity of fault diagnosis, that is, through real-time interaction with the driver, making full use of the driver's perception and experience, and using this as a guide to extract audio features, so that the feature extraction process pays more attention to audio clips related to the fault, and reduces irrelevant information, thereby ensuring the accuracy and effectiveness of audio feature extraction. Furthermore, fault diagnosis based on this audio feature can achieve comprehensive and accurate engine fault diagnosis, improve diagnostic efficiency and driving experience, reduce dependence on professional equipment and personnel, and enable vehicles to perform self-diagnosis more conveniently, thereby reducing diagnostic costs and improving driving safety.

[0023] Figure 1 It is a flow chart of the engine fault diagnosis method provided by the present invention, such as Figure 1 As shown, the method includes: Step 110, acquiring audio data of the engine of the vehicle to be diagnosed and voice data of the driver; Step 120, extracting information from the transcribed text corresponding to the voice data to obtain keywords and semantic information; Step 130, using keywords and semantic information as references, guiding the audio data to perform feature extraction to obtain audio features; Step 140 , performing fault diagnosis based on the audio features to obtain a fault diagnosis result of the engine.

[0024] Specifically, before performing engine fault diagnosis, the vehicle to be diagnosed must be determined first. The vehicle to be diagnosed here is the vehicle whose engine needs to be diagnosed. The vehicle to be diagnosed can be one or more vehicles, and their models, styles, etc. can be arbitrary, and the embodiment of the present invention does not make specific limitations on this.

[0025] After determining the vehicle to be diagnosed and before performing engine fault diagnosis, in an embodiment of the present invention, it is also necessary to obtain a basis for fault diagnosis, that is, the audio data of the engine. Specifically, the sound signal of the engine in the running state is collected through a sound pickup device, such as a vehicle-mounted microphone, and the sound signal is subjected to noise reduction, filtering and other processing to form the audio data of the engine.

[0026] However, considering that fault diagnosis based solely on sound signals can only identify mechanical faults, but cannot effectively identify non-mechanical faults such as combustion abnormalities, in this regard, in an embodiment of the present invention, it is proposed to obtain the driver's voice data to accurately identify engine faults in combination with the driver's voice description and the engine's sound signals, which not only improves the scope of fault diagnosis, but also improves the accuracy of diagnosis.

[0027] Therefore, in the embodiment of the present invention, the basis for engine fault diagnosis also includes the driver's voice data, that is, before performing fault diagnosis, the driver's voice data can also be collected through a sound pickup device. Specifically, the driver's voice description of the engine can be obtained through interaction with the driver during daily driving, thereby obtaining the driver's voice data.

[0028] Here, it should be noted that the collection of voice data can be actively triggered by the driver. For example, if the driver finds that the engine sound is abnormal, he or she will actively raise the abnormal phenomenon found during the interaction so that the on-board fault diagnosis system can detect it. Of course, it can also be passively output by the driver. For example, during the interaction process, the system asks the driver a question, asking him or her whether he or she perceives any abnormality in the engine. The driver can respond based on the question, so that the voice data of the driver describing the engine can be collected.

[0029] Furthermore, after obtaining the driver's voice data, in an embodiment of the present invention, the voice data can be processed to extract key semantics therein, such as the driver's semantic information, words and descriptions related to engine failure, etc.

[0030] Specifically, the voice data may be first transcribed to convert the voice description into text information, thereby obtaining the transcribed text corresponding to the voice data. The voice transcription process here may be implemented by conventional voice transcription engines, models, etc. Then, information extraction may be performed on the transcribed text to obtain the driver's semantics, as well as information related to the engine, faults, etc., thereby obtaining keywords and semantic information corresponding to the transcribed text. Here, the information extraction process may be implemented by traditional information extraction technology, large language models, or by natural language processing technology, that is, keywords and semantic information may be obtained directly by natural language processing technology based on the voice data.

[0031] After that, feature extraction can be performed on the audio data of the engine to obtain audio features. However, considering that the acquired audio data of the engine is in fragments, the features obtained by directly extracting features from the entire audio data are not only very scattered and fragmented, but also unclear in terms of "focus". Based on this, it is completely impossible to establish key features, resulting in the inability to quickly and accurately identify the engine fault in the final fault diagnosis. Based on this, an embodiment of the present invention proposes to extract audio features guided by the driver's voice description, that is, to use the keywords and semantic information obtained in the previous step to guide the feature extraction of the audio data, so that the audio feature extraction process pays more attention to the audio segments related to the engine fault and reduces irrelevant information, thereby ensuring the integrity, accuracy and effectiveness of the extracted audio features, and thus making the fault diagnosis process more comprehensive, efficient and accurate.

[0032] Here, dynamically adjusting the feature extraction process of the audio data based on the keywords and semantic information extracted from the driver's voice description can greatly improve the accuracy and efficiency of audio feature extraction. For example, if the driver describes the engine making a "click-click" sound, the feature extraction process will focus on the transient features and high-frequency components in the sound signal.

[0033] After that, the extracted audio features can be used for fault diagnosis to obtain the fault diagnosis result of the engine on the vehicle to be diagnosed. That is, based on the audio features of the engine obtained by extracting features from the audio data guided by the driver's voice description, the engine is diagnosed to determine its possible faulty parts, fault causes, fault types, solutions, etc., so as to comprehensively obtain the fault diagnosis result of the engine. Here, the fault diagnosis process can be implemented with the help of a preset fault knowledge graph, a fault diagnosis model, etc.

[0034] In the embodiment of the present invention, fault diagnosis is performed based on the keywords and semantic information corresponding to the driver's voice data and the audio features extracted from the audio data. This can comprehensively and accurately identify engine faults and obtain accurate fault diagnosis results. This not only improves the diagnostic efficiency, but also provides timely feedback on the fault diagnosis results during the interaction process, thereby enhancing the driver's understanding of the vehicle condition and improving the driving experience.

[0035] The engine fault diagnosis method provided by the present invention extracts information from the transcribed text corresponding to the driver's voice data to obtain keywords and semantic information; uses the keywords and semantic information as a reference to guide the engine's audio data to perform feature extraction, so that the feature extraction process pays more attention to audio clips related to the fault, reduces irrelevant information, and obtains accurate and effective audio features; performs fault diagnosis based on the audio features to obtain the engine's fault diagnosis results, thereby achieving comprehensive and accurate engine fault diagnosis, improving diagnostic efficiency, broadening the diagnostic scope, reducing dependence on professional equipment and personnel, and enabling the vehicle to perform self-diagnosis more conveniently, thereby reducing diagnostic costs and improving driving safety.

[0036] Based on the above embodiment, step 130 includes: Extract features from the audio data to obtain features of each audio segment in the audio data; Based on keywords and semantic information, the features of each audio clip are fused to obtain audio features.

[0037] Specifically, in step 130, the process of extracting features from audio data with reference to keywords and semantic information to obtain audio features may include: First, feature extraction may be performed on the audio data to extract signal features of each audio segment of the audio data, thereby obtaining features of each audio segment of the engine.

[0038] Here, specifically, feature extraction may be performed based on audio data to extract features of sound signals in each audio segment, such as frequency pattern, amplitude change, time domain features, etc., so as to obtain features of each audio segment.

[0039] Next, the keywords and semantic information corresponding to the driver's voice data can be used to perform feature fusion on the features of each audio clip extracted in the previous step. That is, the scattered features obtained by feature extraction can be fused with reference to the keywords and semantic information obtained by information extraction, so that the fusion process can focus on the features of audio clips related to the fault and ignore irrelevant features according to the guidance of keywords and semantic information, thereby ensuring the completeness, accuracy and effectiveness of the fused audio features.

[0040] Here, specifically, the extracted keywords and semantic information can be normalized first to organize them into expressions commonly used in fault diagnosis. For example, the extracted keywords and semantic information can be matched with a preset NVH fault database to determine relevant fault expressions, keywords, etc. from the database. Then, the audio feature extraction process can be dynamically adjusted according to the matched keywords, that is, the features of each audio clip are fused according to the matched keywords, so that the fusion process focuses more on features related to the described fault.

[0041] Based on the above embodiment, the features of each audio clip are fused based on keywords and semantic information to obtain audio features, including: Determine the weight of each audio segment in the audio data based on keywords and semantic information; Based on the weight of each audio segment in the audio data, the features of each audio segment are fused to obtain audio features.

[0042] Specifically, the process of fusing the features of each audio clip according to the keywords and semantic information to obtain the audio features may include: In order to make the feature fusion process pay more attention to the features related to the fault and be able to dynamically adjust the feature fusion process according to the driver's voice description, in an embodiment of the present invention, an attention mechanism can be introduced to dynamically adjust the attention distribution of the features of each audio segment during fusion according to reference information, so that the attention can be concentrated on the features of the audio segment related to the fault, while ignoring irrelevant features, thereby ensuring the accuracy of the audio features.

[0043] Specifically, in an embodiment of the present invention, weight calculation may be performed first to obtain the attention of the features of each audio segment when fused, that is, the weight of each audio segment. Here, specifically, the keywords and semantic information in the driver's voice description may be matched in a preset NVH fault database, and the attention that should be distributed on the features of each audio segment when the features are fused may be calculated based on the matched keywords, thereby obtaining the weight of each audio segment. Here, the weight calculation may use content-based attention, positional attention, etc.

[0044] Subsequently, the features of each audio clip can be fused according to the calculated weights of each audio clip to obtain audio features. That is, the features of each audio clip can be weighted and summarized according to the weights of each audio clip. Specifically, the weights are applied to the features of each audio clip to summarize the features of different time steps or frequencies, thereby generating summary features that pay more attention to important parts with high weights, namely, audio features.

[0045] In an embodiment of the present invention, an attention mechanism is introduced to process the driver's voice description more flexibly and intelligently, and the fusion process of the features of each audio segment is dynamically adjusted according to the key parts in the description, so that the feature fusion can dynamically focus on the features of the audio segment related to the driver's description, thereby reducing the interference of irrelevant information and improving the accuracy of audio feature extraction.

[0046] Based on the above embodiment, step 140 includes: Deeply fuse the audio features and the semantic features of the transcribed text to obtain fused features; Fault diagnosis is performed based on the fusion features to obtain the engine fault diagnosis results.

[0047] Specifically, in step 140, the process of performing fault diagnosis according to the audio features to obtain the engine fault diagnosis result may specifically include: Considering that the audio features obtained by feature extraction based on keywords and semantic information are single-modal features, in an embodiment of the present invention, in order to further optimize the fault diagnosis process and improve the accuracy of fault diagnosis, the multi-modal information obtained above can be fused so that the information contained in the fused features is richer and more expressive, and fault diagnosis based on this can obtain more accurate and reliable results.

[0048] In detail, in an embodiment of the present invention, first, the audio features and the transcribed text can be deeply fused, that is, the audio features and the semantic features of the transcribed text are fused to obtain fused features. Here, deep fusion is a deep combination of transcribed text and audio features through a deep learning model to obtain complementary features of text semantics and audio information from different angles, thereby producing richer and more expressive fusion features. Deep fusion is not only a simple splicing of information from two modalities together, but more importantly, through modeling and calculation, the information of the two modalities can influence and complement each other, thereby obtaining fusion features with higher-level semantic information.

[0049] Next, the fault diagnosis can be performed based on the fusion features obtained by deep fusion to obtain the engine fault diagnosis results. That is, based on the audio features of the engine obtained by extracting features from the audio data guided by the driver's voice description, the engine fault diagnosis is performed to determine its possible faulty parts, fault causes, fault types, solutions, etc., so as to comprehensively obtain the engine fault diagnosis results. Here, the fault diagnosis process can be implemented with the help of preset fault knowledge graphs, fault diagnosis models, etc.

[0050] Based on the above embodiment, the audio features and the semantic features of the transcribed text are deeply fused to obtain fused features, including: Perform feature mapping on the audio features and the semantic features of the transcribed text to obtain audio embedding and semantic embedding in the same feature space; The audio embedding and semantic embedding are deeply fused to obtain fused features.

[0051] Specifically, the process of deep fusion of audio features and semantic features of the transcribed text to obtain fused features can be implemented through a multimodal learning model. That is, a multimodal learning model based on Transformer or similar architecture can be used to first perform feature mapping on the audio features and semantic features of the transcribed text to map them to a common feature space. Specifically, in order to enable the audio features and semantic features to be compared and fused in the same feature space, they need to be converted to a common embedding space through an embedding layer to obtain the feature representation of the two in the embedding space, namely, audio embedding and semantic embedding.

[0052] Subsequently, the audio embedding and the semantic embedding can be deeply fused in this feature space / embedding space to obtain fused features. That is, the audio embedding and the semantic embedding in the same feature space can be fused through the multi-layer neural network structure in the multimodal learning model, so that the audio embedding and the semantic embedding can interact and fuse in the feature space.

[0053] Here, when audio embedding and semantic embedding are fused in the feature space, the fusion method adopted can be feature concatenation, weighted fusion, cross attention, feature crosses, etc.

[0054] Based on the above embodiment, the audio embedding and the semantic embedding are deeply fused to obtain fusion features, including: Determine the cross dimension based on the correlation between audio embedding and semantic embedding; Based on the cross dimension, the audio embedding and semantic embedding are cross-featured to obtain fused features.

[0055] Specifically, the process of deeply fusing the audio embedding and the semantic embedding to obtain the fusion feature may include: First, the cross dimension can be calculated, that is, the cross dimension can be calculated based on the correlation between the audio embedding and the semantic embedding. Specifically, the potential relationship between the audio embedding and the semantic embedding is analyzed to determine the dimensions on which the two can cross features and obtain the cross dimension. For example, the cross dimension can be determined based on semantic similarity, acoustic characteristic similarity, etc.

[0056] Then, cross features can be performed to obtain fused features, that is, the multi-layer neural network structure in the multimodal learning model can be used to perform cross operations on audio embedding and semantic embedding according to the cross dimension to obtain fused features. Here, the cross operation method can be any one of inner product, outer product, Cartesian product, and neural network cross.

[0057] In the embodiment of the present invention, feature crossover is performed in the multimodal learning model, so that audio features and semantic features can cross-complement each other, thereby generating richer and more expressive feature representations, thereby improving the accuracy and robustness of fault detection.

[0058] It is worth noting that before realizing the deep fusion of different modal features through the multimodal learning model, in the embodiment of the present invention, it is necessary to pre-train the multimodal learning model. That is, the audio features of the sample transcribed text and the sample audio corresponding to the sample speech can be used for model training, so that the audio features of the sample transcribed text and the sample audio can be input into the model at the same time, and the complex relationship between them can be learned through joint training, and weight parameters can be introduced to control the importance of the audio features of the sample transcribed text and the sample audio in the feature space. Furthermore, the loss of the model can be measured by a loss function to maximize the complementarity and compatibility between the audio features of the sample transcribed text and the sample audio, so that the model can automatically learn how to better fuse the information of the two modalities.

[0059] Based on the above embodiment, fault diagnosis is performed based on the fusion features to obtain the engine fault diagnosis result, including: Perform fault diagnosis based on fusion features to obtain engine fault keywords; Determine the engine fault diagnosis result based on fault keywords and fault knowledge graph; Among them, the fault knowledge graph contains multiple keywords and their corresponding fault information, and the fault information includes at least one of the fault component, fault type, fault cause and fault solution.

[0060] Specifically, the process of performing fault diagnosis according to the fusion features to obtain the engine fault diagnosis result may include: Considering the completeness of fault diagnosis and fault root cause analysis, in an embodiment of the present invention, a fault knowledge graph is introduced during fault diagnosis to identify engine faults through entities (keywords and fault information) represented by triples in the fault knowledge graph and the relationships between entities, thereby obtaining engine fault diagnosis results.

[0061] In detail, here we can first use the fusion features obtained by deep fusion to perform fault diagnosis and obtain a preliminary diagnosis result, that is, preliminary fault information, which is called fault keywords here; then, based on this fault keyword, we can use the fault knowledge graph to perform a final diagnosis to correspond the fault keyword with the keyword in the fault knowledge graph, and determine the engine fault diagnosis result based on the fault information associated with this keyword.

[0062] Among them, the fault information includes any one or more of the faulty components, fault types, fault causes and fault solutions. Correspondingly, the fault diagnosis results of the engine determined based on this may also include any one or more of the faulty components, fault causes, fault types and solutions.

[0063] It is worth noting that after obtaining the engine fault diagnosis results, the system can also perform active learning and personalized diagnosis. That is, based on the fault diagnosis results, the system can actively ask the driver for more specific information, such as the frequency of sound occurrence, the driving status of the vehicle to be diagnosed, etc., so as to continuously improve the fault information through interactive learning, thereby improving the accuracy of diagnosis. The system can also establish a personalized fault diagnosis model based on the driver's driving habits and vehicle historical data to more accurately identify specific faults of individual vehicles and provide more accurate maintenance suggestions.

[0064] In addition, a case cloud platform can also be established to achieve system updates and knowledge sharing. That is, a cloud platform can be established to collect the voice data of drivers, the audio data of engines, and the fault diagnosis results of engines on different vehicles to be diagnosed, and these data can be used to continuously optimize and update the system, so that the system can continuously learn new fault modes, thereby improving the generalization ability of fault diagnosis. In addition to data collection and system updates, the cloud platform can also achieve fault diagnosis knowledge sharing, so that different maintenance personnel can refer to the diagnosis cases of other vehicles to improve maintenance efficiency.

[0065] Figure 2 The overall flow chart of the engine fault diagnosis method provided by the present invention is as follows: Figure 2 As shown, the method comprises the following steps: First, the audio data of the engine on the vehicle to be diagnosed and the voice data of the driver are obtained.

[0066] Next, information is extracted from the transcribed text corresponding to the voice data to obtain keywords and semantic information.

[0067] Subsequently, the keywords and semantic information are used as references to guide feature extraction of audio data to obtain audio features.

[0068] Here, keywords and semantic information are used as references to guide feature extraction of audio data to obtain audio features, including: extracting features from audio data to obtain features of each audio segment in the audio data; and based on keywords and semantic information, fusing features of each audio segment to obtain audio features.

[0069] Among them, based on keywords and semantic information, the features of each audio clip are fused to obtain audio features, including: based on keywords and semantic information, determining the weight of each audio clip in the audio data; based on the weight of each audio clip in the audio data, fusing the features of each audio clip to obtain audio features.

[0070] Afterwards, fault diagnosis is performed based on the audio features to obtain the engine fault diagnosis result.

[0071] Here, fault diagnosis is performed based on audio features to obtain engine fault diagnosis results, including: deep fusion of audio features and semantic features of transcribed text to obtain fusion features; fault diagnosis is performed based on the fusion features to obtain engine fault diagnosis results.

[0072] Among them, the audio features and the semantic features of the transcribed text are deeply fused to obtain fused features, including: feature mapping of the audio features and the semantic features of the transcribed text to obtain audio embedding and semantic embedding in the same feature space; deep fusion of the audio embedding and the semantic embedding to obtain fused features.

[0073] The audio embedding and the semantic embedding are deeply fused to obtain fused features, including: determining a cross dimension based on the correlation between the audio embedding and the semantic embedding; and performing feature crossover on the audio embedding and the semantic embedding based on the cross dimension to obtain fused features.

[0074] Performing fault diagnosis based on fused features to obtain engine fault diagnosis results includes: performing fault diagnosis based on fused features to obtain engine fault keywords; determining engine fault diagnosis results based on the fault keywords and a fault knowledge graph; wherein the fault knowledge graph contains multiple keywords and their corresponding fault information, and the fault information includes at least one of a fault component, a fault type, a fault cause, and a fault solution.

[0075] The method provided by the embodiment of the present invention extracts information from the transcribed text corresponding to the driver's voice data to obtain keywords and semantic information; uses the keywords and semantic information as a reference to guide the engine's audio data to perform feature extraction, so that the feature extraction process pays more attention to audio clips related to the fault, reduces irrelevant information, and obtains accurate and effective audio features; performs fault diagnosis based on the audio features to obtain the engine's fault diagnosis results, thereby achieving comprehensive and accurate engine fault diagnosis, improving diagnostic efficiency, broadening the diagnostic scope, reducing dependence on professional equipment and personnel, and enabling the vehicle to perform self-diagnosis more conveniently, thereby reducing diagnostic costs and improving driving safety.

[0076] The engine fault diagnosis device provided by the present invention is described below. The engine fault diagnosis device described below and the engine fault diagnosis method described above can be referred to each other.

[0077] Figure 3 Schematic diagram of the structure of the engine fault diagnosis device provided by the present invention. Figure 3 As shown, the device comprises: The data acquisition unit 310 is used to acquire the audio data of the engine on the vehicle to be diagnosed and the voice data of the driver; An information extraction unit 320 is used to extract information from the transcribed text corresponding to the voice data to obtain keywords and semantic information; A feature extraction unit 330, configured to extract features from the audio data using the keywords and the semantic information as references to obtain audio features; The fault diagnosis unit 340 is used to perform fault diagnosis based on the audio feature to obtain a fault diagnosis result of the engine.

[0078] The engine fault diagnosis device provided by the present invention extracts information from the transcribed text corresponding to the driver's voice data to obtain keywords and semantic information; uses the keywords and semantic information as a reference to guide the engine's audio data to perform feature extraction, so that the feature extraction process pays more attention to audio clips related to the fault, reduces irrelevant information, and obtains accurate and effective audio features; performs fault diagnosis based on the audio features to obtain the engine's fault diagnosis results, thereby achieving comprehensive and accurate engine fault diagnosis, improving diagnostic efficiency, broadening the diagnostic scope, reducing dependence on professional equipment and personnel, and enabling vehicles to perform self-diagnosis more conveniently, thereby reducing diagnostic costs and improving driving safety.

[0079] Based on the above embodiment, the feature extraction unit 330 is used to: Extracting features from the audio data to obtain features of each audio segment in the audio data; Based on the keywords and the semantic information, the features of the audio clips are fused to obtain audio features.

[0080] Based on the above embodiment, the feature extraction unit 330 is used to: Determining a weight of each audio segment in the audio data based on the keyword and the semantic information; Based on the weight of each audio segment in the audio data, feature fusion is performed on the features of each audio segment to obtain audio features.

[0081] Based on the above embodiment, the fault diagnosis unit 340 is used to: Deeply fusing the audio features and the semantic features of the transcribed text to obtain fused features; Fault diagnosis is performed based on the fusion features to obtain a fault diagnosis result of the engine.

[0082] Based on the above embodiment, the fault diagnosis unit 340 is used to: Performing feature mapping on the audio features and the semantic features of the transcribed text to obtain audio embedding and semantic embedding in the same feature space; The audio embedding and the semantic embedding are deeply fused to obtain a fusion feature.

[0083] Based on the above embodiment, the fault diagnosis unit 340 is used to: Determining a cross dimension based on a correlation between the audio embedding and the semantic embedding; Based on the cross dimension, feature crossover is performed on the audio embedding and the semantic embedding to obtain a fusion feature.

[0084] Based on the above embodiment, the fault diagnosis unit 340 is used to: Perform fault diagnosis based on the fusion features to obtain fault keywords of the engine; Determine a fault diagnosis result of the engine based on the fault keywords and the fault knowledge graph; The fault knowledge graph includes multiple keywords and their corresponding fault information, and the fault information includes at least one of a fault component, a fault type, a fault cause, and a fault solution.

[0085] Figure 4 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 4 As shown, the electronic device may include: a processor 410, a communications interface 420, a memory 430 and a communication bus 440, wherein the processor 410, the communications interface 420 and the memory 430 communicate with each other through the communication bus 440. The processor 410 may call the logic instructions in the memory 430 to execute the engine fault diagnosis method, which includes: obtaining the audio data of the engine on the vehicle to be diagnosed and the voice data of the driver; extracting information from the transcribed text corresponding to the voice data to obtain keywords and semantic information; using the keywords and the semantic information as references, guiding the audio data to extract features to obtain audio features; performing fault diagnosis based on the audio features to obtain the fault diagnosis result of the engine.

[0086] In addition, the logic instructions in the above-mentioned memory 430 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0087] On the other hand, the present invention also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute the engine fault diagnosis method provided by the above-mentioned methods, and the method includes: obtaining audio data of the engine on the vehicle to be diagnosed, and voice data of the driver; extracting information from the transcribed text corresponding to the voice data to obtain keywords and semantic information; using the keywords and the semantic information as a reference to guide feature extraction of the audio data to obtain audio features; performing fault diagnosis based on the audio features to obtain fault diagnosis results of the engine.

[0088] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the engine fault diagnosis method provided by the above-mentioned methods, the method comprising: obtaining audio data of the engine on the vehicle to be diagnosed, and voice data of the driver; extracting information from the transcribed text corresponding to the voice data to obtain keywords and semantic information; using the keywords and the semantic information as a reference to guide feature extraction of the audio data to obtain audio features; performing fault diagnosis based on the audio features to obtain fault diagnosis results of the engine.

[0089] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0090] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0091] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An engine fault diagnosis method, characterized in that: include: Acquire audio data of the engine on the vehicle to be diagnosed and voice data of the driver; Extracting information from the transcribed text corresponding to the speech data to obtain keywords and semantic information; Taking the keywords and the semantic information as reference, guiding the audio data to perform feature extraction to obtain audio features; Fault diagnosis is performed based on the audio features to obtain a fault diagnosis result of the engine.

2. The engine fault diagnosis method according to claim 1, characterized in that: The step of taking the keywords and the semantic information as references to guide the audio data to perform feature extraction to obtain audio features includes: Extracting features from the audio data to obtain features of each audio segment in the audio data; Based on the keywords and the semantic information, the features of the audio clips are fused to obtain audio features.

3. The engine fault diagnosis method according to claim 2, characterized in that: The step of fusing the features of the audio clips based on the keywords and the semantic information to obtain audio features includes: Determining a weight of each audio segment in the audio data based on the keyword and the semantic information; Based on the weight of each audio segment in the audio data, feature fusion is performed on the features of each audio segment to obtain audio features.

4. The engine fault diagnosis method according to any one of claims 1 to 3, characterized in that: The performing fault diagnosis based on the audio feature to obtain the fault diagnosis result of the engine includes: Deeply fusing the audio features and the semantic features of the transcribed text to obtain fused features; Fault diagnosis is performed based on the fusion features to obtain a fault diagnosis result of the engine.

5. The engine fault diagnosis method according to claim 4, characterized in that: The deep fusion of the audio features and the semantic features of the transcribed text to obtain fusion features includes: Performing feature mapping on the audio features and the semantic features of the transcribed text to obtain audio embedding and semantic embedding in the same feature space; The audio embedding and the semantic embedding are deeply fused to obtain a fusion feature.

6. The engine fault diagnosis method according to claim 5, characterized in that: The deep fusion of the audio embedding and the semantic embedding to obtain a fusion feature includes: Determining a cross dimension based on a correlation between the audio embedding and the semantic embedding; Based on the cross dimension, feature crossover is performed on the audio embedding and the semantic embedding to obtain a fusion feature.

7. The engine fault diagnosis method according to claim 4, characterized in that: The performing fault diagnosis based on the fusion feature to obtain the fault diagnosis result of the engine includes: Perform fault diagnosis based on the fusion features to obtain fault keywords of the engine; Determine a fault diagnosis result of the engine based on the fault keywords and the fault knowledge graph; The fault knowledge graph includes multiple keywords and their corresponding fault information, and the fault information includes at least one of a fault component, a fault type, a fault cause, and a fault solution.

8. An engine fault diagnosis device, characterized in that: include: A data acquisition unit, used to acquire audio data of the engine on the vehicle to be diagnosed and voice data of the driver; An information extraction unit, used to extract information from the transcribed text corresponding to the voice data to obtain keywords and semantic information; A feature extraction unit, used to guide the audio data to perform feature extraction based on the keywords and the semantic information to obtain audio features; A fault diagnosis unit is used to perform fault diagnosis based on the audio feature to obtain a fault diagnosis result of the engine.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the engine fault diagnosis method according to any one of claims 1 to 7 is implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the engine fault diagnosis method according to any one of claims 1 to 7 is implemented.

Images