An off-line equipment fault diagnosis assistance method, system and electronic device
By using multimodal data acquisition and processing, combined with fault tree analysis and knowledge graphs, the problems of low efficiency, poor accuracy, and data security in fault diagnosis of complex equipment have been solved, achieving efficient and accurate fault diagnosis in a network-free environment.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- BEIJING HOLARDATA TECH CO LTD
- Filing Date
- 2026-03-30
- Publication Date
- 2026-06-26
AI Technical Summary
Existing equipment fault diagnosis methods are inefficient and inaccurate in complex systems, cannot adapt to multi-factor fault scenarios, and cannot reliably diagnose faults in environments without or with limited networks, posing a risk of data leakage.
A multimodal fault data acquisition and processing method is adopted. By recognizing and extracting features from voice, text and image data, a comprehensive feature vector is generated. Fault tree analysis and knowledge graph are combined to perform fault reasoning and generate fault diagnosis results.
It enables efficient and accurate fault diagnosis in offline environments, adapts to complex equipment and multi-factor scenarios, ensures data security, and improves the flexibility and reliability of fault diagnosis.
Smart Images
Figure CN122286574A_ABST
Abstract
Description
Technical Field
[0001] This application belongs to the field of artificial intelligence and equipment support technology, and in particular relates to a fault diagnosis auxiliary method, system and electronic device for offline equipment. Background Technology
[0002] With the rapid development of industrial automation and artificial intelligence technologies, equipment fault diagnosis and maintenance have become core links in ensuring industrial production safety and improving production efficiency.
[0003] Currently, in complex systems such as power, aviation, and shipbuilding, existing equipment fault diagnosis methods mainly rely on manual manual review, traditional expert systems, cloud-based AI diagnostic systems, and general-purpose large models. However, all of these methods have significant technical shortcomings and are difficult to meet actual operational needs: manual manual review is highly dependent on the professional skills of maintenance personnel, resulting in low fault diagnosis efficiency and susceptibility to missed or incorrect fault detections due to human error, making it unable to cope with multi-factor fault scenarios of complex equipment; traditional expert systems rely on fixed rules for fault reasoning, resulting in poor system flexibility and cumbersome knowledge base updates and maintenance processes, making it difficult to adapt to the iteration of equipment technology and the diversification of fault types; while cloud-based AI diagnostic systems have certain intelligent diagnostic capabilities, their operation is highly dependent on the network environment, making reliable diagnosis impossible in scenarios with no or limited network access, such as field operations or confidential operations, and there is also a risk of leakage of equipment operation data and fault data transmitted to the cloud; while general-purpose large models have powerful semantic understanding and generation capabilities, their large size makes them difficult to deploy on resource-constrained edge computing devices, and they suffer from model illusion problems, lacking specialized domain knowledge for equipment fault diagnosis, thus compromising the accuracy and reliability of diagnostic results. Summary of the Invention
[0004] The technical problem to be solved by this application is to provide a fault diagnosis auxiliary method, system and electronic device for offline equipment, so as to solve the problems mentioned in the background art.
[0005] To address the aforementioned technical problems, this application provides the following technical solution: Firstly, this application provides a fault diagnosis auxiliary method for offline equipment, including: Acquire raw multimodal fault data of the target offline equipment, wherein the raw multimodal fault data includes voice fault data, text fault data and image fault data; Content recognition and feature extraction are performed on the original multimodal fault data respectively, and the results of content recognition and feature extraction are mapped to a unified public semantic space to generate a comprehensive feature vector representing the current fault state. The preset fault tree analysis model inputs the comprehensive feature vector and outputs a set of candidate fault paths with high confidence. The set of candidate fault paths includes a sequence of suspected fault nodes and their topological connection relationships. The candidate fault path set is input into the retrieval enhancement generation model, and a hybrid semantic retrieval is performed in the local multimodal knowledge base to obtain historical maintenance cases and technical specifications that match the candidate fault paths; Based on the historical maintenance cases and technical specifications, the reasoning results of the fault tree analysis model are corrected and interpreted through a knowledge graph constraint decoding strategy, generating fault diagnosis results that include fault root cause location and maintenance suggestions.
[0006] Furthermore, acquiring the raw multimodal fault data of the target offline equipment includes: Only the voice fault data recorded by the user through the voice input device is acquired; Alternatively, it may simply acquire the text error data entered by the user through the text input interface; Alternatively, only the image fault data uploaded by the user through the image acquisition device can be obtained; Or it can simultaneously receive input of two or a combination of three types of data: voice fault data, text fault data, and image fault data. When only a single modality of data is received, the processing steps for other non-input modalities are skipped, and subsequent content recognition and feature extraction are performed directly on the currently received modality of data.
[0007] Furthermore, the content recognition and feature extraction of the original multimodal fault data specifically include: When the input is the voice fault data, an automatic speech recognition model is used to convert it into fault description text, and acoustic emotion features are extracted and fused to generate a speech semantic vector. When the input is the image fault data, a pre-trained visual encoder is used to identify equipment parts and defect areas in the image, extract visual texture features and part category labels, and generate visual semantic vectors. When the input is the text fault data, a natural language processing model is used to extract keywords, entity relationships, and fault intent to generate a text semantic vector.
[0008] The technical problem to be solved by this application is to provide a fault diagnosis auxiliary method, system and electronic device for offline equipment, so as to solve the problems mentioned in the background art.
[0009] To address the aforementioned technical problems, this application provides the following technical solution: Firstly, this application provides a fault diagnosis auxiliary method for offline equipment, including: Acquire raw multimodal fault data of the target offline equipment, wherein the raw multimodal fault data includes voice fault data, text fault data and image fault data; Content recognition and feature extraction are performed on the original multimodal fault data respectively, and the results of content recognition and feature extraction are mapped to a unified public semantic space to generate a comprehensive feature vector representing the current fault state. The preset fault tree analysis model inputs the comprehensive feature vector and outputs a set of candidate fault paths with high confidence. The set of candidate fault paths includes a sequence of suspected fault nodes and their topological connection relationships. The candidate fault path set is input into the retrieval enhancement generation model, and a hybrid semantic retrieval is performed in the local multimodal knowledge base to obtain historical maintenance cases and technical specifications that match the candidate fault paths; Based on the historical maintenance cases and technical specifications, the reasoning results of the fault tree analysis model are corrected and interpreted through a knowledge graph constraint decoding strategy, generating fault diagnosis results that include fault root cause location and maintenance suggestions.
[0010] Furthermore, acquiring the raw multimodal fault data of the target offline equipment includes: Only the voice fault data recorded by the user through the voice input device is acquired; Alternatively, it may simply acquire the text error data entered by the user through the text input interface; Alternatively, only the image fault data uploaded by the user through the image acquisition device can be obtained; Or it can simultaneously receive input of two or a combination of three types of data: voice fault data, text fault data, and image fault data. When only a single modality of data is received, the processing steps for other non-input modalities are skipped, and subsequent content recognition and feature extraction are performed directly on the currently received modality of data.
[0011] Furthermore, the content recognition and feature extraction of the original multimodal fault data specifically include: When the input is the voice fault data, an automatic speech recognition model is used to convert it into fault description text, and acoustic emotion features are extracted and fused to generate a speech semantic vector. When the input is the image fault data, a pre-trained visual encoder is used to identify equipment parts and defect areas in the image, extract visual texture features and part category labels, and generate visual semantic vectors. When the input is the text fault data, a natural language processing model is used to extract keywords, entity relationships, and fault intent to generate a text semantic vector. Attached Figure Description
[0012] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0013] Figure 1 This is a flowchart illustrating a fault diagnosis assistance method for offline equipment in an embodiment of this application; Figure 2 This is a schematic diagram of the structure of an offline equipment fault diagnosis auxiliary system according to an embodiment of this application; Figure 3 This is a schematic diagram of the structure of a computer electronic device according to an embodiment of this application. Detailed Implementation
[0014] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0015] Please see Figure 1 The present application provides a fault diagnosis assistance method for offline equipment, which includes at least the following steps: S10. Obtain the original multimodal fault data of the target offline equipment, wherein the original multimodal fault data includes voice fault data, text fault data and image fault data.
[0016] In one embodiment, step S10 includes: Only the voice fault data recorded by the user through the voice input device is acquired; Only the text-based fault data entered by the user through the text input interface is retrieved; Only acquire the image fault data uploaded by the user through the image acquisition device; Or it can simultaneously receive input of two or a combination of three types of data: voice fault data, text fault data, and image fault data. When only a single modality of data is received, the processing steps for other non-input modalities are skipped, and subsequent content recognition and feature extraction are directly performed on the currently received modality of data. Specifically, in this embodiment, multimodal and multi-method fault data can be flexibly collected according to actual conditions, adapting to the diverse operating habits of on-site personnel. It also supports adaptive reception of single-modal / multimodal data, reducing invalid data processing steps and improving collection efficiency. The multimodal fault raw data includes voice fault data, text fault data, and image fault data. The collection method allows users to flexibly select according to the actual on-site situation, specifically: Single-modal data acquisition: It can acquire only voice fault data recorded by users through voice input devices such as microphones and voice recorders (such as verbal statements like "abnormal engine noise, sudden drop in speed"); it can acquire only text fault data recorded by users through terminal text input interfaces and fault reporting systems (such as text descriptions like "hydraulic pipeline leak, system pressure drops to 0.3MPa"); it can acquire only image fault data uploaded by users through image acquisition devices such as industrial cameras and handheld terminal cameras (such as photos and video frame screenshots of equipment faulty parts). Multimodal data combination acquisition: Supports simultaneous reception of combined inputs of two or three types of data (such as simultaneously uploading images of the fault location + text descriptions of the fault phenomena + verbal descriptions of the fault occurrence process). Adaptive process jump: When the system receives only single modal data, it will automatically skip all processing steps for other non-input modal data and directly perform subsequent content recognition and feature extraction operations on the currently received modal data, avoiding meaningless computing power consumption and adapting to the resource-constrained characteristics of edge devices.
[0017] In this step, the acquisition of data for each modality follows the offline storage specifications of edge devices. The raw data after acquisition is directly stored in the local data buffer without any external transmission, thus ensuring the security of equipment fault data from the source.
[0018] S20. Perform content recognition and feature extraction on the original multimodal fault data respectively, and map the results of content recognition and feature extraction to a unified public semantic space to generate a comprehensive feature vector representing the current fault state.
[0019] Specifically, this step is divided into two stages: "modal recognition and extraction" and "cross-modal semantic fusion". First, specific content recognition and feature extraction are performed for the characteristics of different modal data. Then, the distribution differences between modalities are eliminated through unified semantic space mapping, realizing the semantic alignment and fusion of multi-source features, solving the "heterogeneity" problem of multimodal data, and providing a unified feature input for subsequent inference.
[0020] In one embodiment, the modal identification process extracts the original multimodal fault data and performs content recognition and feature extraction on each, including: When the input is the voice fault data, an automatic speech recognition model is used to convert it into fault description text, and acoustic emotion features are extracted and fused to generate a speech semantic vector. When the input is the image fault data, a pre-trained visual encoder is used to identify equipment parts and defect areas in the image, extract visual texture features and part category labels, and generate visual semantic vectors. When the input is the text fault data, a natural language processing model is used to extract keywords, entity relationships, and fault intent to generate a text semantic vector.
[0021] Specifically, in this embodiment, customized models and strategies are used for content recognition and feature extraction based on the different characteristics of voice, text, and image data, ensuring that fault information of each modality is fully extracted. The specific implementation method is as follows: Voice fault data processing: An offline automatic speech recognition (ASR) model based on deep neural networks is used to convert voice fault data into standardized fault description text. At the same time, the acoustic emotional features of the voice signal are extracted by Mel-frequency cepstral coefficients (MFCC) (such as abrupt and low voice features can help represent the urgency of the fault). The text semantic features and acoustic emotional features are fused to generate a voice semantic vector with fixed dimensions, so as to realize the digital representation of voice fault information. Image fault data processing: A visual encoder (such as YOLO and ResNet series models) pre-trained in the field of equipment faults is used to perform target detection and feature extraction on image fault data. The core components of the equipment and fault defect areas (such as cracks, leaks, deformations, etc.) in the image are accurately identified. At the same time, the visual texture features (such as color difference, texture, shape) and component category labels of the image are extracted. The component features, defect features and label information are fused to generate a visual semantic vector with fixed dimensions, so as to realize the structured representation of image fault information. Text fault data processing: Natural language processing (NLP) models (such as BERT-CRF and RoBERTa) that integrate a vocabulary specific to the equipment fault domain are used to segment, identify entities, and extract relationships from the text fault data. Keywords, entity relationships (such as the causal relationship between "pipeline" and "leakage") and fault intent (such as description of fault phenomena, location of fault location, and explanation of fault consequences) are extracted from the fault description. Based on the above extraction results, a text semantic vector with fixed dimensions is generated to realize the semantic representation of text fault information.
[0022] In one embodiment, mapping the results of content recognition and feature extraction to a unified public semantic space to generate a comprehensive feature vector representing the current fault state includes: Construct a shared high-dimensional semantic projection layer; The speech semantic vector, visual semantic vector, or text semantic vector is respectively input into the shared high-dimensional semantic projection layer; The distribution differences between modalities are eliminated by projection transformation, and they are mapped to the comprehensive feature vector with consistent dimensions and semantic alignment.
[0023] Specifically, in this embodiment, since the feature representation methods of each modality data are different, direct splicing will lead to semantic misalignment and feature redundancy between modalities. Therefore, this application achieves seamless fusion of multimodal features by constructing a unified public semantic space. The specific steps are as follows: Construct a shared high-dimensional semantic projection layer for the equipment failure domain. This projection layer is pre-trained with massive multimodal equipment failure data and has the ability to map semantic vectors of different modalities to the same high-dimensional space. The extracted speech semantic vector, visual semantic vector, and text semantic vector (or the semantic vector of the corresponding modality when inputting a single modality) are respectively input into the shared high-dimensional semantic projection layer; The projection layer eliminates the distribution differences and semantic gaps between different modal features through linear / nonlinear projection transformation, mapping each modal semantic vector into feature vectors with consistent dimensions and semantic alignment. The mapped feature vectors are weighted and fused (the weights are dynamically allocated according to the richness of fault information in each modality) to generate a unique comprehensive feature vector that can fully characterize the current fault state of the equipment.
[0024] This integrated feature vector not only retains the core information of each modality of fault data, but also achieves the organic integration of cross-modal information, providing accurate and unified feature input for subsequent fault tree inference.
[0025] S30. Input the comprehensive feature vector into the preset fault tree analysis model and output a set of candidate fault paths with high confidence. The set of candidate fault paths includes a sequence of suspected fault nodes and their topological connection relationships.
[0026] In one embodiment, step S30 includes: S301. Calculate the similarity score between the comprehensive feature vector and the feature embedding of each basic event node in the fault tree; S302. Mark basic event nodes with similarity scores higher than a preset threshold as active, and initialize the occurrence probability of basic events to the similarity scores; S303. Based on the logic gate structure of the fault tree, the state probabilities of intermediate events and top events are passed and updated layer by layer from bottom to top. S304. Select several fault propagation paths that lead to the occurrence of the top event and have the highest cumulative probability, and use them as a set of candidate fault paths.
[0027] Specifically, in this embodiment, Fault Tree Analysis (FTA) is used as the core logical framework. The comprehensive feature vector is deeply integrated with the fault tree model. Intelligent reasoning of fault paths is achieved through feature matching, probability propagation, and path filtering. The output candidate fault path set includes a sequence of suspected fault nodes and their topological connections, providing clear logical guidance for subsequent knowledge retrieval. The fault tree analysis model is an offline fault tree model pre-constructed for the target equipment, including the top event (final equipment failure phenomenon), intermediate events (fault propagation links), basic events (fault root causes), and logical gate structures (AND gates, OR gates, XOR gates) between each event. Feature embedding vectors are pre-generated for each event node. The specific reasoning process is as follows: Feature matching and node activation: Calculate the cosine similarity score between the comprehensive feature vector and the feature embeddings of all basic event nodes in the fault tree. Mark basic event nodes with similarity scores higher than a preset threshold (e.g., 0.8) as active and initialize the occurrence probability of the basic event to the corresponding similarity score. Set the occurrence probability of inactive nodes to 0. Probability is propagated layer by layer from bottom to top: Based on the logic gate structure of the fault tree, starting from the activated basic event node, the state probabilities of intermediate events and the top event are calculated and updated layer by layer from bottom to top. Among them, the event probability of the AND gate is the product of the probabilities of all input nodes, and the event probability of the OR gate is 1 minus the product of the probabilities of all input nodes not occurring, ensuring that the probability propagation conforms to the physical propagation logic of equipment faults; High-confidence fault path screening: Based on the triggering result of the top event, trace all fault propagation paths that could lead to the occurrence of the top event, calculate the cumulative probability of each path (the product of the probabilities of occurrence of all nodes in the path), and screen out several paths with the highest cumulative probability (such as the top 10) as a set of high-confidence candidate fault paths. Each path contains a clear sequence of suspected fault nodes (such as "basic event - intermediate event - top event") and the topological connection relationship between nodes (such as logic gate type and propagation order).
[0028] It should be noted that, in order to improve the on-site adaptability of the candidate fault path set and avoid inference bias caused by fixed logic in the fault tree model, in one embodiment of this application, a dynamic weight adjustment mechanism is also introduced in the process of passing and updating the state probabilities of intermediate events and top events layer by layer: Based on the real-time operating parameters of the target offline equipment, the prior probabilities of each basic event node are weighted and corrected. If the comprehensive feature vector contains feature information indicating a specific working condition, the probability weight of the fault path that is strongly correlated with that working condition is increased to enhance the field adaptability of the candidate fault path set.
[0029] Specifically, firstly, the prior probabilities of each basic event node are weighted and corrected by combining the real-time operating condition parameters of the target offline equipment (such as operating speed, operating temperature, load pressure, running time, etc., which are collected in real time by the equipment's local sensors). For example, when the equipment is running under high load for a long time, the probability weight of basic events related to "component fatigue wear" is increased. If the integrated feature vector contains feature information indicating a specific operating condition (such as the text description "fault occurs during high load operation" or the image recognition "component discoloration due to high temperature"), then the probability weight of the fault path that is strongly correlated with the operating condition is further increased, while the probability of irrelevant fault paths is weakened, ensuring that the inferred candidate fault path is highly matched with the actual operating status of the equipment on site.
[0030] S40. Input the candidate fault path set into the retrieval enhancement generation model, perform hybrid semantic retrieval in the local multimodal knowledge base, and obtain historical maintenance cases and technical specifications that match the candidate fault paths.
[0031] Specifically, in one embodiment, step S40 includes: S401, First-way search: Based on the component names and fault mode keywords in the suspected fault node sequence, perform Boolean matching search in the local knowledge base to obtain the first candidate set; S402, Second-path retrieval: Using the semantic representation of the comprehensive feature vector or candidate fault path as the query vector, perform cosine similarity retrieval in the local vector database to obtain the second candidate set; S403. Merge and sort the first candidate set and the second candidate set, remove duplicates, and select the documents with the highest ranking as the historical maintenance cases and technical specifications.
[0032] Specifically, in this embodiment, a Retrieval Enhanced Generative (RAG) model combined with a local multimodal knowledge base provides real-world fault repair knowledge support for candidate fault paths, avoiding the problem of fault tree reasoning relying solely on logic and lacking practical case references. Simultaneously, a hybrid semantic retrieval strategy is employed to improve the matching accuracy between retrieval results and candidate fault paths. The local multimodal knowledge base is an offline knowledge base pre-built and deployed on edge devices, containing multi-dimensional information such as equipment fault repair cases, technical manuals, component parameters, fault handling specifications, and historical fault data. The knowledge base adopts a hybrid storage architecture of "relational database + vector database," supporting keyword retrieval and vector semantic retrieval. The retrieval process is a dual-path retrieval + fusion sorting method, specifically implemented as follows: First approach: Boolean matching retrieval: The sequence of suspected fault nodes in the candidate fault path set is parsed to extract the equipment component names and fault mode keywords (such as "engine-crankshaft-wear" and "hydraulic pipeline-seals-leakage"). Based on the above keywords, Boolean matching retrieval (such as exact matching, fuzzy matching, and combined matching) is performed in the relational database of the local knowledge base to quickly filter out documents containing relevant keywords and form the first candidate set. The second approach: Cosine similarity retrieval: The comprehensive feature vector representing the fault state or the semantic representation vector of the candidate fault path is used as the query vector. Cosine similarity retrieval is performed in the local vector database. The similarity between the query vector and the feature vectors of all documents in the knowledge base is calculated. Documents with similarity higher than a preset threshold are selected to form the second candidate set. Candidate set fusion and ranking: The first and second candidate sets are merged, duplicate document content is removed, and a multi-dimensional weighted scoring strategy (such as keyword matching degree accounting for 40% and vector similarity accounting for 60%) is used to comprehensively rank the merged documents. The top-ranked documents are selected as historical maintenance cases and technical specifications that highly match the candidate fault paths, including maintenance procedures matching fault phenomena, handling methods corresponding to the root causes of faults, replacement / maintenance specifications for relevant components, and precautions for similar faults.
[0033] The entire retrieval process in this step is completed locally on the edge device without relying on external networks, ensuring the real-time nature of the retrieval and data security. At the same time, by integrating dual-path retrieval, both retrieval speed and accuracy are taken into account.
[0034] S50. Based on the historical maintenance cases and technical specifications, the reasoning results of the fault tree analysis model are corrected and interpreted through a knowledge graph constraint decoding strategy to generate fault diagnosis results that include fault root cause location and maintenance suggestions.
[0035] In one embodiment of this application, step S50 includes: S501. Transform the topological connection relationships in the candidate fault path set into logical constraint rules, and construct the vocabulary whitelist and state transition restriction matrix during generation. S502. During the decoding process of generating fault diagnosis results, monitor the next candidate word output by the large language model in real time. S503. If the candidate word does not conform to the logical constraint rules, then the probability of the candidate word is set to zero or significantly reduced. S504. Set the large language model to sample only within the vocabulary space that conforms to the fault tree logic path, to ensure that the generated fault root cause localization and maintenance suggestions conform to physical logic.
[0036] Specifically, in this embodiment, a knowledge graph-constrained decoding strategy is used to combine the logical reasoning results of the fault tree with the actual knowledge of the knowledge base. This process corrects, verifies, and interprets the reasoning results while simultaneously constraining the generation process of the large language model. This ensures that the output fault diagnosis results conform to the physical logic of the equipment and actual maintenance specifications, ultimately generating standardized diagnostic results that include fault root cause location and maintenance suggestions. The knowledge graph is a pre-constructed equipment fault knowledge graph, containing entities such as equipment components, fault phenomena, fault root causes, maintenance methods, and technical specifications, as well as the causal, attribution, and resolution relationships between these entities. It is a structured representation of physical logic and knowledge rules. The core of the knowledge graph-constrained decoding strategy is to integrate the logical constraints of the fault tree and the entity constraints of the knowledge graph into the decoding process of the large language model, avoiding problems such as "logical errors, knowledge bias, and inconsistencies with reality" in the generated results. The specific implementation process is as follows: Construct logical constraint rules: Transform the topological connections (such as node sequences, logic gate structures, and propagation order) in the candidate fault path set into logical constraint rules that can be recognized by a large language model. At the same time, based on the equipment fault knowledge graph, construct a vocabulary whitelist (containing only words related to the current fault, such as equipment components, root causes of the fault, and maintenance methods) and a state transition constraint matrix (limiting reasonable connections between words, such as "seal" can be connected with "leakage" but cannot be connected with "ablation" which is not related). Real-time monitoring of the decoding process: The candidate fault paths from the fault tree reasoning, historical maintenance cases retrieved from the knowledge base, and technical specifications are taken as input and fed into a large language model to generate fault diagnosis results. During the decoding process, the next candidate word output by the model is monitored in real time. Candidate word logic filtering: Logical verification is performed on the detected candidate words. If the candidate word does not meet the preset logical constraint rules (such as exceeding the vocabulary whitelist or violating the state transition limit), the generation probability of the candidate word is set to zero or significantly reduced to avoid the model generating invalid or incorrect words. Sampling generation within constraints: The large language model is set to sample only within the lexical space that conforms to the logical path of the fault tree and the association relationship of the knowledge graph, to ensure that the content generated by the model strictly follows the physical logic of the equipment and the fault propagation law.
[0037] Through the above constraint decoding process, the large language model will correct the candidate fault paths output by the fault tree analysis model: eliminate invalid paths without actual knowledge support, verify the rationality of highly matched paths, quantify the probability of occurrence of the root cause of the fault, and combine historical maintenance cases and technical specifications to generate targeted maintenance suggestions (such as maintenance steps, required tools, consumable parameters, operating procedures, precautions, etc.) for the located root cause of the fault, and finally generate standardized, practical, and physically logical fault diagnosis results.
[0038] The fault diagnosis auxiliary method for offline equipment provided in this application has the following advantages: 1. Enables flexible acquisition and adaptive processing of multimodal data to meet diverse on-site operational needs. This application supports fault data acquisition methods that combine single-modal or multi-modal approaches, including voice, text, and image. It can automatically skip processing steps for non-input modal data based on the actual input modal data, which not only caters to the diverse operating habits of field personnel but also effectively reduces the ineffective computing power consumption of edge devices. It is suitable for the rapid data acquisition needs in field, classified, and other scenarios. At the same time, all modal data is stored and processed locally throughout the process, avoiding the risk of data transmission leakage from the source and ensuring the security of equipment fault data.
[0039] 2. Overcome the barriers of heterogeneity in multimodal data to achieve accurate and unified characterization of fault features. This application performs customized content recognition and feature extraction based on the characteristics of different modal fault data. Then, by constructing a shared high-dimensional semantic projection layer, the semantic vectors of each modality are mapped to a unified common semantic space, eliminating the distribution differences and semantic gaps between modalities. The generated comprehensive feature vector not only retains the core information of each modal fault data, but also realizes the organic integration of cross-modal information. This solves the problems of semantic misalignment and feature redundancy in traditional multimodal data fusion, provides accurate and unified feature input for subsequent fault reasoning, and improves the comprehensiveness and accuracy of fault feature representation.
[0040] 3. Improve the on-site adaptability and confidence of the reasoning results of fault tree reasoning. This application deeply integrates feature vectors with fault tree analysis models. It activates fault nodes through feature matching and realizes fault path reasoning through bottom-up probability propagation. It also introduces a dynamic weight adjustment mechanism based on the real-time operating conditions of the equipment. The probability of fault nodes and path weights are dynamically corrected according to the field operating parameters and fault feature information. This ensures that the reasoned candidate fault paths are highly matched with the actual operating state of the equipment, effectively avoiding the reasoning bias caused by the fixed logic of traditional fault tree models, and greatly improving the field adaptability and confidence of candidate fault paths.
[0041] 4. Integrates logical reasoning with practical knowledge support to avoid the limitations of single-reasoning approaches. This application combines a retrieval-enhanced generative model with a local multimodal knowledge base to perform a dual-path hybrid semantic retrieval of Boolean matching and cosine similarity on candidate fault paths. This rapidly obtains matching historical maintenance cases and technical specifications, providing the structured logical reasoning of fault trees with real maintenance knowledge and practical cases as support. This solves the problem that traditional fault tree reasoning relies solely on logic and lacks practical references. At the same time, the fusion and ranking of the dual-path retrieval balances retrieval speed and matching accuracy, ensuring that the retrieval results closely match the candidate fault paths, and providing a reliable knowledge basis for fault root cause localization.
[0042] 5. Constrain the logic for generating diagnostic results to completely avoid the model illusion problem and ensure the physical rationality and feasibility of the conclusions. The knowledge graph constraint decoding strategy designed in this application transforms the topological connections of the fault tree into logical constraint rules. It also combines the equipment fault knowledge graph to construct a vocabulary whitelist and a state transition constraint matrix. During the diagnostic result generation process, the output of the large language model is monitored and logically filtered in real time, limiting the model to sample and generate only within the vocabulary space that conforms to physical logic and knowledge rules. This effectively avoids the illusion problem that is prone to occur in general large models, ensuring that the generated fault root cause localization conforms to the equipment fault propagation law and that the maintenance suggestions match the actual maintenance specifications, so that the diagnostic results are logical, reasonable and feasible.
[0043] 6. The entire process is executed offline at the edge, balancing real-time diagnostics with resource adaptability. All models and knowledge bases in this application are pre-deployed in edge computing devices. The entire process of fault data collection, feature extraction, fault reasoning, knowledge retrieval, and result generation does not rely on cloud networks or external data interaction. This not only meets the fault diagnosis needs of remote or classified scenarios without network or with weak network, but also significantly reduces the latency of data transmission and cloud reasoning, improving the real-time performance of diagnosis. At the same time, the algorithm design of each stage takes into account the resource-constrained characteristics of edge devices. Through adaptive process jumps, lightweight feature extraction, and accurate retrieval, the computational power consumption is reduced, achieving efficient resource utilization of edge devices.
[0044] In summary, the method of this application integrates multimodal adaptation, cross-modal fusion, logical reasoning, knowledge retrieval, and logical constraints, realizing intelligent offline equipment fault diagnosis on edge devices throughout the entire process. It effectively improves the flexibility, accuracy, real-time performance, and feasibility of offline fault diagnosis, and can be widely applied to offline fault diagnosis and auxiliary maintenance of complex equipment in fields such as power, aviation, shipbuilding, and rail transportation. It has extremely high practical application value and promotion value.
[0045] Please see Figure 2 This application also provides an offline equipment fault diagnosis auxiliary system 200, including: The data acquisition module 201 is used to acquire the raw multimodal fault data of the target offline equipment, which includes voice fault data, text fault data and image fault data. The feature vector construction module 202 is used to perform content recognition and feature extraction on the original multimodal fault data, and map the results of content recognition and feature extraction to a unified public semantic space to generate a comprehensive feature vector representing the current fault state. The fault reasoning module 203 is used to input the comprehensive feature vector into a preset fault tree analysis model and output a set of candidate fault paths with high confidence. The set of candidate fault paths includes a sequence of suspected fault nodes and their topological connection relationships. The knowledge retrieval module 204 is used to input the candidate fault path set into the retrieval enhancement generation model, perform hybrid semantic retrieval in the local multimodal knowledge base, and obtain historical maintenance cases and technical specifications that match the candidate fault paths; The fault diagnosis module 205 is used to correct and interpret the reasoning results of the fault tree analysis model based on the historical maintenance cases and technical specifications, using a knowledge graph constraint decoding strategy, and generate fault diagnosis results that include fault root cause location and maintenance suggestions.
[0046] Please see Figure 3 This application also provides a computer electronic device 300, including a memory 503 and a processor 302. The memory 303 stores a computer program, and the processor 302 executes the computer program to implement the steps of the fault diagnosis auxiliary method for offline equipment described in any of the above claims.
[0047] Specifically, the computer electronic device 300 includes a transceiver 301, a bus interface, and a processor 302. The processor 302 is used to acquire multimodal fault raw data of the target offline equipment, including voice fault data, text fault data, and image fault data. It performs content recognition and feature extraction on the multimodal fault raw data, and maps the results of content recognition and feature extraction to a unified public semantic space to generate a comprehensive feature vector representing the current fault state. The comprehensive feature vector is input into a preset fault tree analysis model, which outputs a set of candidate fault paths with high confidence. The set of candidate fault paths includes a sequence of suspected fault nodes and their topological connections. The set of candidate fault paths is input into a retrieval enhancement generation model, which performs a hybrid semantic retrieval in a local multimodal knowledge base to obtain historical maintenance cases and technical specifications matching the candidate fault paths. Based on the historical maintenance cases and technical specifications, the reasoning results of the fault tree analysis model are corrected and interpreted using a knowledge graph constraint decoding strategy to generate a fault diagnosis result containing fault root cause location and maintenance suggestions.
[0048] In this embodiment of the application, the computer electronic device 300 further includes a memory 303. Figure 3 In this context, the bus architecture can include any number of interconnected buses and bridges, specifically linking various circuits of one or more processors 302 (represented by processor 302) and various circuits of memory 303 (represented by memory 303). The bus architecture can also link various other circuits such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and therefore will not be described further herein. The bus interface provides an interface. The transceiver 301 can be multiple elements, including transmitters and receivers, providing a unit for communicating with various other devices over a transmission medium. The processor 302 is responsible for managing the bus architecture and general processing, and the memory 303 can store data used by the processor 302 during operation.
[0049] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the fault diagnosis assistance method for offline equipment described above.
[0050] In this embodiment, the computer-readable storage medium can be a non-volatile storage medium or a volatile storage medium. For example, the computer storage medium may include, but is not limited to, various media capable of storing program code, such as a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.
[0051] In all examples shown and described herein, any specific values should be interpreted as merely exemplary and not as limitations; therefore, other examples of exemplary embodiments may have different values.
[0052] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.
[0053] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can also be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the flowcharts and block diagrams in the accompanying drawings show the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that, in alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagram and / or flowchart, and combinations of blocks in the block diagram and / or flowchart, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.
[0054] In addition, the functional modules or units in the various embodiments of this application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.
[0055] If the aforementioned functions are implemented as software functional modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a terminal device (which may be a smartphone, personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application.
[0056] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.
Claims
1. A fault diagnosis auxiliary method for offline equipment, characterized in that, include: Acquire raw multimodal fault data of the target offline equipment, wherein the raw multimodal fault data includes voice fault data, text fault data and image fault data; Content recognition and feature extraction are performed on the original multimodal fault data respectively, and the results of content recognition and feature extraction are mapped to a unified public semantic space to generate a comprehensive feature vector representing the current fault state. The preset fault tree analysis model inputs the comprehensive feature vector and outputs a set of candidate fault paths with high confidence. The set of candidate fault paths includes a sequence of suspected fault nodes and their topological connection relationships. The candidate fault path set is input into the retrieval enhancement generation model, and a hybrid semantic retrieval is performed in the local multimodal knowledge base to obtain historical maintenance cases and technical specifications that match the candidate fault paths; Based on the historical maintenance cases and technical specifications, the reasoning results of the fault tree analysis model are corrected and interpreted through a knowledge graph constraint decoding strategy, generating fault diagnosis results that include fault root cause location and maintenance suggestions.
2. The method according to claim 1, characterized in that, The acquisition of raw multimodal fault data of the target offline equipment includes: Only the voice fault data recorded by the user through the voice input device is acquired; Alternatively, it may simply acquire the text error data entered by the user through the text input interface; Alternatively, only the image fault data uploaded by the user through the image acquisition device can be obtained; Or it can simultaneously receive input of two or a combination of three types of data: voice fault data, text fault data, and image fault data. When only a single modality of data is received, the processing steps for other non-input modalities are skipped, and subsequent content recognition and feature extraction are performed directly on the currently received modality of data.
3. The method according to claim 2, characterized in that, The process of performing content recognition and feature extraction on the original multimodal fault data specifically includes: When the input is the voice fault data, an automatic speech recognition model is used to convert it into fault description text, and acoustic emotion features are extracted and fused to generate a speech semantic vector. When the input is the image fault data, a pre-trained visual encoder is used to identify equipment parts and defect areas in the image, extract visual texture features and part category labels, and generate visual semantic vectors. When the input is the text fault data, a natural language processing model is used to extract keywords, entity relationships, and fault intent to generate a text semantic vector.
4. The method according to claim 3, characterized in that, The process of mapping the results of content recognition and feature extraction to a unified public semantic space to generate a comprehensive feature vector representing the current fault state includes: Construct a shared high-dimensional semantic projection layer; The speech semantic vector, visual semantic vector, or text semantic vector is respectively input into the shared high-dimensional semantic projection layer; The distribution differences between modalities are eliminated by projection transformation, and they are mapped to the comprehensive feature vector with consistent dimensions and semantic alignment.
5. The method according to claim 1, characterized in that, The preset fault tree analysis model, which inputs the comprehensive feature vector, outputs a set of candidate fault paths with high confidence, including: Calculate the similarity score between the comprehensive feature vector and the feature embeddings of each basic event node in the fault tree; Basic event nodes with similarity scores higher than a preset threshold are marked as active, and the occurrence probability of the basic event is initialized to the similarity score; Based on the logic gate structure of the fault tree, the state probabilities of intermediate events and top events are passed and updated layer by layer from bottom to top; Several fault propagation paths that lead to the occurrence of the top event and have the highest cumulative probability are selected as a set of candidate fault paths.
6. The method according to claim 5, characterized in that, The method further includes a dynamic weight adjustment mechanism introduced during the process of passing and updating the state probabilities of intermediate and top events layer by layer. Based on the real-time operating parameters of the target offline equipment, the prior probabilities of each basic event node are weighted and corrected. If the comprehensive feature vector contains feature information indicating a specific working condition, the probability weight of the fault path that is strongly correlated with that working condition is increased to enhance the field adaptability of the candidate fault path set.
7. The method according to claim 1, characterized in that, The hybrid semantic retrieval in the local multimodal knowledge base includes: First-path retrieval: Based on the component names and fault mode keywords in the suspected fault node sequence, Boolean matching retrieval is performed in the local knowledge base to obtain the first candidate set; Second-path retrieval: Using the semantic representation of the comprehensive feature vector or candidate fault path as the query vector, cosine similarity retrieval is performed in the local vector database to obtain the second candidate set; The first and second candidate sets are merged and sorted, duplicates are removed, and the documents with the highest ranking are selected as the historical maintenance cases and technical specifications.
8. The method according to claim 1, characterized in that, The step of correcting and interpreting the reasoning results of the fault tree analysis model using a knowledge graph-constrained decoding strategy includes: The topological connection relationships in the candidate fault path set are transformed into logical constraint rules, and a vocabulary whitelist and state transition restriction matrix are constructed during generation. During the decoding process of generating fault diagnosis results, the next candidate word output by the large language model is monitored in real time; If the candidate word does not conform to the logical constraint rules, then the probability of the candidate word is set to zero or significantly reduced. The large language model is configured to sample only within the lexical space that conforms to the logical path of the fault tree, ensuring that the generated root cause localization and repair suggestions conform to the physical logic.
9. A fault diagnosis auxiliary system for offline equipment, characterized in that, include: The data acquisition module is used to acquire the raw multimodal fault data of the target offline equipment, which includes voice fault data, text fault data and image fault data. The feature vector construction module is used to perform content recognition and feature extraction on the original multimodal fault data, and map the results of content recognition and feature extraction to a unified public semantic space to generate a comprehensive feature vector representing the current fault state. The fault reasoning module is used to input the comprehensive feature vector into a preset fault tree analysis model and output a set of candidate fault paths with high confidence. The set of candidate fault paths includes a sequence of suspected fault nodes and their topological connection relationships. The knowledge retrieval module is used to input the candidate fault path set into the retrieval enhancement generation model, perform hybrid semantic retrieval in the local multimodal knowledge base, and obtain historical maintenance cases and technical specifications that match the candidate fault paths; The fault diagnosis module is used to correct and interpret the reasoning results of the fault tree analysis model based on the historical maintenance cases and technical specifications, using a knowledge graph constraint decoding strategy, and generate fault diagnosis results that include fault root cause location and maintenance suggestions.
10. A computer electronic device, characterized in that, It includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the steps of the fault diagnosis assistance method for offline equipment as described in any one of claims 1-8.