Fault diagnosis methods, devices, media and electronic equipment

Abstract

This disclosure belongs to the field of fault diagnosis technology and relates to a fault diagnosis method, device, medium, and electronic device. The method includes: acquiring multimodal data of an electro-hydrogen coupling device using multiple sensors, and constructing a knowledge graph using the multimodal data. The knowledge graph consists of fault entities, attributes of fault entities, and relationships between fault entities; acquiring the data to be detected from the electro-hydrogen coupling device and inputting the data to be detected into a pre-trained fault detection model, so that the fault detection model outputs a fault feature vector; performing semantic matching and reasoning between the fault feature vector and the knowledge graph to obtain a text feature vector, and generating a fault diagnosis result for the data to be detected based on the text feature vector. This disclosure provides multi-dimensional and multi-level data support for more comprehensive and accurate fault diagnosis, not only eliminating the subjective dependence on empirical manual fault detection, but also having the advantages of adaptability and ease of implementation in various scenarios.

Description

Technical Field

[0001] This disclosure relates to the field of fault diagnosis technology, specifically to a fault diagnosis method, a fault diagnosis device, a non-transitory computer-readable storage medium, and an electronic device. Background Technology

[0002] As electro-hydrogen coupling devices become increasingly complex and automated, fault diagnosis of external equipment and core components becomes crucial. Fault diagnosis for electro-hydrogen coupling devices can be categorized into experience-based diagnosis and model-based detection, based on fault assessment strategies. While these traditional methods are applicable to most fault diagnosis needs across various fields, they struggle to address the diverse factors affecting the accuracy and stability of fault diagnosis from different angles, domains, or levels.

[0003] Human intervention relies on experienced operators to diagnose equipment faults, but this is subject to subjectivity, as different operators may arrive at different conclusions for the same fault. On the other hand, traditional model-based methods often depend on single-modal information for judgment, limiting the comprehensive utilization of contextual information. Furthermore, these methods typically only detect known or common faults and cannot effectively handle unknown or sudden faults. In addition, traditional fault diagnosis methods usually require equipment shutdown for testing purposes, resulting in unnecessary downtime and production losses. Summary of the Invention

[0004] To overcome the problems existing in related technologies, this disclosure provides a fault diagnosis method, a fault diagnosis device, a non-transitory computer-readable storage medium, and an electronic device.

[0005] According to a first aspect of the present disclosure, a fault diagnosis method is provided, the method comprising: Multimodal data of an electro-hydrogen coupling device is collected using multiple sensors, and a knowledge graph is constructed using the multimodal data. The knowledge graph consists of fault entities, the attributes of the fault entities, and the relationships between the fault entities. The test data of the electro-hydrogen coupling device is obtained and input into a pre-trained fault detection model so that the fault detection model outputs a fault feature vector. The fault feature vector is semantically matched and reasoned with the knowledge graph to obtain a text feature vector, and a fault diagnosis result of the data to be detected is generated based on the text feature vector.

[0006] Optionally, the method of acquiring multimodal data of the electro-hydrogen coupling device using multiple sensors includes: Multimodal data of the electro-hydrogen coupling device are collected using multiple sensors, and the multimodal data is preprocessed to obtain preprocessed multimodal data.

[0007] Optionally, constructing a knowledge graph using the multimodal data includes: Determine the encoder corresponding to the multimodal data, and use the encoder to extract the fault entities, the attributes of the fault entities, and the relationships between the fault entities from the multimodal data; A knowledge graph is obtained by constructing the faulty entities, their attributes, and the relationships between them into a graph structure.

[0008] Optionally, before inputting the data to be detected into the pre-trained fault detection model, the method further includes: The multimodal data is processed to extract data features, and the data features are used to train a fault detection model to obtain a pre-trained fault detection model.

[0009] Optionally, the step of semantically matching and reasoning with the knowledge graph to obtain a text feature vector includes: The similarity between the fault feature vector and the fault entities in the knowledge graph, the attributes of the fault entities, and the relationships between the fault entities is calculated to obtain the calculation result, and the text feature vector is determined based on the calculation result.

[0010] Optionally, determining the text feature vector based on the calculation result includes: When the calculation result is multiple sets of fault entities, attributes of the fault entities, and relationships between the fault entities in the knowledge graph that are similar to the fault feature vector, the multiple sets of fault entities, attributes of the fault entities, and relationships between the fault entities in the knowledge graph are sorted according to the calculation result to obtain a sorting result, and the text feature vector is determined according to the sorting result; or When the calculation result is multiple sets of fault entities, attributes of the fault entities, and relationships between the fault entities in the knowledge graph that are similar to the fault feature vector, multiple text feature vectors corresponding to the multiple sets of fault entities, attributes of the fault entities, and relationships between the fault entities are determined, and a text feature vector is determined from the multiple text feature vectors according to the calculation result.

[0011] Optionally, generating the fault diagnosis result of the data to be detected based on the text feature vector includes: The text feature vector is input into a text generation model so that the text generation model outputs the fault diagnosis result of the data to be detected. The fault diagnosis result includes one or more of mechanical faults, electrical faults, chemical faults and software faults.

[0012] According to a second aspect of the present disclosure, a fault diagnosis apparatus is provided, comprising: The knowledge graph construction module is configured to collect multimodal data of the electro-hydrogen coupling device using multiple sensors, and to construct a knowledge graph using the multimodal data. The knowledge graph consists of fault entities, the attributes of the fault entities, and the relationships between the fault entities. The vector generation module is configured to acquire the data to be detected from the electro-hydrogen coupling device and input the data to be detected into a pre-trained fault detection model so that the fault detection model outputs a fault feature vector. The fault diagnosis module is configured to perform semantic matching and reasoning between the fault feature vector and the knowledge graph to obtain a text feature vector, and generate a fault diagnosis result for the data to be detected based on the text feature vector.

[0013] According to a third aspect of the present disclosure, a non-transitory computer-readable storage medium is provided, having stored thereon computer program instructions that, when executed by a processor, implement the steps of the method described in any of the first aspects of the present disclosure.

[0014] According to a fourth aspect of the present disclosure, an electronic device is provided, comprising: processor; Memory used to store processor-executable instructions; The processor is configured to execute the executable instructions to implement the steps of any of the methods described in the first aspect of this disclosure.

[0015] The technical solutions provided by the embodiments of this disclosure may include the following beneficial effects: In the methods and apparatus provided in the exemplary embodiments of this disclosure, the present disclosure utilizes multiple sensors to collect multimodal data of the electro-hydrogen coupling device, which can fully utilize existing detection data and provide multi-dimensional, multi-angle, and multi-level data support for more comprehensive and accurate fault diagnosis. Furthermore, the fusion of multimodal data and knowledge graphs for fault diagnosis and knowledge reasoning not only eliminates the subjective dependence on empirical manual fault detection but also has the advantages of adaptability and ease of implementation in various scenarios. It also provides technical support for renewable energy hybrid hydrogen production systems, overcomes the difficulties and pain points of fault diagnosis technology in hybrid hydrogen production, and lays the foundation for the transformation from a new power system to a new energy system.

[0016] Other features and advantages of this disclosure will be described in detail in the following detailed description section.

[0017] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0018] The accompanying drawings are provided to further illustrate the present disclosure and form part of the specification. They are used together with the following detailed description to explain the present disclosure, but do not constitute a limitation thereof. In the drawings: Figure 1 The schematic diagram illustrates a flowchart of a fault diagnosis method according to an exemplary embodiment of the present disclosure; Figure 2 The illustration schematically shows a flowchart of a method for constructing a knowledge graph in an exemplary embodiment of this disclosure; Figure 3 The schematic diagram illustrates a flowchart of a method for determining text feature vectors in an exemplary embodiment of this disclosure; Figure 4 This schematically illustrates a system architecture diagram of a fault diagnosis method in an application scenario of an exemplary embodiment of this disclosure; Figure 5 The illustration shows a flowchart of a method for constructing a knowledge graph in an application scenario according to an exemplary embodiment of this disclosure; Figure 6 The schematic diagram illustrates the interface of the fault diagnosis method in an application scenario of an exemplary embodiment of this disclosure; Figure 7 The schematic diagram illustrates a flowchart of a fault diagnosis method in an application scenario of an exemplary embodiment of this disclosure; Figure 8 The schematic diagram illustrates the interface diagram of the fault diagnosis results in an application scenario of the exemplary embodiments of this disclosure; Figure 9 This schematic diagram illustrates the structure of a fault diagnosis device according to an exemplary embodiment of the present disclosure; Figure 10 The illustration schematically depicts an electronic device for implementing a fault diagnosis method according to an exemplary embodiment of the present disclosure; Figure 11 This illustration schematically shows another electronic device for implementing a fault diagnosis method in an exemplary embodiment of the present disclosure. Detailed Implementation

[0019] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0020] It should be noted that all actions involving the acquisition of signals, information, or data in this disclosure are carried out in compliance with the relevant data protection laws and policies of the country where the location is situated, and with authorization from the owner of the relevant device.

[0021] In related technologies, a method for fault diagnosis of electrolytic cells based on integrated depth autoencoder technology is provided.

[0022] First, a sample set is constructed by collecting production operation parameters under different conditions and labeling fault types. Then, multiple different deep autoencoder models are built, each employing a different activation function, with a classifier connected after the last hidden layer of each model. These models are then selected and trained using the sample set. During training, all models are integrated at both the feature and result levels to form a final trained ensemble model. This ensemble model is used for fault diagnosis to improve diagnostic accuracy.

[0023] This method only uses potential and temperature data, employs only single-mode information, does not consider information from other modes, and cannot diagnose multiple faults.

[0024] In addition, a diagnostic method and apparatus for aluminum electrolytic cells based on reinforcement learning are also provided.

[0025] First, the parameters of the electrolytic cell are used as samples and preprocessed. Second, a dynamic label propagation model is constructed to sample the data, and the tail samples are updated multiple times to improve the data imbalance. Then, a reinforcement learning algorithm is used to construct and train an aluminum electrolytic cell diagnostic model. Finally, the electrolytic cell parameters to be predicted are preprocessed and input into the trained diagnostic model to achieve the diagnosis of the aluminum electrolytic cell condition.

[0026] This method also does not use multimodal information, and the model used is relatively complex, but the fault diagnosis results are only 4 types, covering fewer working conditions.

[0027] Furthermore, it provides a diagnostic method and system for electrolytic cells based on curve fitting analysis and efficiency optimization.

[0028] The system comprises several components: an extraction unit for selecting a suitable operating range, a filtering unit, a curve fitting unit for performing linear or nonlinear regression, and a curve fitting quality analysis unit. The system also includes a curve fitting parameter characterization unit for classifying curve fitting parameters based on electrolyzer battery component technology and reference values, and a database for storing fitting coefficients and their characteristics.

[0029] This method only uses curve fitting to diagnose faults in electrolytic cells, without considering multiple operating conditions and the occurrence of multiple faults.

[0030] It is evident that, based on fault assessment strategies, fault diagnosis of electro-hydrogen coupling devices can be categorized into diagnosis based on human experience and detection based on models. While these traditional methods are applicable to most fault diagnosis needs across various fields, they struggle to address the diverse factors affecting the accuracy and stability of fault diagnosis from different angles, domains, or levels.

[0031] Manual intervention relies on experienced operators to diagnose equipment faults, but this is subject to subjectivity, as different operators may arrive at different conclusions for the same fault. On the other hand, traditional model-based methods often depend on single-modal information for judgment, limiting the comprehensive utilization of contextual information. Furthermore, these methods typically only detect known or common faults and cannot effectively handle unknown or sudden faults. In addition, traditional fault diagnosis methods usually require equipment shutdown for testing purposes, resulting in unnecessary downtime and production losses.

[0032] Existing equipment fault diagnosis methods lack effective integration of multimodal information and knowledge graphs. These methods either rely on a single modality to diagnose equipment faults or use knowledge graphs to only query and match answers from expert question-and-answer systems. These methods are unsuitable for complex electro-hydrogen coupling devices and often produce unsatisfactory fault diagnosis results due to limited data.

[0033] Existing technological research mainly involves fault diagnosis of single alkaline electrolysis systems or PEM (Proton Exchange Membrane) electrolysis devices, and only collects information from a single mode for fault diagnosis.

[0034] However, in terms of fault diagnosis for industrial systems, electro-hydrogen coupling systems are far more complex than single-type hydrogen production systems. They are not simply a superposition of individual systems, thus significantly increasing system complexity. Because hybrid systems incorporate multiple technologies and equipment, their safety parameters may involve multiple components and complex interactions. Fault diagnosis of the electro-hydrogen coupling device can ensure the stable and high-efficiency operation of each module within the system. Currently, commercially available single-mode fault diagnosis devices suffer from large detection delays, high diurnal fluctuations, and poor anti-interference capabilities under extreme environments, leading to problems such as misidentification of faults or difficulty in identifying the cause of the fault.

[0035] As the complexity and automation of electro-hydrogen coupling devices increase, fault diagnosis of external equipment and core components becomes crucial. Traditional fault diagnosis methods rely on single-modal information, such as text, vision, and audio, which suffer from problems such as complex models, high data requirements, and poor robustness. Furthermore, these methods typically can only detect known or common faults and cannot effectively handle unknown or sudden faults.

[0036] This disclosure provides a fault diagnosis method. Figure 1 This is a flowchart illustrating a fault diagnosis method according to an exemplary embodiment, such as... Figure 1 As shown, the method may include at least the following steps: Step S110. Collect multimodal data of the electro-hydrogen coupling device using multiple sensors, and construct a knowledge graph using the multimodal data. The knowledge graph consists of fault entities, the attributes of fault entities, and the relationships between fault entities.

[0037] Step S120. Obtain the data to be tested from the electro-hydrogen coupling device and input the data to be tested into the pre-trained fault detection model so that the fault detection model outputs a fault feature vector.

[0038] Step S130. Perform semantic matching and reasoning between the fault feature vector and the knowledge graph to obtain the text feature vector, and generate the fault diagnosis result of the data to be detected based on the text feature vector.

[0039] In the exemplary embodiments disclosed herein, multimodal data of the electro-hydrogen coupling device is collected using multiple sensors, which fully utilizes existing detection data to provide multi-dimensional, multi-angle, and multi-level data support for more comprehensive and accurate fault diagnosis. Furthermore, the fusion of multimodal data and knowledge graphs for fault diagnosis and knowledge reasoning not only eliminates the subjective dependence on empirical manual fault detection but also offers advantages in adaptability and ease of implementation across various scenarios. This solves the problem of losses caused by downtime during fault diagnosis, enhances fault diagnosis and fault tolerance capabilities, ensures the safe and efficient operation of the electro-hydrogen coupling device in complex environments, and provides technical support for renewable energy hybrid hydrogen production systems. It overcomes the difficulties and pain points of fault diagnosis technology in hybrid hydrogen production, laying the foundation for the transformation from a new power system to a new energy system.

[0040] The following section provides a detailed explanation of each step in the fault diagnosis method.

[0041] In step S110, multimodal data of the electro-hydrogen coupling device is collected using multiple sensors, and a knowledge graph is constructed using the multimodal data. The knowledge graph consists of fault entities, the attributes of fault entities, and the relationships between fault entities.

[0042] In exemplary embodiments of this disclosure, the electro-hydrogen coupling device improves the utilization efficiency of new energy sources by combining the production, storage and conversion of electrical energy and hydrogen, and is a key technology for promoting energy structure transformation.

[0043] This multimodal data can be acquired by a multimodal system parameter detection device to detect operating parameters at different locations in the electro-hydrogen coupling system. Specifically, the detection device may include current sensors, voltage sensors, temperature sensors, pressure sensors, flow sensors, pH (Pondus Hydrogenii) sensors, humidity sensors, gas concentration sensors, vibration sensors, image sensors, and acoustic sensors, etc.

[0044] In an optional embodiment, multimodal data of the electro-hydrogen coupling device is collected using multiple sensors, and the multimodal data is preprocessed to obtain preprocessed multimodal data.

[0045] The multimodal data may include electrolytic cell power, electrolytic cell heat generation, electrolytic cell voltage, reversible voltage of the electrolysis process, current density, device operating temperature, circulating pump power, water pump power, device operating pressure, fiber optic sensing data of safety parameters in the gas-liquid separator, and high-speed camera images of bubbles in the outlet pipe of the alkali tank or gas-liquid separator.

[0046] After collecting multimodal data, it can be preprocessed.

[0047] Generally, preprocessing methods can include cleaning, imbalance handling, decomposition, transformation and standardization of different types of data, as well as time synchronization, spatial processing, semantic fusion and interactive fusion.

[0048] In an optional embodiment, Figure 2 A flowchart illustrating the method for constructing a knowledge graph is shown, such as... Figure 2 As shown, the method may include at least the following steps: in step S210, an encoder corresponding to the multimodal data is determined, and the encoder is used to extract fault entities, attributes of fault entities, and relationships between fault entities from the multimodal data.

[0049] When performing representation learning for knowledge graphs, modal encoders are designed based on the input data. For example, Transformers are used to process text, CNNs (Convolutional Neural Networks) are used to process images, and deep learning models are used to process vibration signals.

[0050] Since entity recognition, attribute extraction, and relationship recognition and extraction are also key steps, after determining the encoder, the encoder can be used to identify entities, attributes and their relationships from multimodal data.

[0051] In step S220, the faulty entities, their attributes, and the relationships between them are constructed into a graph structure to obtain a knowledge graph.

[0052] Furthermore, the extracted faulty entities, their attributes, and the relationships between them are constructed into a graph structure to obtain a knowledge graph, which is then stored using a suitable storage system.

[0053] Among them, the fault entity can include equipment name, fault type, etc., the attributes of the fault entity can include equipment status, fault characteristics, etc., and the relationship between fault entities can include causal relationship, composition relationship, etc.

[0054] The updating and maintenance of knowledge graphs is an ongoing process, including dynamic updates and error correction. Ultimately, multimodal knowledge graphs can be applied to fields such as query retrieval, intelligent question answering, and recommendation systems. Representation learning methods for knowledge graphs include, but are not limited to, distance-based models, random walk-based models, semantic matching-based models, and graph neural network-based models.

[0055] In step S120, the data to be tested from the electro-hydrogen coupling device is acquired and input into a pre-trained fault detection model so that the fault detection model outputs a fault feature vector.

[0056] In an exemplary embodiment of this disclosure, an untrained fault detection model may be trained before the data to be detected is input into the trained fault detection model.

[0057] In an optional embodiment, feature extraction processing is performed on the multimodal data to obtain data features, and the data features are used to train a fault detection model to obtain a pre-trained fault detection model.

[0058] Feature engineering can extract the most useful features for model prediction from preprocessed multimodal data. Specifically, feature engineering methods include using statistical analysis, feature correlation analysis, feature importance analysis, time series analysis, or frequency analysis to highlight key information and reduce unnecessary data dimensionality.

[0059] Specifically, fault detection models can include machine learning models, deep learning models, generative adversarial models, autoregressive integrated mobile models, self-attention models, ensemble learning models, LSTM (Long Short-Term Memory) networks, autoencoders, and Transformers, etc.

[0060] After obtaining the data features, the fault detection model can be trained using these features to obtain a pre-trained fault detection model. Simultaneously, during training, techniques such as cross-validation, grid search, and Bayesian optimization are used to optimize the model's hyperparameters, thereby improving the model's accuracy and generalization ability.

[0061] Furthermore, after inputting the data to be detected into a pre-trained fault detection model, the model can output a fault feature vector of the data. This fault feature vector represents the probability value of different entities, attributes, and relationships.

[0062] In step S130, the fault feature vector is semantically matched and reasoned with the knowledge graph to obtain the text feature vector, and the fault diagnosis result of the data to be detected is generated based on the text feature vector.

[0063] In an exemplary embodiment of this disclosure, after obtaining the fault feature vector, the fault feature vector can be semantically matched and reasoned with the knowledge graph.

[0064] In an optional embodiment, similarity calculations are performed on the fault feature vector and the fault entities, attributes of the fault entities, and relationships between the fault entities in the knowledge graph to obtain the calculation results, and the text feature vector is determined based on the calculation results.

[0065] After obtaining the feature vectors of the faulty text representing the target scene, this information needs to be mapped to entities, attributes, and relationships stored in the knowledge graph. Cosine similarity is used to achieve this mapping. This is used to calculate the similarity between the feature vector of the faulty text and the content in the knowledge graph, so as to achieve one-to-one matching between entities, attributes, and relations, thereby determining the text feature vector of the faulty triple. and These represent the components of vector A and vector B, respectively.

[0066] In an optional embodiment, Figure 3 A flowchart illustrating the method for determining text feature vectors is shown, such as... Figure 3As shown, the method may include at least the following steps: In step S310, when the calculation result is multiple sets of fault entities, fault entity attributes and relationships between fault entities in knowledge graphs similar to fault feature vectors, the fault entities, fault entity attributes and relationships between fault entities in multiple sets of knowledge graphs are sorted according to the calculation result to obtain a sorting result, and the text feature vector is determined according to the sorting result.

[0067] When the calculation results can determine multiple sets of fault entities, their attributes, and relationships that are similar to the fault feature vector, the ranking results can be obtained by sorting the multiple sets of fault entities, their attributes, and relationships based on the similarity represented by the calculation results.

[0068] Furthermore, based on the sorting results, the group with the highest similarity can be determined from multiple groups of faulty entities, the attributes of faulty entities, and the relationships between faulty entities as the text feature vector.

[0069] In step S320, when the calculation result is multiple sets of fault entities, fault entity attributes and relationships between fault entities in knowledge graphs similar to fault feature vectors, multiple text feature vectors corresponding to fault entities, fault entity attributes and relationships between fault entities in multiple sets of knowledge graphs are determined, and a text feature vector is determined from multiple text feature vectors based on the calculation result.

[0070] When the calculation results can determine multiple sets of fault entities, their attributes, and relationships similar to the fault feature vectors, multiple text feature vectors corresponding to these fault entities, their attributes, and relationships can be obtained.

[0071] Furthermore, based on the similarity values ​​represented by the calculation results, the text feature vector with the highest similarity is selected as the final text feature vector.

[0072] In an optional embodiment, the text feature vector is input into the text generation model so that the text generation model outputs the fault diagnosis result of the data to be detected. The fault diagnosis result includes one or more of mechanical faults, electrical faults, chemical faults and software faults.

[0073] To enable text feature vectors to generate interpretable fault text, the text feature vectors can be input into a text generation model, which then outputs fault diagnosis results for the data to be detected. These text generation models can include statistical language models, such as N-grams (a language model), deep learning-based sequence-to-sequence models, such as Long Short-Term Memory networks and Gated Recurrent Units (GRUs), attention-based Transformers and their variants BERT (Bidirectional Encoder Representations from Transformers), GPT (Generative Pre-Trained Transformer, a deep learning model), and state-of-the-art multimodal fusion models, such as BEiT-3 (a general-purpose multimodal foundation model). These models learn language patterns and structures to automatically generate coherent and natural fault text content.

[0074] The fault diagnosis results of the data to be tested may include the time, location and phenomenon of the fault, wherein the phenomenon may include one or more of mechanical faults, electrical faults, chemical faults and software faults.

[0075] After determining the fault diagnosis results, control adjustments can be made based on the current operating status. The PEM can be actively adjusted, including changes to command parameters, control parameters, and fault protection parameters, while the alkaline electrolysis layer can be passively adjusted, including changes to command parameters and shutdown.

[0076] The parameter detection method in this embodiment will be described in detail below with reference to an application scenario.

[0077] In this application scenario, a fault diagnosis algorithm and system are proposed that utilizes real-time operational data from an electro-hydrogen coupling device for multimodal knowledge graph reasoning. The electro-hydrogen coupling device includes one or more systems such as electro-hydrogen storage, electro-hydrogen ammonia, and electro-hydrogen alcohol; the electrolyzer includes one or more types such as alkaline electrolysis, proton exchange membrane electrolysis, solid oxide electrolysis, and anion exchange membrane electrolysis. Fault diagnosis includes one or more combinations of chemical faults, mechanical faults, electrical faults, and software faults. Multimodal analysis includes, but is not limited to, the potential data of the electrolyzer's cathode and anode electrodes, fiber optic sensing data in the gas-liquid separator, high-speed camera images of bubbles in the outlet pipe of the alkaline tank or gas-liquid separator, and the device's operating pressure.

[0078] The fault diagnosis algorithm and system for multimodal knowledge graph reasoning includes a data acquisition and processing unit, feature engineering and multimodal information fusion, fault knowledge graph representation learning, multimodal fault detection and classification algorithm unit, multimodal knowledge graph reasoning, fault information generation, and algorithm deployment unit.

[0079] The data acquisition module acquires measurable parameters of the device through different types of sensors. The amount of measurable parameters acquired is not limited to electrolytic cell power, electrolytic cell heat generation, electrolytic cell voltage, reversible voltage of the electrolysis process, current density, device operating temperature, circulating pump power, water pump power, device operating pressure, fiber optic sensing data of safety parameters in the gas-liquid separator, and high-speed camera images of bubbles in the outlet pipe of the electrolytic cell or gas-liquid separator. The data processing module preprocesses the various types of data acquired, including cleaning, imbalance handling, decomposition, conversion, and standardization of different types of data.

[0080] The feature engineering and multimodal fusion unit includes feature engineering on the collected and processed historical data and corresponding feature fusion based on the constructed model. Feature engineering extracts the most useful features for model prediction from the processed historical data. This includes, but is not limited to, using statistical analysis methods, feature correlation analysis, feature importance analysis, time series analysis, or frequency analysis to highlight key information and reduce unnecessary data dimensionality. The multimodal fusion unit integrates information from different sensors and technologies. The integration process includes, but is not limited to, data-level integration, feature-level integration, model-level fusion, and decision-level integration.

[0081] The fault knowledge graph representation learning unit first performs representation learning of the knowledge graph, designing a modal encoder based on the input data, such as using a Transformer to process text, a CNN to process images, and a deep learning model to process vibration signals. Entity recognition, attribute extraction, and relationship recognition and extraction are also key steps, involving identifying entities, attributes, and their relationships from the data. Then, the extracted information is constructed into a graph structure and stored using a suitable storage system. Updating and maintaining the knowledge graph is also an ongoing process, including dynamic updates and error correction. Finally, multimodal knowledge graphs can be applied to query retrieval, intelligent question answering, and recommendation systems. Representation learning methods include, but are not limited to, distance-based models, random walk-based models, semantic matching-based models, and graph neural network-based models.

[0082] The multimodal fault detection and classification algorithm unit is a dynamic model, including fault detection and fault diagnosis algorithms. These algorithms include, but are not limited to, machine learning models, deep learning models, generative adversarial models, autoregressive integrated moving models, self-attention models, ensemble learning models, LSTM, Autoencoder, and Transformer, and are trained using processed data. During training, techniques such as cross-validation, grid search, and Bayesian optimization are used to optimize the model's hyperparameters to improve its accuracy and generalization ability. The output of this module is the fault feature vector of the current input data, where each fault feature vector represents the probability value of different entities, attributes, and relationships.

[0083] Multimodal knowledge graph reasoning utilizes multimodal fault detection and classification algorithms to extract feature vectors from input data entities to obtain fault feature vectors. After obtaining the textual feature vectors representing faults within the target scene, this information needs to be mapped to entities, attributes, and relationships stored in the knowledge base. To achieve this mapping, cosine similarity is used to calculate the similarity between entity information obtained from the visual model and content within the knowledge graph. This allows for one-to-one matching between entities, attributes, and relationships.

[0084] After retrieving query nodes from the knowledge graph, the fault information generation unit uses a text generation model to process the sorted text feature vectors and generate interpretable fault text. Text generation models include, but are not limited to, statistical language models such as N-grams, deep learning-based sequence-to-sequence models such as Long Short-Term Memory networks and gated recurrent units, attention-based Transformers and their variants BERT and GPT, and the latest multimodal fusion models such as BEiT-3. These models learn language patterns and structures to automatically generate coherent and natural fault text content.

[0085] The algorithm deployment unit ensures that the model can run stably in the target environment, which includes, but is not limited to, GPUs, TPUs, FPGAs, cloud computing, and embedded chips.

[0086] Taking the alkaline and PEM hybrid hydrogen production unit as an example, information such as port voltage, current, operating temperature, electrolyzer operating pressure, gas-liquid separator pressure, alkaline flow rate, and oxygen and hydrogen flow rate of the alkaline and PEM hybrid hydrogen production unit are collected respectively. The data processing module performs data preprocessing on the selected information, including filtering, modulation and demodulation, encoding and decoding, transform domain processing, cleaning, decomposition, conversion and standardization.

[0087] Next, the feature-engineered data is fused at the model level. The fused model is aligned according to time scale and space. The aligned fused features are then input into a multimodal classification model for training, enabling it to predict the fault feature vector of the device based on the input features.

[0088] The fault vector is input into a multimodal knowledge graph for inference. During the inference phase, cosine similarity is used to match the detected feature vector with entity and relation vectors in the knowledge graph attributes.

[0089] Finally, in the fault text generation stage, the BERT text generation model generates the final equipment fault diagnosis result based on the inference results of the multimodal knowledge graph.

[0090] Figure 4 The system architecture diagram of the fault diagnosis method in the application scenario is shown, such as... Figure 4 As shown, the system consists of multiple modules connected in parallel. Each module includes an alkaline hydrogen electrolyzer, a PEM hydrogen electrolyzer, a hydrogen production power supply, a cooling water subsystem, a gas-liquid separation subsystem, a drying subsystem, and multi-modal safety parameter detection equipment. Alkaline electrolysis hydrogen production and PEM electrolysis hydrogen production are combined according to different capacities and quantities. After hydrogen drying, they are coupled together by a gas pressure regulator and then sent to the hydrogen purification unit.

[0091] The hydrogen production power supply subsystem consists of several power electronic converter modules connected in parallel. The converter module topology can be a DC / DC converter (Direct Current to Direct Current Converter) with DC input or an AC / DC converter (Alternating Current to Direct Current Converter) with AC input. Each converter is connected to the corresponding alkaline hydrogen production electrolyzer and PEM electrolyzer. The cooling water subsystem controls the flow rate of the circulating pump to ensure that the temperature of the mixed hydrogen production electrolyzer operates within the normal range. The oxygen and hydrogen gas-liquid separation subsystem is used to separate the hydrogen and oxygen generated in the electrolyzer from the water vapor and electrolyte. At the same time, it controls the outlet pressure of hydrogen and oxygen, and controls the oxygen content in hydrogen and the hydrogen content in oxygen to ensure normal chemical reactions and system safety in the electrolyzer. The drying subsystem is mainly used to dry the hydrogen and reduce its humidity. The purification subsystem is mainly used to further purify the generated hydrogen to meet the production requirements.

[0092] This system collects data from multiple sources, including acoustic vibration, infrared, and spectral sensors. First, it uses historical data to learn and store a knowledge graph representation. Then, a multimodal fault detection and classification module learns fault feature vectors online from the current data. These feature vectors are input into the fault knowledge graph for reasoning, resulting in a set of fault triples. The fault information generation module learns the features of the fault triples and outputs detailed fault information. These modules work together to improve the system's safety and reliability. The system also includes fault-tolerant control functions to ensure normal operation and timely response.

[0093] Figure 5 The flowchart illustrates the method for constructing a knowledge graph in an application scenario, such as... Figure 5 As shown, a conceptual framework is constructed based on historical data, including the definition of fault attribute relationships and category hierarchy. This conceptual framework is then mapped to the construction of the data layer. Multimodal data is then collected, including text, images, and sound, and entity alignment, spatial alignment, and temporal alignment are used to ensure data integrity and consistency. Subsequently, entity information and relationships between entities are extracted from this multimodal data, refining the attributes of each entity and the relationships between them. Finally, this information is stored in a database for later use.

[0094] Figure 6 The diagram illustrates the interface of a fault diagnosis method in an application scenario, such as... Figure 6 As shown, real-time multimodal data is first collected from the electro-hydrogen coupling device, and then preprocessed to improve its quality. Next, a multimodal fault detection and classification model is trained using historical data, enabling it to predict the device's fault feature vector based on the input features. The fault vector is then input into a multimodal knowledge graph for inference. During the inference phase, cosine similarity is used to match the detected feature vectors with entity and relation vectors in the knowledge graph attributes. Finally, in the fault text generation phase, the BERT text generation model generates the final device fault diagnosis result based on the inference results from the multimodal knowledge graph. The BERT model employs a standard Transformer-based sequence-to-sequence structure, which combines a bidirectional Transformer encoder with a unidirectional autoregressive Transformer encoder. Pre-training involves denoising and reconstructing the input text.

[0095] Figure 7 A flowchart illustrating a fault diagnosis method in an application scenario is shown, such as... Figure 7 As shown, it first receives multimodal data such as sound, light, electricity and electricity at the current moment, processes and fuses them to ensure data consistency and availability.

[0096] Next, a multimodal fault detection and classification model is trained using the fused data. During training, the system continuously adjusts the network parameters based on newly acquired data to improve the model's accuracy. After multiple iterations and optimizations, a fault detection model with optimal performance is finally obtained.

[0097] Then, the fault feature vectors are input into the multimodal knowledge graph for inference. During the inference phase, cosine similarity is used to match the detected feature vectors with the entity and relation vectors in the knowledge graph attributes to obtain a set of fault triples.

[0098] Finally, in the fault text generation stage, the BERT text generation model generates the final equipment fault diagnosis result based on the inference results of the multimodal knowledge graph.

[0099] Figure 8 The diagram illustrates the interface of fault diagnosis results in an application scenario, such as... Figure 8 As shown, the fault diagnosis results include single faults and multiple faults. Specifically, detailed information on three equipment faults that occurred between July 24 and 25, 2024, is presented.

[0100] The first failure occurred at 14:00 on July 24, 2024, involving an abnormality in the cooling water of the hydrogen separator in module 4.

[0101] The second failure occurred at 11:38 on July 25, 2024, which was a power failure of the alkaline electrolyzer in module 2, and the hydrogen content in the oxygen at the outlet of the gas-liquid separator exceeded the standard.

[0102] The third failure also occurred at 15:22 on July 25, 2024, which was the failure of the alkaline solution circulation pump in the alkaline electrolysis cell of module 3 and the malfunction of the pneumatic diaphragm valve.

[0103] In the exemplary embodiments disclosed herein, unlike safety parameter detection and early warning, the focus is on what kind of failure has occurred in different components of the electro-hydrogen coupling device at the current moment, including but not limited to one or more of mechanical failures, electrical failures, chemical failures, and software failures.

[0104] The proposed method and system, applied in various scenarios, combine computer vision, machine learning, knowledge graphs, and natural language processing to obtain interpretable multimodal fault diagnosis information. Compared with traditional industrial fault detection models, it can fully utilize existing detection data and combine multimodal information from different angles and levels for fault diagnosis. It can use knowledge graphs to reason about and distinguish multiple faults, eliminating the subjective dependence on empirical manual fault detection. It also has advantages such as adaptability across various scenarios and ease of implementation.

[0105] Therefore, fault diagnosis methods and systems based on multimodal knowledge graphs are particularly important. This system, by integrating sensing technologies for multiple signals such as sound, light, and electricity, can provide more comprehensive and accurate fault diagnosis. Currently, no algorithms have been found that utilize real-time multimodal data from the electro-hydrogen coupling device for fault diagnosis. The integrated application of these technologies can enhance the system's fault diagnosis and fault tolerance capabilities, ensuring the safe and efficient operation of the electro-hydrogen coupling device in complex environments.

[0106] In summary, in the development of renewable energy hydrogen production systems, especially considering the characteristics of electro-hydrogen coupling devices, traditional fault diagnosis is generally based on a single modality, such as text, vision, or audio. However, traditional fault diagnosis techniques have drawbacks such as relatively complex models, high data requirements, and poor robustness. Therefore, a fault diagnosis method and system based on multimodal knowledge graphs are proposed.

[0107] This method places particular emphasis on multimodal information from different hydrogen production system components, such as electrolyzers and auxiliary equipment. It fully utilizes existing detection data and combines multimodal information from different angles and levels for fault diagnosis. This method and system can promptly identify and respond to faults, ensuring the safety of system operation. It not only fully considers the characteristics of various modal information under electrolyzer start-up / shutdown states and steady-state operation, but also performs knowledge reasoning on multiple faults through multimodal fusion and knowledge graph algorithms. Finally, it uses semantic matching templates to derive the reasoned fault diagnosis results.

[0108] It is worth noting that this fault diagnosis method is not limited to the application of electro-hydrogen coupling devices. It can also be applied to electro-to-X devices or fields according to actual conditions and needs. For example, it can be electro-to-heat, electro-to-hydrogen, electro-to-electricity, electro-to-chemicals, etc. This exemplary embodiment does not make any special limitations in this regard.

[0109] Furthermore, in an exemplary embodiment of this disclosure, a fault diagnosis apparatus is also provided. Figure 9 A schematic diagram of the fault diagnosis device is shown, such as... Figure 9 As shown, the fault diagnosis device 900 may include: a map construction module 910, a vector generation module 920, and a fault diagnosis module 930. Wherein: The knowledge graph construction module 910 is configured to collect multimodal data of the electro-hydrogen coupling device using multiple sensors, and to construct a knowledge graph using the multimodal data. The knowledge graph consists of fault entities, the attributes of the fault entities, and the relationships between the fault entities. The vector generation module 920 is configured to acquire the data to be detected from the electro-hydrogen coupling device and input the data to be detected into a pre-trained fault detection model so that the fault detection model outputs a fault feature vector. The fault diagnosis module 930 is configured to perform semantic matching and reasoning between the fault feature vector and the knowledge graph to obtain a text feature vector, and generate a fault diagnosis result for the data to be detected based on the text feature vector.

[0110] In an exemplary embodiment of the present invention, the map construction module 910 is configured as follows: Multimodal data of the electro-hydrogen coupling device are collected using multiple sensors, and the multimodal data is preprocessed to obtain preprocessed multimodal data.

[0111] In an exemplary embodiment of the present invention, the map construction module 910 is configured as follows: Determine the encoder corresponding to the multimodal data, and use the encoder to extract the fault entities, the attributes of the fault entities, and the relationships between the fault entities from the multimodal data; A knowledge graph is obtained by constructing the faulty entities, their attributes, and the relationships between them into a graph structure.

[0112] In an exemplary embodiment of the present invention, the fault diagnosis device 900 is further configured to: The multimodal data is processed to extract data features, and the data features are used to train a fault detection model to obtain a pre-trained fault detection model.

[0113] In an exemplary embodiment of the present invention, the fault diagnosis module 930 is configured as follows: The similarity between the fault feature vector and the fault entities in the knowledge graph, the attributes of the fault entities, and the relationships between the fault entities is calculated to obtain the calculation result, and the text feature vector is determined based on the calculation result.

[0114] In an exemplary embodiment of the present invention, the fault diagnosis module 930 is configured as follows: When the calculation result is multiple sets of fault entities, attributes of the fault entities, and relationships between the fault entities in the knowledge graph that are similar to the fault feature vector, the multiple sets of fault entities, attributes of the fault entities, and relationships between the fault entities in the knowledge graph are sorted according to the calculation result to obtain a sorting result, and the text feature vector is determined according to the sorting result; or When the calculation result is multiple sets of fault entities, attributes of the fault entities, and relationships between the fault entities in the knowledge graph that are similar to the fault feature vector, multiple text feature vectors corresponding to the multiple sets of fault entities, attributes of the fault entities, and relationships between the fault entities are determined, and a text feature vector is determined from the multiple text feature vectors according to the calculation result.

[0115] In an exemplary embodiment of the present invention, the fault diagnosis module 930 is configured as follows: The text feature vector is input into a text generation model so that the text generation model outputs the fault diagnosis result of the data to be detected. The fault diagnosis result includes one or more of mechanical faults, electrical faults, chemical faults and software faults.

[0116] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0117] Figure 10 This is a block diagram illustrating an electronic device 1000 according to an exemplary embodiment. For example... Figure 10 As shown, the electronic device 1000 may include: a processor 1001 and a memory 1002. The electronic device 1000 may also include one or more of a multimedia component 1003, an input / output (I / O) interface 1004, and a communication component 1005.

[0118] The processor 1001 controls the overall operation of the electronic device 1000 to complete all or part of the steps in the aforementioned fault diagnosis method. The memory 1002 stores various types of data to support the operation of the electronic device 1000. This data may include, for example, instructions for any application or method operating on the electronic device 1000, and application-related data such as contact data, sent and received messages, pictures, audio, video, etc. The memory 1002 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read-Only Memory (EPROM), Programmable Read-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. Multimedia component 1003 may include a screen and an audio component. The screen may be, for example, a touchscreen, and the audio component is used to output and / or input audio signals. For example, the audio component may include a microphone for receiving external audio signals. The received audio signals may be further stored in memory 1002 or transmitted via communication component 1005. The audio component also includes at least one speaker for outputting audio signals. I / O interface 1004 provides an interface between processor 1001 and other interface modules, such as a keyboard, mouse, buttons, etc. These buttons may be virtual or physical buttons. Communication component 1005 is used for wired or wireless communication between the electronic device 1000 and other devices. Wireless communication, such as Wi-Fi, Bluetooth, Near Field Communication (NFC), 2G, 3G, 4G, NB-IoT, eMTC, or other 5G technologies, or combinations thereof, is not limited here. Therefore, the corresponding communication component 1005 may include: a Wi-Fi module, a Bluetooth module, an NFC module, etc.

[0119] In an exemplary embodiment, the electronic device 1000 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the fault diagnosis method described above.

[0120] In another exemplary embodiment, a computer-readable storage medium including program instructions is also provided, which, when executed by a processor, implement the steps of the fault diagnosis method described above. For example, the computer-readable storage medium may be the memory 1002 including program instructions described above, which may be executed by the processor 1001 of the electronic device 1000 to complete the fault diagnosis method described above.

[0121] Figure 11 This is a block diagram illustrating an electronic device 1100 according to an exemplary embodiment. For example, the electronic device 1100 may be provided as a server. (Refer to...) Figure 11 The electronic device 1100 includes a processor 1122, which may be one or more, and a memory 1132 for storing computer programs executable by the processor 1122. The computer program stored in the memory 1132 may include one or more modules, each corresponding to a set of instructions. Furthermore, the processor 1122 may be configured to execute the computer program to perform the aforementioned fault diagnosis method.

[0122] Additionally, the electronic device 1100 may also include a power supply component 1126 and a communication component 1150. The power supply component 1126 can be configured to perform power management of the electronic device 1100, and the communication component 1150 can be configured to enable communication of the electronic device 1100, such as wired or wireless communication. Furthermore, the electronic device 1100 may also include an input / output (I / O) interface 1158. The electronic device 1100 can operate on an operating system stored in the memory 1132.

[0123] In another exemplary embodiment, a computer-readable storage medium including program instructions is also provided, which, when executed by a processor, implement the steps of the fault diagnosis method described above. For example, the non-transitory computer-readable storage medium may be the memory 1132 including the program instructions described above, which may be executed by the processor 1122 of the electronic device 1100 to complete the fault diagnosis method described above.

[0124] In another exemplary embodiment, a computer program product is also provided, the computer program product comprising a computer program executable by a programmable device, the computer program having a code portion for performing the above-described fault diagnosis method when executed by the programmable device.

[0125] The preferred embodiments of this disclosure have been described in detail above with reference to the accompanying drawings. However, this disclosure is not limited to the specific details of the above embodiments. Within the scope of the technical concept of this disclosure, various simple modifications can be made to the technical solutions of this disclosure, and these simple modifications all fall within the protection scope of this disclosure.

[0126] It should also be noted that the various specific technical features described in the above embodiments can be combined in any suitable manner without contradiction. To avoid unnecessary repetition, this disclosure will not describe the various possible combinations separately.

[0127] Furthermore, various different embodiments of this disclosure can be combined in any way, as long as they do not violate the spirit of this disclosure, they should also be regarded as the content disclosed in this disclosure.

Claims

1. A fault diagnosis method, characterized in that, The method includes: Multimodal data of an electro-hydrogen coupling device is collected using multiple sensors, and a knowledge graph is constructed using the multimodal data. The knowledge graph consists of fault entities, the attributes of the fault entities, and the relationships between the fault entities. The test data of the electro-hydrogen coupling device is obtained and input into a pre-trained fault detection model so that the fault detection model outputs a fault feature vector. The fault feature vector is semantically matched and reasoned with the knowledge graph to obtain a text feature vector, and a fault diagnosis result of the data to be detected is generated based on the text feature vector.

2. The fault diagnosis method according to claim 1, characterized in that, The method of acquiring multimodal data from the electro-hydrogen coupling device using multiple sensors includes: Multimodal data of the electro-hydrogen coupling device are collected using multiple sensors, and the multimodal data is preprocessed to obtain preprocessed multimodal data.

3. The fault diagnosis method according to claim 1, characterized in that, The construction of a knowledge graph using the multimodal data includes: Determine the encoder corresponding to the multimodal data, and use the encoder to extract the fault entities, the attributes of the fault entities, and the relationships between the fault entities from the multimodal data; A knowledge graph is obtained by constructing the faulty entities, their attributes, and the relationships between them into a graph structure.

4. The fault diagnosis method according to claim 1, characterized in that, Before inputting the data to be detected into the pre-trained fault detection model, the method further includes: The multimodal data is processed to extract data features, and the data features are used to train a fault detection model to obtain a pre-trained fault detection model.

5. The fault diagnosis method according to claim 1, characterized in that, The step of semantically matching and reasoning with the knowledge graph to obtain the text feature vector includes: The similarity between the fault feature vector and the fault entities in the knowledge graph, the attributes of the fault entities, and the relationships between the fault entities is calculated to obtain the calculation result, and the text feature vector is determined based on the calculation result.

6. The fault diagnosis method according to claim 5, characterized in that, Determining the text feature vector based on the calculation result includes: When the calculation result is multiple sets of fault entities, attributes of the fault entities, and relationships between the fault entities in the knowledge graph that are similar to the fault feature vector, the multiple sets of fault entities, attributes of the fault entities, and relationships between the fault entities in the knowledge graph are sorted according to the calculation result to obtain a sorting result, and the text feature vector is determined according to the sorting result; or When the calculation result is multiple sets of fault entities, attributes of the fault entities, and relationships between the fault entities in the knowledge graph that are similar to the fault feature vector, multiple text feature vectors corresponding to the multiple sets of fault entities, attributes of the fault entities, and relationships between the fault entities are determined, and a text feature vector is determined from the multiple text feature vectors according to the calculation result.

7. The fault diagnosis method according to claim 1, characterized in that, The step of generating the fault diagnosis result of the data to be detected based on the text feature vector includes: The text feature vector is input into a text generation model so that the text generation model outputs the fault diagnosis result of the data to be detected. The fault diagnosis result includes one or more of mechanical faults, electrical faults, chemical faults and software faults.

8. A fault diagnosis device, characterized in that, include: The knowledge graph construction module is configured to collect multimodal data of the electro-hydrogen coupling device using multiple sensors, and to construct a knowledge graph using the multimodal data. The knowledge graph consists of fault entities, the attributes of the fault entities, and the relationships between the fault entities. The vector generation module is configured to acquire the data to be detected from the electro-hydrogen coupling device and input the data to be detected into a pre-trained fault detection model so that the fault detection model outputs a fault feature vector. The fault diagnosis module is configured to perform semantic matching and reasoning between the fault feature vector and the knowledge graph to obtain a text feature vector, and generate a fault diagnosis result for the data to be detected based on the text feature vector.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the program implements the steps of the method described in any one of claims 1-7.

10. An electronic device, characterized in that, include: A memory on which computer programs are stored; A processor for executing the computer program in the memory to implement the steps of the method according to any one of claims 1-7.

Images