Equipment fault diagnosis method and device, electronic equipment and computer program product

Abstract

The invention discloses an equipment fault diagnosis method and device, electronic equipment and a computer program product. The method comprises the following steps: acquiring multi-modal data of equipment to be diagnosed; a pre-trained multi-modal model is adopted to extract multi-modal features from the multi-modal data, and the multi-modal features are fused into fusion features; under the condition that it is determined that the to-be-diagnosed equipment has a fault according to the fusion features, a multi-modal model is adopted, attribution analysis is conducted on the multi-modal features, an attribution evidence chain is obtained, and attribution analysis at least comprises the steps that attribution analysis is conducted on time sequence features in the multi-modal features, and the attribution evidence chain is obtained; and performing causal reasoning on the multi-modal features according to the guidance information generated by the attribution evidence chain by adopting a multi-modal model to obtain the fault root cause of the to-be-diagnosed equipment. According to the invention, the technical problem of inaccurate fault diagnosis due to the fact that fault diagnosis of equipment in the prior art excessively depends on single-mode data is solved.

Description

Technical Field

[0001] This invention relates to the field of industrial control, and more specifically, to a method, apparatus, electronic device, and computer program product for diagnosing equipment faults. Background Technology

[0002] In process industries, energy, power, chemical, and manufacturing sectors, the stable operation of equipment is crucial for production safety and economic efficiency. Traditional methods for equipment fault early warning and diagnosis mainly rely on:

[0003] 1. Mechanism-based approach – This approach requires modeling the operating mechanism of the equipment, but the model building process is complex and difficult to adapt to the differences between multiple operating conditions and multiple devices.

[0004] 2. Statistical or machine learning methods based on single sensor time series data are effective for anomaly detection, but they cannot make full use of multi-source data (such as operation logs, maintenance records, images, and text descriptions), resulting in insufficient diagnostic accuracy.

[0005] 3. Expert systems based on rule bases or case bases – rely on human knowledge accumulation, struggle to cope with new equipment and unknown faults, and have poor scalability.

[0006] The latest equipment fault warning and diagnosis methods mainly rely on:

[0007] 1. Deep learning methods based on multi-sensor time series analysis – require specific modeling for specific devices, are difficult to transfer to other devices or similar devices with different sensor installations, and have high investment costs.

[0008] 2. Anomaly detection methods based on large LLM models are limited by context, cannot observe longer historical time windows, cannot adapt to complex changes in operating conditions, and are prone to illusion problems.

[0009] 3. Anomaly detection method based on time-series images — For the same type of device, the same waveform can cause different anomalies in different devices. Image recognition cannot distinguish the different subdivisions caused by the underlying mechanism.

[0010] In summary, existing technologies cannot fully utilize the characteristics of multimodal data in equipment fault early warning and diagnosis, and have obvious limitations, especially in terms of multimodal data fusion, robustness under complex working conditions, interpretability of fault diagnosis, and closed-loop mechanism of human-machine collaboration.

[0011] The existing technologies for equipment fault diagnosis rely too heavily on single-modal data, which leads to inaccurate fault diagnosis, and no effective solution has yet been proposed. Summary of the Invention

[0012] This invention provides a method, apparatus, electronic device, and computer program product for diagnosing equipment faults, which at least solves the technical problem that existing technologies rely too heavily on single-modal data for equipment fault diagnosis, resulting in inaccurate fault diagnosis.

[0013] According to one aspect of the present invention, a method for diagnosing equipment failure is provided, comprising: acquiring multimodal data of a device to be diagnosed; extracting multimodal features from the multimodal data using a pre-trained multimodal model, and fusing the multimodal features into fused features; if it is determined that the device to be diagnosed has a failure based on the fused features, performing attribution analysis on the multimodal features using the multimodal model to obtain an attribution evidence chain, wherein the attribution analysis includes at least: performing attribution analysis on temporal features in the multimodal features to obtain the attribution evidence chain, and supplementing the attribution evidence chain with at least one modal feature other than the temporal features in the multimodal features; and using the multimodal model to perform causal reasoning on the multimodal features based on guiding information generated by the attribution evidence chain to obtain the root cause of the failure of the device to be diagnosed.

[0014] Optionally, after extracting multimodal features from the multimodal data using a pre-trained multimodal model and fusing the multimodal features into the fused features, the method further includes: using the multimodal model to perform predictive inference on the multimodal features to obtain predicted data of the device to be diagnosed at a target time, wherein the target time is later than the acquisition time of the multimodal data, the predictive inference is based at least on the temporal and process features in the multimodal features, and combined with the image and semantic features in the multimodal features, to infer the predicted features of the device to be diagnosed at the target time, and to determine the predicted data based on the predicted features, wherein the predicted features include: the multimodal features of the device to be diagnosed at the target time and the fused features, and the predicted data includes: the multimodal data reconstructed based on the multimodal features at the target time.

[0015] Optionally, before extracting multimodal features from the multimodal data using a pre-trained multimodal model and fusing the multimodal features into the fused features, the method further includes: acquiring multimodal sample data, wherein the multimodal sample data includes single-modal sample data of multiple modalities; using a preset multimodal model to perform representation encoding learning on the single-modal sample data of different modalities to obtain a first model, wherein the preset multimodal model is the initial model of the multimodal model before training, and the representation encoding learning is used to enable the first model to learn to extract the multimodal features from the multimodal data; The first model is used to perform fusion alignment learning on the single-modal sample features of different modalities to obtain a second model, wherein the single-modal sample features are extracted from the single-modal sample data using the first model, and the fusion alignment learning is used to enable the second model to learn to align and fuse modal features of different modalities; the second model is used to perform feature mask reconstruction on the multimodal sample data to obtain the multimodal model, wherein the feature mask reconstruction is used to enable the multimodal model to learn the dependencies between modal data of different modalities, and the dependencies are used at least for causal inference and predictive inference.

[0016] Optionally, when determining that the device to be diagnosed is faulty based on the fusion features, before using the multimodal model to perform attribution analysis on the multimodal features to obtain the attribution evidence chain, the method further includes at least one of the following: performing a first-level judgment on the multimodal data using pre-set judgment conditions, wherein the judgment conditions are generated using a pre-set language model based on judgment statements described in natural language; if the result of the first-level judgment indicates that the multimodal data does not meet the judgment conditions, determining that the device to be diagnosed is faulty; performing a second-level judgment on the reconstruction error of the multimodal data using a pre-set error threshold, wherein the reconstruction error is based on the fusion features... The difference between the multimodal reconstructed data and the multimodal data is determined when the reconstruction error exceeds the error threshold in the secondary determination. A third-level determination is then performed on the fused features using a pre-defined feature library, which includes a first database and a second database. The first database contains positive sample features from multiple non-faulty devices, and the second database contains negative sample features from multiple faulty devices. The device is determined to be faulty when the third-level determination indicates that the fused features do not have matching positive sample features, or that the fused features have matching negative sample features.

[0017] Optionally, before performing a three-level judgment on the fused features using a pre-defined feature library, the method further includes: acquiring multimodal historical data of the device during its historical operation phase, wherein the multimodal historical data includes: multimodal positive sample data of non-faulty devices and multimodal negative sample data of faulty devices; performing multimodal fusion on the multimodal historical data using the multimodal model to obtain multimodal historical features, wherein the multimodal historical features include: multimodal positive sample features determined based on the multimodal positive sample data and multimodal negative sample features obtained based on the multimodal negative sample data; and further processing the multimodal historical features. Clustering is performed to obtain multiple cluster feature clusters, wherein the cluster feature clusters include: positive sample feature clusters obtained by clustering the multimodal positive sample features, and negative sample clusters obtained by clustering the multimodal negative sample features; feature mapping aggregation is performed on each cluster feature cluster to obtain sample features, and the sample features are stored in the feature library, wherein the feature mapping aggregation is used to compress multiple multimodal historical features in the same cluster feature cluster into a single sample feature, and the sample feature includes: positive sample features obtained based on the positive sample feature clusters, and negative sample features obtained based on the negative sample feature clusters.

[0018] Optionally, when it is determined that the device to be diagnosed has a fault based on the fusion features, the multimodal model is used to perform attribution analysis on the multimodal features to obtain an attribution evidence chain, including: performing attention analysis on the temporal features in the multimodal features to obtain an attention map, wherein the attention map represents the dependency relationship between different temporal features; performing gradient attribution analysis on the attention map to obtain an attribution score for each temporal feature in the attention map, wherein the attribution score is positively correlated with the feature contribution of the temporal feature; determining the attribution evidence chain based on the attribution score, wherein the attribution evidence chain is determined based on the temporal features whose attribution scores are higher than a preset score threshold, and the dependency relationship between the temporal features; and selecting a target historical case that matches the attribution evidence chain from multiple historical cases recorded in the historical case library for supplementation, wherein each historical case records a corresponding historical evidence chain, and the target historical case is the historical case whose historical evidence chain matches the attribution evidence chain.

[0019] Optionally, the multimodal data includes at least: time-series data, image data, text data, and process data. A pre-trained multimodal model is used to extract multimodal features from the multimodal data, and the multimodal features are fused into a fused feature. This includes: using a first feature extraction model to extract time-series features from the time-series data, wherein the first feature extraction model employs a Transformer network architecture; using a second feature extraction model to extract image features from the image data, wherein the second feature extraction model employs a CNN network architecture or a ViT network architecture; using a third feature extraction model to extract semantic features from the text data, wherein the third feature extraction model employs a BERT network architecture or a GPT network architecture; and using a fourth feature extraction model to extract process features from the process data, wherein the fourth feature extraction model employs a GNN network architecture; and fusing the time-series features, the image features, the semantic features, and the process features to obtain the fused feature.

[0020] According to another aspect of the present invention, a device for diagnosing equipment failure is also provided, comprising: an acquisition module for acquiring multimodal data of a device to be diagnosed; a feature fusion module for extracting multimodal features from the multimodal data using a pre-trained multimodal model and fusing the multimodal features into fused features; an attribution analysis module for performing attribution analysis on the multimodal features using the multimodal model to obtain an attribution evidence chain when it is determined that the device to be diagnosed has a failure based on the fused features, wherein the attribution analysis includes at least: performing attribution analysis on temporal features in the multimodal features to obtain the attribution evidence chain, and supplementing the attribution evidence chain with at least one modal feature other than the temporal features in the multimodal features; and a root cause reasoning module for performing causal reasoning on the multimodal features using the multimodal model based on guiding information generated by the attribution evidence chain to obtain the root cause of the failure of the device to be diagnosed.

[0021] According to another aspect of the present invention, an electronic device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to execute the above-described device fault diagnosis method through the computer program.

[0022] According to another aspect of the present invention, a computer program product is also provided, including computer instructions that, when executed by a processor, implement the steps of the above-described device fault diagnosis method.

[0023] The embodiments described above acquire multimodal data of the device to be diagnosed, and use a pre-trained multimodal model to extract and fuse features from this data to form fused features for device fault diagnosis. Then, based on the determination that the device has a fault, a multimodal model is used to perform deep attribution analysis to form an attribution evidence chain. The attribution evidence chain is based on the attribution analysis of time-series features, supplemented by other modal features. Causal reasoning is then performed based on the guiding information generated by the attribution evidence chain to further analyze the multimodal features and derive the root cause diagnosis of the device fault. This overcomes the limitations of traditional single-modal fault diagnosis, effectively improves the accuracy and generalization ability of the diagnosis, and solves the technical problem of inaccurate fault diagnosis caused by the over-reliance on single-modal data in existing technologies. Attached Figure Description

[0024] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this application, illustrate exemplary embodiments of the invention and, together with their description, serve to explain the invention and do not constitute an undue limitation thereof. In the drawings:

[0025] Figure 1 This is a flowchart of a method for diagnosing equipment faults according to an embodiment of the present invention;

[0026] Figure 2 This is a schematic diagram of an anomaly detection method according to an embodiment of the present invention;

[0027] Figure 3 This is a schematic diagram of a general equipment fault early warning, diagnosis and linkage process based on a time series multimodal large model according to an embodiment of the present invention;

[0028] Figure 4 This is a schematic diagram of a device for diagnosing equipment faults according to an embodiment of the present invention;

[0029] Figure 5 This is a structural block diagram of a computer terminal according to an embodiment of the present invention. Detailed Implementation

[0030] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0031] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0032] First, some nouns or terms that appear in the description of the embodiments of this application shall be interpreted as follows:

[0033] Time series data: In this application, time series data specifically refers to data with time attributes that are continuously collected by device sensors or by humans.

[0034] Tag: The name of data collected by different sensors, also known as features. In order to distinguish it from the latent space features of the model, this application uses tag to represent the name of data collected from different data sources.

[0035] Multimodal data: data from different sensor domains or collected manually from different sensor domains.

[0036] Data lake: A data storage and management platform for storing data of different modalities.

[0037] According to an embodiment of the present invention, a method for diagnosing device faults is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0038] Figure 1 This is a flowchart of a device fault diagnosis method according to an embodiment of the present invention, such as... Figure 1 As shown, the method includes the following steps:

[0039] Step S102: Obtain multimodal data of the device to be diagnosed;

[0040] Step S104: Using a pre-trained multimodal model, extract multimodal features from the multimodal data and fuse the multimodal features into fused features;

[0041] Step S106: If it is determined that the device to be diagnosed has a fault based on the fusion features, a multimodal model is used to perform attribution analysis on the multimodal features to obtain an attribution evidence chain. The attribution analysis includes at least: performing attribution analysis on the temporal features in the multimodal features to obtain an attribution evidence chain, and supplementing the attribution evidence chain with at least one modal feature other than the temporal features in the multimodal features.

[0042] Step S108: Using a multimodal model, causal reasoning is performed on the multimodal features based on the guiding information generated by the attribution evidence chain to obtain the root cause of the fault of the device to be diagnosed.

[0043] The embodiments described above acquire multimodal data of the device to be diagnosed, and use a pre-trained multimodal model to extract and fuse features from this data to form fused features for device fault diagnosis. Then, based on the determination that the device has a fault, a multimodal model is used to perform deep attribution analysis to form an attribution evidence chain. The attribution evidence chain is based on the attribution analysis of time-series features, supplemented by other modal features. Causal reasoning is then performed based on the guiding information generated by the attribution evidence chain to further analyze the multimodal features and derive the root cause diagnosis of the device fault. This overcomes the limitations of traditional single-modal fault diagnosis, effectively improves the accuracy and generalization ability of the diagnosis, and solves the technical problem of inaccurate fault diagnosis caused by the over-reliance on single-modal data in existing technologies.

[0044] The above embodiments of this application combine attribution evidence chains and causal reasoning to improve the transparency and interpretability of the diagnostic process, enabling maintenance personnel to understand and verify the diagnostic results, enhancing the credibility of the system. Furthermore, the diagnostic results can guide the control system and the operation and maintenance team to achieve automated linkage in fault handling, improving maintenance efficiency and production safety. In addition, the equipment department diagnostic method provided by this application can effectively address the technical problems of equipment fault early warning and diagnosis relying too much on single-modal data and lacking interpretability and linkage capabilities.

[0045] In step S102 above, the multimodal data includes at least: time series data, image data, text data and process data. The process data is used to represent the process flow of the device to be diagnosed. The correlation between the data of each modality can be determined based on the process flow. For example, a certain process flow may require the operating condition data in the previous process flow. The operating condition data can be time series data, image data or text data.

[0046] Optionally, time series signals (also known as time series data, which may include: temperature, pressure, flow rate, vibration, etc.); image / video information (i.e., image data, which may include: thermal imaging, monitoring video, infrared imaging); text information (i.e., text data, which may include: maintenance records, maintenance reports, process descriptions); and structured data (i.e. process data, which may include: operating parameters, process conditions).

[0047] As an optional example, after acquiring multimodal data, a time-series multimodal data lake can be constructed based on the multimodal data, specifically including:

[0048] Step S211: Collect or access various data sources such as sensors (to obtain time-series data), image / infrared cameras (to obtain image data), control system logs (to obtain text data or time-series data), maintenance / inspection texts (to obtain text data), and process settings (to obtain process data), and perform standardized protocol processing to obtain raw multimodal data + preliminary tags (factory ID, equipment ID, mode, time, line number, etc.).

[0049] Step S212: Perform storage preprocessing on the real-time acquired multimodal data, such as denoising, smoothing, and stopping the removal of time series data; and perform quality checks and compression on the images to obtain preprocessed multimodal data + metadata information.

[0050] Step S213 involves centralized storage, format standardization, and structured retrieval. For example, time-series data is stored in a TSDB (Time Series Database), and text data is stored in a full-text retrieval system according to device / sensor identifiers. All data entering the data lake undergoes standard format conversion (units, measurement standards, naming conventions, etc.), and a structured retrieval system is established to facilitate the extraction of any time-series data and its corresponding metadata (modality, data version, format, attached text, device identifier, factory, sampling frequency, etc.).

[0051] As an optional example, after acquiring multimodal data, preprocessing can be performed on the multimodal data, specifically including:

[0052] Step S214: Perform a quality assessment on the data in the data lake, such as missing rate, latency, format consistency, unit correctness, image blurriness, and text integrity.

[0053] Step S215: Semi-automatic data cleaning and management with manual assistance, such as clearing faulty sensor data, time alignment, and filling in data.

[0054] Step S216: Prepare data according to the training format and optimize retrieval to improve training and inference efficiency;

[0055] Step S217: Multimodal data clustering provides topological links for data of the same type, facilitating retrieval and retrieval.

[0056] As an optional embodiment, the multimodal data includes at least: time-series data, image data, text data, and process data. A pre-trained multimodal model is used to extract multimodal features from the multimodal data and fuse these features into a fused feature. This includes: using a first feature extraction model to extract time-series features from the time-series data, wherein the first feature extraction model employs a Transformer network architecture; using a second feature extraction model to extract image features from the image data, wherein the second feature extraction model employs a CNN network architecture or a ViT network architecture; using a third feature extraction model to extract semantic features from the text data, wherein the third feature extraction model employs a BERT network architecture or a GPT network architecture; and using a fourth feature extraction model to extract process features from the process data, wherein the fourth feature extraction model employs a GNN network architecture. The time-series features, image features, semantic features, and process features are then fused to obtain the fused feature.

[0057] In the embodiments described above, the multimodal data covers at least four types of modal information: time-series data, image data, text data, and process data. Each data type undergoes feature extraction using a corresponding pre-trained model. Specifically, time-series data utilizes a first feature extraction model based on a Transformer network to acquire its dynamic characteristics; image data employs a second feature extraction model based on a CNN or ViT network to capture spatial features; text data uses a third feature extraction model based on a BERT or GPT network to extract semantic vectors; and process data is represented as structured variables using a fourth feature extraction model based on a GNN network. Subsequently, a feature fusion mechanism is used to integrate these multimodal data. Features are integrated into a fusion feature, which not only contains comprehensive information about equipment operation, but also enhances the robustness of fault early warning and the accuracy of diagnosis through mutual verification and supplementation between multiple modalities. This fusion feature is used for anomaly detection, root cause diagnosis, linkage control, and human feedback mechanisms, realizing a closed-loop process from data fusion to fault early warning and diagnosis, and then to system linkage and continuous model optimization. Based on this closed-loop process, not only is the efficiency and accuracy of anomaly detection improved, but also efficient collaboration between the model and human experts is promoted, enabling the system to continuously improve its diagnostic and early warning capabilities over time, thereby providing strong support for the operation and maintenance of industrial equipment.

[0058] Optionally, the temporal features, image features, semantic features, and process features can be fused to obtain the fused features. Feature fusion mechanisms include gating networks, cross-attention, or contrastive learning.

[0059] As an alternative example, multimodal feature learning is achieved through multimodal fusion, specifically including:

[0060] Step S221: Encode the time series data: Use a Transformer network such as PatchTST / TimeNet to extract dynamic features, that is, determine the time series features;

[0061] Step S222: Feature encoding of image data: Extracting image representation features using networks such as CNN / ViT, that is, determining image features;

[0062] Step S223, feature encoding of text data: extract semantic vectors using BERT / GPT, that is, determine semantic features;

[0063] Step S224, feature encoding of the association graph: using GNN to encode structured variable representations to obtain process features;

[0064] Step S225: Multi-modal feature fusion is achieved through cross-attention mechanism, gating network, alignment mechanism, and contrastive learning to obtain fused features.

[0065] It's important to note that PatchTST is a deep learning model architecture specifically designed for processing time series data. It was developed based on the successful patch-based Transformer model in computer vision. PatchTST divides time series data into a series of fixed-length "time slices" (Patches), similar to segmenting an image into multiple small patches in image processing. Then, it leverages the powerful sequence modeling capabilities of the Transformer architecture to extract features and model relationships between these time slices.

[0066] As an alternative example, in the standard Vision Transformer, an image is segmented into a series of small image patches. These patches are flattened and transformed into vectors, which are then input into the Transformer for processing. PatchTST follows this idea, decomposing time-series data into a series of consecutive time windows, each of which can be viewed as a "time slice." These time slices are transformed into a form suitable for Transformer processing and then a self-attention mechanism is used to capture long-term dependencies and potential dynamic patterns within the time series.

[0067] In the embodiments described above, PatchTST is used as a core component for time series data processing. It is responsible for extracting dynamic features from device sensor data, providing strong support for subsequent anomaly detection, fault warning, and root cause diagnosis. Compared with traditional LSTM or other time series models, PatchTST may be more suitable for processing multivariate, multi-scale time series data, especially in scenarios requiring the processing of long sequences and prediction using large amounts of historical data, where it exhibits better performance and efficiency.

[0068] It should be noted that the multimodal model can also predict whether the device under diagnosis will have a fault at the target time if the fault does not occur, based on multimodal prediction data or multimodal prediction features. Then, based on the multimodal prediction data or multimodal prediction features, the device fault diagnosis method provided in the application can be used to predict whether the device under diagnosis will have a fault at the target time. Furthermore, if the fault is predicted to occur at the target time, the root cause of the fault can be predicted in advance, thereby achieving early warning of device faults.

[0069] As an optional embodiment, after using a pre-trained multimodal model to extract multimodal features from multimodal data and fusing the multimodal features into fused features, the method further includes: using the multimodal model to perform predictive inference on the multimodal features to obtain predicted data of the device to be diagnosed at a target time, wherein the target time is later than the acquisition time of the multimodal data, the predictive inference is based at least on the temporal features and process features in the multimodal features, and combined with the image features and semantic features in the multimodal features to infer the predicted features of the device to be diagnosed at the target time, and the predicted data is determined based on the predicted features, wherein the predicted features include: the multimodal features and fused features of the device to be diagnosed at the target time, and the predicted data includes: multimodal data reconstructed based on the multimodal features at the target time.

[0070] The embodiments described above employ a multimodal model for predictive reasoning based on multimodal features and fusion features in multimodal data. This predictive reasoning process fully utilizes temporal and process features as the primary basis, supplemented by image and semantic features, to jointly infer the predicted features of the equipment at the target time, thereby deriving predicted data. These predicted features comprehensively cover the multimodal and fusion features at the target time, while the predicted data is multimodal data reconstructed based on the predicted features, including the future forms of temporal data, image data, text data, and process data. By utilizing the predictive reasoning mechanism of the multimodal model, not only can the operating status of the equipment be proactively warned, but also the accuracy of the warning and the comprehensiveness of the diagnosis are significantly improved by integrating information from different modalities. This helps to take measures in advance to prevent potential failures and ensure the stable operation of the equipment.

[0071] Optionally, multimodal models also allow for continuous enhancement of predictive inference performance and improvement of the intelligence level of equipment management through continuous data updates and model iterations.

[0072] Optionally, the target time can be a target point in time or a target time period.

[0073] As an optional embodiment, before extracting multimodal features from multimodal data and fusing the multimodal features into fused features, the method further includes: acquiring multimodal sample data, wherein the multimodal sample data includes single-modal sample data of multiple modalities; using a preset multimodal model to perform representation encoding learning on the single-modal sample data of different modalities to obtain a first model, wherein the preset multimodal model is the initial model of the multimodal model before training, and the representation encoding learning is used to enable the first model to learn to extract multimodal features from the multimodal data; using the first model to perform fusion alignment learning on the single-modal sample features of different modalities to obtain a second model, wherein the single-modal sample features are extracted from the single-modal sample data using the first model, and the fusion alignment learning is used to enable the second model to learn to align and fuse the modal features of different modalities; using the second model to perform feature mask reconstruction on the multimodal sample data to obtain a multimodal model, wherein the feature mask reconstruction is used to enable the multimodal model to learn the dependencies between the modal data of different modalities, and the dependencies are used at least for causal inference and predictive inference.

[0074] In the embodiments described above, a first model is obtained by using a preset multimodal model to perform representation encoding learning on single-modal sample data of different modalities. This model can extract features of each modality from the multimodal data. Subsequently, the first model is used to perform fusion and alignment learning on the features of single-modal samples of different modalities to obtain a second model. This stage enables the model to learn to align and fuse various modal features to construct a unified view in a high-dimensional shared latent space. Finally, the second model is used to reconstruct the feature mask of the multimodal sample data to learn the dependencies between different modal data and obtain a multimodal model. These dependencies are used not only for causal reasoning but also for predictive reasoning, thereby enabling the model to have stronger interpretability and generalization ability when processing multimodal data, and thus improving the accuracy of equipment fault early warning and diagnosis.

[0075] It should be noted that the entire multi-stage learning process gradually moves from modal-independent learning to modal fusion, ultimately forming a large multi-modal model capable of handling complex industrial scenarios. This model is robust and adaptive, and can still achieve data-driven fault warning, diagnosis, and linkage even in the absence of a complete knowledge base.

[0076] Optionally, the single-modal sample data of multiple modalities in the multimodal sample data can be: time series data, image data, text data, and process data.

[0077] It should be noted that in the field of deep learning, especially when dealing with multimodal data, representation encoding learning, fusion alignment learning, and feature mask reconstruction are three key concepts. They play different roles in the model training process and aim to improve the model's ability to understand, fuse, and be robust to data.

[0078] Optionally, representation encoding learning is a foundational stage of multimodal deep learning, with the primary goal of generating meaningful, compact feature vectors for each modality of data. In this process, the model learns to extract key, patterned information from the input data.

[0079] For example, for time-series data, such as sensor readings of temperature and pressure, the model may use Transformer, LSTM, or GRU to learn dynamic patterns; for image data, the model may use convolutional neural networks (CNN) or visual Transformers to extract visual features; and for text data, BERT, GPT, or other pre-trained language models are used to encode semantic information.

[0080] It should be noted that the result of representation encoding learning is to transform the original input into feature vectors in a high-dimensional space. These feature vectors should fully reflect the intrinsic characteristics of their respective modalities.

[0081] Optionally, the goal of Fusion Alignment Learning is to align feature vectors from different modalities and fuse them in a multimodal fusion space, so that the model can comprehensively utilize information from all modalities.

[0082] Alternatively, common methods for incorporating alignment learning include:

[0083] Cross-Attention: Allows feature interaction between different modalities, enabling features of one modality to be adjusted based on features of another modality.

[0084] Multimodal alignment network: Design a specialized network structure to reduce feature differences between modalities and ensure that the model can seamlessly process data from different modalities.

[0085] Joint embedding space: Creates a shared embedding space in which feature vectors of all modalities can be represented, making it easier to fuse and compare them.

[0086] It's important to note that the key to fusion-alignment learning lies in establishing a collaborative mechanism that allows information from different sources to complement each other, rather than simply being superimposed. In this way, the model can understand the potential relationships between multimodal data, thereby improving its performance on multimodal tasks.

[0087] Optionally, feature masking reconstruction is a technique to enhance model learning capabilities, particularly suitable for self-supervised or semi-supervised learning scenarios. Its basic idea is to randomly mask a portion of the input data during training, then require the model to predict or reconstruct the masked portion based on the remaining uncovered parts. In this way, the model is forced to learn the inherent structure of the data and the dependencies between modalities, rather than simply memorizing input-output correspondences.

[0088] For example, in time series, data at specific time points may be randomly occluded, and the model is trained to predict the values ​​of the occluded points; in images, some pixels may be occluded, and the model is asked to try to recover the visual content of that part; in text, some words may be occluded, and the model is required to infer the missing words based on the context.

[0089] It's important to note that feature masking and reconstruction not only improves the model's generalization ability but also helps it function effectively even with missing or poor-quality data, enhancing its robustness. Furthermore, through masking and reconstruction, the model can learn the redundancy and complementarity relationships between features, which is particularly crucial for the fusion of multimodal data.

[0090] In the embodiments described above, the three learning stages—representation encoding learning, fusion alignment learning, and feature mask reconstruction—are typically performed sequentially during the pre-training of the multimodal model. First, each modality is encoded independently. Then, the model learns how to align and fuse these features in the fusion space. Finally, feature mask reconstruction enhances the model's ability to learn dependencies between features. This hierarchical learning process helps build a robust and comprehensive multimodal model that can effectively learn and process complex scenarios even without complete labeled data. It represents an important methodology for modern AI systems to address the challenges of multimodal data.

[0091] As an optional implementation, after using the second model to reconstruct the feature mask of the multimodal sample data to obtain the multimodal model, the trained multimodal model can be fine-tuned in a supervised manner according to different task requirements.

[0092] For example, when a multimodal model is required to have predictive capabilities, such as predicting the predictive features or predictive data of the device to be diagnosed at a target time, the multimodal model can be fine-tuned in a supervised manner based on a pre-calibrated prediction task; when a multimodal model is required to have anomaly diagnosis capabilities or root cause diagnosis capabilities, such as detecting whether the device to be diagnosed has a fault or determining the root cause of the fault, the multimodal model can be fine-tuned in a supervised manner based on a pre-calibrated anomaly detection task or root cause detection task.

[0093] As an optional implementation, model pre-training and fine-tuning on multimodal data specifically includes:

[0094] Step S231: The multimodal model pre-training adopts a multi-stage training method. First, the representation encoding learning is performed on the data of different modalities. Then, the fusion and alignment learning is performed on the data of different modalities. Next, the feature mask reconstruction is performed on the multimodal data to obtain the pre-trained multimodal model base.

[0095] Step S232: The pre-trained multimodal model base already has a relatively strong ability to represent different modal data combinations. Next, we will use prediction tasks, imputation tasks and labeled anomaly detection tasks to perform supervised fine-tuning, guide and stimulate the ability of the multimodal model in time-series scenarios.

[0096] Step S233: Use data from different business scenarios to quickly adapt the model, such as task-level LoRA fine-tuning, so that the model can quickly adapt to the scenarios and data of the business scenarios.

[0097] As an optional embodiment, when the device to be diagnosed is determined to be faulty based on fusion features, before using a multimodal model to perform attribution analysis on the multimodal features to obtain the attribution evidence chain, the method further includes at least one of the following: using pre-set judgment conditions to perform a first-level judgment on the multimodal data, wherein the judgment conditions are generated using a pre-set language model based on judgment statements described in natural language; if the result of the first-level judgment indicates that the multimodal data does not meet the judgment conditions, the device to be diagnosed is determined to be faulty; using a pre-set error threshold to perform a second-level judgment on the reconstruction error of the multimodal data, wherein the reconstruction error is the difference between the reconstructed multimodal data based on the multimodal features and the multimodal data; if the result of the second-level judgment indicates that the reconstruction error exceeds the error threshold, the device to be diagnosed is determined to be faulty; using a pre-set feature library to perform a third-level judgment on the fusion features, wherein the feature library includes: a first database and a second database, the first database contains positive sample features of multiple non-faulty devices, and the second database contains negative sample features of multiple faulty devices; if the result of the third-level judgment indicates that the fusion features do not have matching positive sample features, or that the fusion features have matching negative sample features, the device to be diagnosed is determined to be faulty.

[0098] In the embodiments described above, when the method determines signs of a fault in the device to be diagnosed based on fusion features, it further includes using a multimodal model for multi-level anomaly judgment to enhance the accuracy and timeliness of diagnosis. Using pre-set judgment conditions, a language model generates data based on natural language descriptions to perform a first-level judgment on the multimodal data. If the data does not meet these conditions, a potential device fault is quickly identified. Simultaneously, a pre-set error threshold is used to perform a second-level judgment on the reconstruction error of the multimodal data. When the error exceeds the threshold, the existence of a device fault is further confirmed. Furthermore, a pre-built feature library, including positive sample features from non-faulty devices and negative sample features from faulty devices, is used for a third-level judgment. If the fusion feature does not match a positive sample or matches a negative sample, a device fault is determined. This multi-level judgment mechanism effectively improves the sensitivity and specificity of anomaly detection, ensuring timely and accurate judgment of device status under various operating conditions, reducing false alarms and missed alarms, and providing important basis for subsequent attribution analysis and equipment maintenance.

[0099] Figure 2 This is a schematic diagram of an anomaly detection method according to an embodiment of the present invention, such as... Figure 2As shown, manually imported conditions are processed into code using an LLM algorithm to obtain direct judgment statements. A first-level judgment is made based on the current data, predicted data, and imported conditions. A second-level judgment is made using the reconstruction error of the current data and an adaptive threshold. The adaptive threshold is calculated using labels if available, and automatically labeled using a multimodal large model after time series to image conversion if no labels are available. A third-level judgment is made by comparing features between the current data and predicted data using a normal feature library built from normal data and an abnormal feature library built from abnormal data.

[0100] It should be noted that when multimodal data is identified as abnormal at the first level, it means that the equipment's operating parameters deviate from the set language model judgment criteria, which is often an early sign of equipment failure. The second level of judgment, through quantitative analysis and reconstruction error, can more intuitively measure the deviation between model prediction and actual operating status. When the error exceeds a reasonable range, it indicates that the equipment status change exceeds normal fluctuations and requires immediate attention. The third level of judgment, by comparing with a feature library, can identify the similarity between the equipment status and historical failure patterns. Even in the absence of obvious abnormal indicators, potential hidden dangers can be discovered through pattern matching. Through the effective combination of these three levels of judgment, early warnings can be issued in the early stages of failure, and detailed information on fault location can be provided through subsequent attribution analysis, enhancing the practicality and reliability of the fault early warning system.

[0101] As an optional embodiment, before performing three-level judgment on the fused features using a pre-defined feature library, the method further includes: acquiring multimodal historical data of the device during its historical operation phase, wherein the multimodal historical data includes: multimodal positive sample data of non-faulty devices and multimodal negative sample data of faulty devices; performing multimodal fusion on the multimodal historical data using a multimodal model to obtain multimodal historical features, wherein the multimodal historical features include: multimodal positive sample features determined based on multimodal positive sample data and multimodal negative sample features obtained based on multimodal negative sample data; Clustering of multimodal historical features yields multiple cluster feature clusters. These cluster feature clusters include positive sample feature clusters obtained by clustering multimodal positive sample features and negative sample clusters obtained by clustering multimodal negative sample features. Feature mapping aggregation is performed on each cluster feature cluster to obtain sample features, which are then stored in a feature library. Feature mapping aggregation is used to compress multiple multimodal historical features within the same cluster feature cluster into a single sample feature. The sample features include positive sample features obtained based on positive sample feature clusters and negative sample features obtained based on negative sample feature clusters.

[0102] The embodiments described above in this application extract multimodal negative sample features representing non-faulty devices and multimodal positive sample features representing faulty devices from multimodal historical data that has already been used for fault diagnosis. These features are then compressed and placed into a feature library through multimodal fusion, clustering, and feature mapping aggregation. This ensures that the feature library stores only the most representative sample features, and fault detection of the device to be diagnosed is performed based on this feature library. This approach can improve diagnostic efficiency while ensuring the accuracy of fault diagnosis.

[0103] It should be noted that the feature library is a multimodal feature library covering normal operation and various known fault modes. It makes more accurate three-level judgments on the real-time equipment status, namely whether the current equipment status deviates from the feature range of normal operation and whether it is close to a specific fault feature cluster, so as to issue early warnings and diagnoses in a timely manner, effectively prevent and handle equipment failures, and ensure production safety and efficiency.

[0104] Alternatively, the feature library construction process can be optimized using more clustering algorithms and techniques to improve the feature library's discriminative power and the model's diagnostic accuracy.

[0105] Optionally, the feature library can be continuously updated to incorporate new positive and negative sample data to adapt to changes in equipment operating conditions and the evolution of fault types, thereby enhancing the system's adaptability and robustness.

[0106] It should be noted that the feature mapping aggregation process can be adaptively optimized with the help of deep learning technology to achieve more accurate feature compression and representation, thereby improving the quality and usability of the feature library.

[0107] As an optional example, the process of building a feature library specifically includes:

[0108] Step S241: Using the pre-trained multimodal model, extract multimodal representations of a large amount of historical data (divided into normal data and abnormal data; if abnormal data has labels, then classified into normal data, abnormal label 1 data, abnormal label 2 data, etc.).

[0109] Step S242: Perform internal clustering on the normal and abnormal data after coarse classification, dividing them into normal or abnormal feature clusters with feature discrimination.

[0110] Step S243: Perform feature mapping aggregation on each feature cluster and compress them all into a single data feature size, thereby completing the construction of different normal and abnormal feature libraries.

[0111] It should be noted that when it is determined that the device to be diagnosed is faulty, a multimodal model can be used to perform attribution analysis on the fused multimodal features in order to construct an attribution evidence chain, and to infer the root cause of the fault based on the attribution evidence chain.

[0112] As an optional embodiment, when the fault is determined in the device to be diagnosed based on the fusion features, a multimodal model is used to perform attribution analysis on the multimodal features to obtain the attribution evidence chain, including: performing attention analysis on the temporal features in the multimodal features to obtain an attention map, wherein the attention map represents the dependency relationship between different temporal features; performing gradient attribution analysis on the attention map to obtain the attribution score of each temporal feature in the attention map, wherein the attribution score is positively correlated with the feature contribution of the temporal feature; determining the attribution evidence chain based on the attribution score, wherein the attribution evidence chain is determined based on the temporal features with attribution scores higher than a preset score threshold and the dependency relationship between the temporal features; and selecting a target historical case that matches the attribution evidence chain from multiple historical cases recorded in the historical case library to supplement the attribution evidence chain, wherein each historical case records a corresponding historical evidence chain, and the target historical case is a historical case whose historical evidence chain matches the attribution evidence chain.

[0113] In the embodiments described above, the multimodal model can perform attention analysis on time-series features to generate an attention map, which reveals the interdependencies between different time-series features. Then, by performing gradient attribution analysis on the attention map, the attribution score of each time-series feature is quantified. The attribution score is proportional to the feature's contribution to the fault. Based on the time-series features with attribution scores higher than a preset threshold and their interdependencies, an attribution evidence chain is determined. Furthermore, based on this attribution evidence chain, target historical cases matching the attribution evidence chain can be selected from a historical case library for supplementary explanation. This supplementary attribution evidence chain not only enhances the interpretability of fault diagnosis but also provides a more reliable basis for fault analysis through instance comparison, improving the accuracy and efficiency of diagnosis.

[0114] Alternatively, attribution analysis can also be extracted directly from the pre-trained model, further accelerating the diagnostic speed and enhancing the system's real-time response capability.

[0115] It should be noted that each historical case record has a corresponding historical evidence chain, while the target historical case is the case with the highest matching degree between the historical evidence chain and the attribution evidence chain.

[0116] It should be noted that after an alarm is triggered during the anomaly detection process, anomaly diagnosis is performed through a multi-stage diagnostic pipeline. First, attribution analysis is used to identify "which variables / time points" are involved; then, image localization is performed in parallel to locate spatial anomalies; simultaneously, RAG / vector retrieval is used to search for similar events in the historical case library and return "explanation paths"; finally, this evidence is sent to the causal reasoning module (e.g., including structured priors + temporal causal discovery) to establish "possible failure links" and provide actionable suggestions.

[0117] It should be noted that RAG (Retrieve-Augmented Generation) is a natural language processing technique that combines retrieval and generation. Its core idea is to combine external knowledge bases or document retrieval mechanisms with generative models.

[0118] It's important to note that during text generation, the model not only relies on its own internal knowledge but also retrieves relevant information from external knowledge bases and integrates it into the generation process. This means the model can access and reference real-time or historical data, professional literature, case databases, etc., to provide more accurate, detailed, and fact-based answers or text. Specifically, RAG's workflow is as follows:

[0119] Retrieve: Given a query or input, the model will search an external knowledge base for documents or paragraphs that are relevant to the input.

[0120] Augmentation: The retrieved relevant information is encoded as features and fed into the generative model along with the original input to enhance the model's generative capabilities.

[0121] Generation: The model uses the enhanced input to generate responses or text. This process can better capture external knowledge and provide more specific and targeted answers.

[0122] In the embodiments described above, RAG is used to enhance the model's root cause diagnosis capabilities. When a fault warning or diagnosis occurs, the system uses RAG technology to query similar events and corresponding handling solutions in the historical case database, using this as additional evidence to enrich and refine the diagnostic results, thereby improving the accuracy and reliability of the diagnosis.

[0123] As an optional example, identifying the chain of attribution evidence, and the causal reasoning process for the root cause of the failure based on the chain of attribution evidence, specifically includes:

[0124] Step S251, Attribution Analysis: Attention maps are extracted using a multimodal model, and gradient attribution analysis is used to obtain the attribution score for each feature; SHAP is used to evaluate the feature contribution of high-value windows, providing another chain of evidence; then, according to the time window, the above attribution method is used to form a time-domain attribution accumulation to obtain long and short time-series attributions; finally, the attribution evidence is fused to obtain the final attribution confidence.

[0125] Step S252, Image Spatial Anomaly Localization: If there is an image with corresponding anomaly points, then image anomaly detection and identification are performed. At the same time, the abnormal data is converted into an image for anomaly identification. Then, it is aligned and merged with the temporal attribution to form joint evidence of time and image, thereby supplementing the attribution evidence chain.

[0126] Step S253: Construct a multimodal case retrieval system: Each case must contain at least the following information: device type, fault label, time-series feature vector, image embedding, text maintenance report, attribution evidence chain, disassembly conclusion, etc. Use RAG technology to search the historical case library for the current attribution evidence chain and corresponding data. If there are similar historical cases, add them to the attribution evidence chain.

[0127] Step S254, Causal Reasoning: Construct the attribution evidence chain generated in the previous steps into a Prompt containing scene, device, phenomenon, attribution evidence chain, and key data. Utilize the capabilities of a multimodal large model to perform causal reasoning and construct a text description that engineers can understand as well as control codes that DCS and other systems can recognize.

[0128] As an optional embodiment, after using a multimodal model and performing causal reasoning on multimodal features based on the guiding information generated by the attribution evidence chain to obtain the root cause of the fault of the equipment to be diagnosed, the method further includes: pushing linkage instructions (such as load reduction, switching to backup equipment) to the control system (DCS / PLC); and providing natural language diagnostic reports to maintenance personnel.

[0129] As an optional embodiment, after using a multimodal model and performing causal reasoning on the multimodal features based on the guiding information generated by the attribution evidence chain to obtain the root cause of the fault in the device to be diagnosed, the method further includes: obtaining feedback information on the root cause of the fault, and correcting the multimodal model based on the feedback information to achieve closed-loop management of the multimodal model.

[0130] As an optional example, obtaining feedback information on the root causes of failures and modifying the multimodal model based on this feedback information includes: closed-loop management can be achieved when human experts provide feedback on the algorithm's execution results or the current device status. The specific steps are as follows:

[0131] Step S261, Algorithm Alarm Feedback. This feedback is non-mandatory; human experts can provide feedback to the system when they wish, indicating whether the alarm is correct or incorrect, or whether the absence of an alarm is correct or incorrect. LLM is used to convert expert feedback into correct or incorrect labels for the data detection results at the time of feedback. This high-value data is then updated in the feature library. If the algorithm result is incorrect, this data is included in the online updated dataset. Once the dataset reaches the required size for updates, RLHF technology is used for online fine-tuning.

[0132] Step S262, Fault Disassembly Feedback. This feedback is mandatory. After the human expert disassembles and repairs the equipment, they need to upload the true cause of the fault or details of any misjudgment, along with the corresponding materials, to the system. First, the system uses LLM to format and organize the expert's feedback, and then injects the feedback information into the knowledge base as important search content for RAG. In subsequent diagnostics, similar alarms can be quickly identified and diagnosed.

[0133] Figure 3 This is a schematic diagram of a general equipment fault early warning, diagnosis, and linkage process based on a time series multimodal large model according to an embodiment of the present invention, such as... Figure 3 As shown, the method includes the following steps:

[0134] Step S31, Data Acquisition and Preprocessing.

[0135] Optionally, multimodal data (such as time series, images, text, and process parameters) from the equipment can be collected to construct a time-series multimodal data lake; and the multimodal data can be semi-automatically cleaned, aligned, normalized, and have missing values ​​processed.

[0136] Step S32, multimodal representation learning.

[0137] Optionally, for time series, dynamic features are extracted using a Transformer / temporal self-attention model; for images, spatial features are extracted using a CNN / visual Transformer; for text, semantic features are extracted using a pre-trained language model (such as BERT, GPT); for process data, graph neural networks (GNN) or structured coding analysis are used; finally, the processed data are aligned and fused in a high-dimensional shared latent space to obtain a multimodal representation.

[0138] Step S33, anomaly detection and fault warning.

[0139] Optionally, contrastive learning or probabilistic modeling (such as variational autoencoder VAE) can be used to reconstruct the current data; the prediction residuals can be used to predict the future state; when the deviation exceeds the threshold, or the state violates the import conditions, an anomaly alarm or an early warning can be triggered.

[0140] Step S34, root cause diagnosis.

[0141] Optionally, after an alarm / warning is issued, the model combines multimodal features to perform causal reasoning and fault location, specifically including: analyzing the attribution of key variables in the time series; locating abnormal areas in the image; performing template case queries in the feature case library; providing supplementary textual explanations; and finally outputting the fault type and possible causes.

[0142] Step S35: Handling and Coordination.

[0143] Optionally, push linkage commands (such as load reduction or switching to backup equipment) to the control system (DCS / PLC); and provide natural language diagnostic reports to maintenance personnel.

[0144] Step S36: Manual feedback collection.

[0145] Optionally, human feedback includes both non-mandatory and mandatory feedback.

[0146] For example, non-mandatory feedback: maintenance personnel can manually confirm algorithm alarms ("correct / false alarm / uncertain");

[0147] Mandatory feedback: If the equipment is disassembled for repair, the actual cause and location of the fault must be recorded.

[0148] Step S37, feedback-driven model update.

[0149] Optionally, feedback-driven model updates include: RAG (Retrieval Enhanced Generation): indexing new feedback as knowledge feature fragments into the fault feature case library for subsequent inference reference; RLHF (Human Feedback Reinforcement Learning): updating the reward / penalty signals on the diagnostic results to improve the model's decision preferences; supporting delayed supervision: for lagging disassembly feedback, the system will automatically backtrack and label historical data fragments and retrain; and using newly collected data and feedback for online fine-tuning or knowledge supplementation to enhance adaptability to unknown faults.

[0150] The embodiments described above in this application use a time-series Transformer as the core backbone, which has the ability to model cross-time dependencies; introduce a multimodal alignment mechanism to achieve data fusion through cross-attention or joint embedding space; use causal relationship learning to improve the interpretability of fault attribution; introduce a human feedback mechanism to improve the credibility of the model in complex scenarios; and design a linkage interface protocol to realize the connection with industrial control systems and operation and maintenance systems.

[0151] The embodiments described above in this application can still achieve accurate fault warning and diagnosis through data-driven approaches even in the absence of a complete knowledge base or insufficient mechanistic models; they can integrate multimodal inputs such as time series, images, text, and process data to improve diagnostic accuracy and generalization ability; they provide linkage capabilities to achieve a closed loop of warning → diagnosis → handling suggestions, supporting automatic linkage between control and maintenance systems; they provide a general methodology applicable to different equipment (such as pumps, compressors, heat exchangers, and boilers) and different industrial scenarios, reducing manual debugging; they can accept sparse and time-delayed knowledge feedback, improve credibility through human-machine collaboration, and efficiently update model knowledge.

[0152] As an alternative example, a large petrochemical plant has several critical pieces of equipment, including centrifugal pumps, compressors, heat exchangers, and boilers. To improve the safety and stability of equipment operation, the plant introduced the aforementioned general equipment fault early warning, diagnosis, and linkage system based on a time-series multimodal large-scale model. This system enables fault early warning and diagnosis of equipment in the chemical plant area based on a time-series multimodal large-scale model. The specific implementation process is as follows:

[0153] Step S1, Device data acquisition and storage. Specifically includes:

[0154] Sensor installation: Install temperature, pressure, flow, vibration and other sensors on the bearings and inlet / outlet of centrifugal pumps and compressors; arrange inlet / outlet temperature difference and flow meter sensors on heat exchangers; and configure flue gas temperature, oxygen content, temperature and pressure sensors on boilers.

[0155] Data acquisition: Sensors collect data in real time and upload it to the data lake management platform.

[0156] Manual recording: During daily inspections, maintenance personnel record equipment status, maintenance details, and operation logs, and upload them to the platform's data lake.

[0157] Image acquisition: Inspection workers regularly take infrared thermal images and images of fixed-point monitoring devices, and upload them to the system's data lake.

[0158] Step S2, data preprocessing. Specifically, this includes:

[0159] Data cleaning: (manual experience import + program processing) Remove invalid data during equipment shutdown and sensor startup processes, as well as data from faulty sensors.

[0160] Data normalization: (processing) Data of different dimensions (temperature, pressure, vibration, etc.) are standardized into the 0-1 range.

[0161] Data alignment: (Program processing) Multimodal data (time series, images, text) are structured into datasets based on their composition.

[0162] Step S3, multimodal large model pre-training. Specifically includes:

[0163] Data preparation: (Automatic program retrieval) Collect historical operating data of various devices, covering normal and abnormal states.

[0164] Model architecture: (Design of this invention) adopts the Transformer backbone, combined with temporal PatchTST, visual Transformer and text encoder, and introduces cross-attention fusion.

[0165] Pre-training (engineer training) employs a multi-task learning approach, including temporal prediction, anomaly detection and mask reconstruction tasks, and contrastive learning, to enhance the model's cross-modal feature extraction capabilities.

[0166] Step S4, feature library construction. This specifically includes:

[0167] Feature extraction: (Program processing) Use pre-trained models to extract normal operation features and form health status vectors for equipment such as pumps and compressors.

[0168] Feature storage: (Program processing) The features are stored in the database as a benchmark for real-time monitoring.

[0169] Step S5, fine-tuning the training. This specifically includes:

[0170] Data labeling: (Manual processing) Semi-automatic labeling is performed by combining expert experience and some alarm data.

[0171] Prompt Fine-tuning: (Engineer Training) Introduces learnable prompts for different devices to adapt to specific operating conditions.

[0172] Step S6, Real-time monitoring and anomaly detection. Specifically, this includes:

[0173] Anomaly Detection: (Program Processing) Real-time detection of device status for large models, including reconstruction errors, feature mismatches, and import condition filtering.

[0174] Anomaly warning: (Program processing) The large model predicts the equipment status in real time, including trend deviation, residual over-limit, feature mismatch, and import condition filtering.

[0175] Judgment Logic: (Program Processing) Error judgment adopts the adaptive quantile judgment method. For example, when identifying the working condition, the percentile threshold is obtained from the data under the working condition using graphic images. If the error is higher than the threshold, it is judged as abnormal. Feature comparison adopts cosine similarity pairing, with the threshold set to 0.95. If it is lower than the threshold, it is judged as abnormal. Condition filtering performs anomaly judgment based on the imported conditions.

[0176] Step S7, abnormal alarm handling and manual feedback. Specifically, this includes:

[0177] Alarm Classification: (Program Processing) Anomalies are classified into process alarms (temperature, pressure, etc.) and equipment alarms (motors, bearings, seals, etc.).

[0178] Human feedback mechanism:

[0179] Non-mandatory feedback: Maintenance personnel provide feedback on alarms as "correct / false alarm / uncertain".

[0180] Mandatory feedback: If the equipment is disassembled for repair, the actual cause of the fault must be uploaded.

[0181] Feedback-driven updates:

[0182] The system uses RAG to store feedback into a case library.

[0183] RLHF is used to optimize the large diagnostic model using reward / penalty methods.

[0184] Upon receiving feedback about the delayed disassembly, we retrospectively labeled and incrementally trained the historical data.

[0185] Step S8, coordination and handling. Specifically includes:

[0186] Linkage control: (Program processing) When the system diagnoses an abnormality in the centrifugal pump bearing, it automatically sends a "switch to standby pump" command to the DCS.

[0187] Recommended action: (Program generated) The system outputs a natural language diagnostic report, suggesting "check the quality of the lubricating oil or the condition of the bearing".

[0188] The embodiments described above, through the introduction of a fault early warning and diagnosis system based on a time-series multimodal large model, enable the factory to achieve a closed loop of multimodal fusion monitoring, early warning diagnosis, coordinated response, and manual feedback. This not only significantly improves the safety and stability of equipment operation but also enhances maintenance efficiency and production benefits, bringing significant economic value and management improvements to the factory. The system can monitor equipment operation, real-time monitoring the operating status of pumps, compressors, heat exchangers, boilers, and other equipment. It can detect abnormalities such as insufficient lubrication of pump bearings and compressor surge in advance, avoiding equipment downtime losses. It improves maintenance efficiency, allowing maintenance personnel to directly conduct inspections based on the system's diagnostic suggestions, reducing blind troubleshooting. Through manual feedback and delayed disassembly feedback, the system continuously optimizes diagnostic accuracy. It also improves production efficiency; a successful predictive alarm prevented a centrifugal pump downtime accident, reducing losses by hundreds of thousands of yuan. The system continuously learns, maintaining stable diagnostic capabilities over the long term and reducing maintenance costs.

[0189] According to an embodiment of the present invention, an embodiment of a device for diagnosing equipment faults is also provided. It should be noted that the device for diagnosing equipment faults can be used to execute the device fault diagnosis method in the embodiment of the present invention, and the device fault diagnosis method in the embodiment of the present invention can be executed in the device for diagnosing equipment faults.

[0190] Figure 4 This is a schematic diagram of a device for diagnosing equipment faults according to an embodiment of the present invention, such as... Figure 4 As shown, the device may include: an acquisition module 42 for acquiring multimodal data of the device to be diagnosed; a feature fusion module 44 for extracting multimodal features from the multimodal data using a pre-trained multimodal model and fusing the multimodal features into fused features; an attribution analysis module 46 for performing attribution analysis on the multimodal features using the multimodal model to obtain an attribution evidence chain when the device to be diagnosed is determined to have a fault based on the fused features, wherein the attribution analysis includes at least: performing attribution analysis on the temporal features in the multimodal features to obtain an attribution evidence chain, and supplementing the attribution evidence chain with at least one modal feature other than the temporal features in the multimodal features; and a root cause reasoning module 48 for performing causal reasoning on the multimodal features based on the guiding information generated by the attribution evidence chain using the multimodal model to obtain the root cause of the fault of the device to be diagnosed.

[0191] It should be noted that the acquisition module 42 in this embodiment can be used to execute step S102 in this application embodiment, the feature fusion module 44 in this embodiment can be used to execute step S104 in this application embodiment, the attribution analysis module 46 in this embodiment can be used to execute step S106 in this application embodiment, and the root cause reasoning module 48 in this embodiment can be used to execute step S108 in this application embodiment. The examples and application scenarios implemented by the above modules and corresponding steps are the same, but are not limited to the content disclosed in the above embodiments.

[0192] The embodiments described above acquire multimodal data of the device to be diagnosed, and use a pre-trained multimodal model to extract and fuse features from this data to form fused features for device fault diagnosis. Then, based on the determination that the device has a fault, a multimodal model is used to perform deep attribution analysis to form an attribution evidence chain. The attribution evidence chain is based on the attribution analysis of time-series features, supplemented by other modal features. Causal reasoning is then performed based on the guiding information generated by the attribution evidence chain to further analyze the multimodal features and derive the root cause diagnosis of the device fault. This overcomes the limitations of traditional single-modal fault diagnosis, effectively improves the accuracy and generalization ability of the diagnosis, and solves the technical problem of inaccurate fault diagnosis caused by the over-reliance on single-modal data in existing technologies.

[0193] As an optional embodiment, the apparatus further includes: a prediction inference module, used to extract multimodal features from multimodal data using a pre-trained multimodal model, and fuse the multimodal features into fused features, and then use the multimodal model to perform prediction inference on the multimodal features to obtain prediction data of the device to be diagnosed at a target time, wherein the target time is later than the acquisition time of the multimodal data, and the prediction inference is based at least on the temporal features and process features in the multimodal features, and combined with the image features and semantic features in the multimodal features, to infer the prediction features of the device to be diagnosed at the target time, and to determine the prediction data based on the prediction features, wherein the prediction features include: the multimodal features and fused features of the device to be diagnosed at the target time, and the prediction data includes: multimodal data reconstructed based on the multimodal features at the target time.

[0194] As an optional embodiment, the apparatus further includes: an acquisition submodule, configured to acquire multimodal sample data before extracting multimodal features from multimodal data using a pre-trained multimodal model and fusing the multimodal features into fused features, wherein the multimodal sample data includes single-modal sample data of multiple modalities; and a first learning submodule, configured to perform representation encoding learning on single-modal sample data of different modalities using a preset multimodal model to obtain a first model, wherein the preset multimodal model is the initial model of the multimodal model before training, and the representation encoding learning is used to enable the first model to learn to extract multimodal features from the multimodal data. The system comprises three sub-modules: a first learning module and a second learning module. The first learning module is used to perform fusion and alignment learning on the single-modal sample features of different modalities to obtain a second model. The single-modal sample features are extracted from the single-modal sample data using the first model. The fusion and alignment learning is used to enable the second model to learn to align and fuse the modal features of different modalities. The third learning module is used to perform feature mask reconstruction on the multimodal sample data using the second model to obtain a multimodal model. The feature mask reconstruction is used to enable the multimodal model to learn the dependencies between the modal data of different modalities. The dependencies are used at least for causal inference and predictive inference.

[0195] As an optional embodiment, the device further includes at least one of the following: a first determination submodule, configured to, when determining that the device to be diagnosed is faulty based on fusion features, perform a first-level determination on the multimodal data using a multimodal model to conduct attribution analysis on the multimodal features and obtain an attribution evidence chain, before doing so, use pre-set determination conditions, wherein the determination conditions are generated using a pre-set language model based on determination statements described in natural language, and if the result of the first-level determination indicates that the multimodal data does not meet the determination conditions, determine that the device to be diagnosed is faulty; and a second determination submodule, configured to perform a second-level determination on the reconstruction error of the multimodal data using a pre-set error threshold. The reconstruction error is defined as the difference between the reconstructed multimodal data based on multimodal features and the multimodal data. If the reconstruction error exceeds the error threshold in the second-level judgment, the device to be diagnosed is determined to be faulty. The third judgment submodule is used to perform a third-level judgment on the fused features using a pre-set feature library. The feature library includes a first database and a second database. The first database contains positive sample features of multiple non-faulty devices, and the second database contains negative sample features of multiple faulty devices. If the third-level judgment indicates that there are no matching positive sample features in the fused features, or that there are matching negative sample features in the fused features, the device to be diagnosed is determined to be faulty.

[0196] As an optional embodiment, the apparatus further includes: an acquisition subunit, configured to acquire multimodal historical data of the device during its historical operation phase before performing a three-level judgment on the fused features using a pre-defined feature library, wherein the multimodal historical data includes: multimodal positive sample data of non-faulty devices and multimodal negative sample data of faulty devices; and a fusion subunit, configured to perform multimodal fusion on the multimodal historical data using a multimodal model to obtain multimodal historical features, wherein the multimodal historical features include: multimodal positive sample features determined based on multimodal positive sample data and multimodal negative sample features obtained based on multimodal negative sample data. The clustering subunit is used to cluster multimodal historical features to obtain multiple cluster feature clusters. The cluster feature clusters include: positive sample feature clusters obtained by clustering multimodal positive sample features, and negative sample clusters obtained by clustering multimodal negative sample features. The aggregation subunit is used to perform feature mapping aggregation on each cluster feature cluster to obtain sample features and store the sample features in the feature library. The feature mapping aggregation is used to compress multiple multimodal historical features in the same cluster feature cluster into a single sample feature. The sample features include: positive sample features obtained based on positive sample feature clusters, and negative sample features obtained based on negative sample feature clusters.

[0197] As an optional embodiment, the attribution analysis module includes: a first analysis unit, used to perform attention analysis on temporal features in multimodal features to obtain an attention map, wherein the attention map represents the dependency relationship between different temporal features; a second analysis unit, used to perform gradient attribution analysis on the attention map to obtain the attribution score of each temporal feature in the attention map, wherein the attribution score is positively correlated with the feature contribution of the temporal feature; a determination unit, used to determine the attribution evidence chain based on the attribution score, wherein the attribution evidence chain is determined based on temporal features with attribution scores higher than a preset score threshold and the dependency relationship between temporal features; and a supplementation unit, used to select a target historical case that matches the attribution evidence chain from multiple historical cases recorded in the historical case library for supplementation, wherein each historical case records a corresponding historical evidence chain, and the target historical case is a historical case whose historical evidence chain matches the attribution evidence chain.

[0198] As an optional embodiment, the multimodal data includes at least: time-series data, image data, text data, and process data. The feature fusion module includes: a first feature extraction unit, used to extract time-series features from the time-series data using a first feature extraction model, wherein the first feature extraction model adopts a Transformer network architecture; a second feature extraction unit, used to extract image features from the image data using a second feature extraction model, wherein the second feature extraction model adopts a CNN network architecture or a ViT network architecture; a third feature extraction unit, used to extract semantic features from the text data using a third feature extraction model, wherein the third feature extraction model adopts a BERT network architecture or a GPT network architecture; a fourth feature extraction unit, used to extract process features from the process data using a fourth feature extraction model, wherein the fourth feature extraction model adopts a GNN network architecture; and a feature fusion unit, used to fuse the time-series features, image features, semantic features, and process features to obtain fused features.

[0199] Embodiments of the present invention can provide an electronic device, which can be a computer terminal, and the computer terminal can be any one of a group of computer terminal devices. Optionally, in this embodiment, the computer terminal can also be replaced by a mobile terminal or other terminal device.

[0200] Optionally, in this embodiment, the computer terminal may be located in at least one of a plurality of network devices in a computer network.

[0201] In this embodiment, the computer terminal described above can execute the program code for the following steps in the equipment fault diagnosis method: acquiring multimodal data of the device to be diagnosed; using a pre-trained multimodal model to extract multimodal features from the multimodal data and fusing the multimodal features into fused features; if it is determined that the device to be diagnosed has a fault based on the fused features, using the multimodal model to perform attribution analysis on the multimodal features to obtain an attribution evidence chain, wherein the attribution analysis includes at least: performing attribution analysis on the temporal features in the multimodal features to obtain an attribution evidence chain, and supplementing the attribution evidence chain with at least one modal feature other than the temporal features in the multimodal features; using the multimodal model to perform causal reasoning on the multimodal features based on the guiding information generated by the attribution evidence chain to obtain the root cause of the fault of the device to be diagnosed.

[0202] Figure 5 This is a structural block diagram of a computer terminal according to an embodiment of the present invention, such as... Figure 5 As shown, the computer terminal 50 may include one or more (only one is shown in the figure) processors 52 and memory 54.

[0203] The memory can be used to store software programs and modules, such as the program instructions / modules corresponding to the device fault diagnosis method and apparatus in this embodiment of the invention. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, thereby realizing the aforementioned device fault diagnosis method. The memory may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include memory remotely located relative to the processor, and these remote memories can be connected to the terminal 50 via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0204] The processor can access information and applications stored in memory via a transmission device to perform the following steps: acquiring multimodal data of the device to be diagnosed; extracting multimodal features from the multimodal data using a pre-trained multimodal model and fusing the multimodal features into fused features; if the device to be diagnosed is determined to be faulty based on the fused features, performing attribution analysis on the multimodal features using the multimodal model to obtain an attribution evidence chain, wherein the attribution analysis includes at least: performing attribution analysis on the temporal features in the multimodal features to obtain an attribution evidence chain, and supplementing the attribution evidence chain with at least one modal feature other than the temporal features in the multimodal features; and using the multimodal model to perform causal reasoning on the multimodal features based on the guiding information generated by the attribution evidence chain to obtain the root cause of the fault in the device to be diagnosed.

[0205] Optionally, the processor may also execute program code for the following steps: using a multimodal model to perform predictive inference on multimodal features to obtain predicted data of the device to be diagnosed at a target time, wherein the target time is later than the acquisition time of the multimodal data, the predictive inference is based at least on the temporal and process features in the multimodal features, and combined with the image and semantic features in the multimodal features to infer the predicted features of the device to be diagnosed at the target time, and to determine the predicted data based on the predicted features, wherein the predicted features include: the multimodal features and fusion features of the device to be diagnosed at the target time, and the predicted data includes: multimodal data reconstructed based on the multimodal features at the target time.

[0206] Optionally, the processor may also execute program code for the following steps: acquiring multimodal sample data, wherein the multimodal sample data includes single-modal sample data of multiple modalities; using a preset multimodal model to perform representation encoding learning on the single-modal sample data of different modalities to obtain a first model, wherein the preset multimodal model is the initial model before training the multimodal model, and the representation encoding learning is used to enable the first model to learn to extract multimodal features from the multimodal data; using the first model to perform fusion alignment learning on the single-modal sample features of different modalities to obtain a second model, wherein the single-modal sample features are extracted from the single-modal sample data using the first model, and the fusion alignment learning is used to enable the second model to learn to align and fuse the modal features of different modalities; using the second model to perform feature mask reconstruction on the multimodal sample data to obtain a multimodal model, wherein the feature mask reconstruction is used to enable the multimodal model to learn the dependencies between the modal data of different modalities, and the dependencies are used at least for causal inference and predictive inference.

[0207] Optionally, the processor may also execute program code for the following steps: First, using pre-defined judgment conditions, perform a first-level judgment on the multimodal data, wherein the judgment conditions are generated using a pre-defined language model based on judgment statements described in natural language; if the result of the first-level judgment indicates that the multimodal data does not meet the judgment conditions, determine that the device to be diagnosed is faulty; second, using a pre-defined error threshold, perform a second-level judgment on the reconstruction error of the multimodal data, where the reconstruction error is the difference between the reconstructed multimodal data based on the multimodal features and the original multimodal data; if the result of the second-level judgment indicates that the reconstruction error exceeds the error threshold, determine that the device to be diagnosed is faulty; third, using a pre-defined feature library, perform a third-level judgment on the fused features, wherein the feature library includes: a first database and a second database, the first database containing positive sample features of multiple non-faulty devices, and the second database containing negative sample features of multiple faulty devices; if the result of the third-level judgment indicates that the fused features do not have matching positive sample features, or that the fused features have matching negative sample features, determine that the device to be diagnosed is faulty.

[0208] Optionally, the processor may also execute program code for the following steps: acquiring multimodal historical data of the device during its historical operation phase, wherein the multimodal historical data includes: multimodal positive sample data of non-faulty devices and multimodal negative sample data of faulty devices; performing multimodal fusion on the multimodal historical data using a multimodal model to obtain multimodal historical features, wherein the multimodal historical features include: multimodal positive sample features determined based on multimodal positive sample data and multimodal negative sample features obtained based on multimodal negative sample data; and processing the multimodal historical features. Clustering is performed to obtain multiple cluster feature clusters, which include: positive sample feature clusters obtained by clustering multimodal positive sample features, and negative sample clusters obtained by clustering multimodal negative sample features. Feature mapping aggregation is performed on each cluster feature cluster to obtain sample features, and the sample features are stored in a feature library. Feature mapping aggregation is used to compress multiple multimodal historical features in the same cluster feature cluster into a single sample feature. The sample features include: positive sample features obtained based on positive sample feature clusters, and negative sample features obtained based on negative sample feature clusters.

[0209] Optionally, the processor may also execute program code for the following steps: performing attention analysis on temporal features in multimodal features to obtain an attention map, wherein the attention map represents the dependencies between different temporal features; performing gradient attribution analysis on the attention map to obtain an attribution score for each temporal feature in the attention map, wherein the attribution score is positively correlated with the feature contribution of the temporal feature; determining an attribution evidence chain based on the attribution score, wherein the attribution evidence chain is determined based on temporal features with attribution scores higher than a preset score threshold and the dependencies between temporal features; and selecting target historical cases that match the attribution evidence chain from multiple historical cases recorded in the historical case library for supplementation, wherein each historical case records a corresponding historical evidence chain, and the target historical case is a historical case whose historical evidence chain matches the attribution evidence chain.

[0210] Optionally, the multimodal data includes at least: time-series data, image data, text data, and process data. The processor may also execute program code for the following steps: using a first feature extraction model to extract time-series features from the time-series data, wherein the first feature extraction model uses a Transformer network architecture; using a second feature extraction model to extract image features from the image data, wherein the second feature extraction model uses a CNN network architecture or a ViT network architecture; using a third feature extraction model to extract semantic features from the text data, wherein the third feature extraction model uses a BERT network architecture or a GPT network architecture; using a fourth feature extraction model to extract process features from the process data, wherein the fourth feature extraction model uses a GNN network architecture; and fusing the time-series features, image features, semantic features, and process features to obtain fused features.

[0211] Those skilled in the art will understand that Figure 5 The structure shown is for illustrative purposes only. The computer terminal can also be a smartphone (such as an Android phone, an iOS phone, etc.), a tablet computer, a mobile internet device (MID), a PAD, and other terminal devices. Figure 5 This does not limit the structure of the aforementioned electronic device. For example, computer terminal 50 may also include components that are more advanced than those described above. Figure 5 The more or fewer components shown (such as network interfaces, display devices, etc.), or having the same Figure 5 The different configurations shown.

[0212] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a computer program instructing the hardware related to the terminal device. The computer program can be stored in a non-volatile medium, which may include: flash drive, read-only memory (ROM), random access memory (RAM), magnetic disk or optical disk, etc.

[0213] Embodiments of the present invention also provide a non-volatile storage medium. Optionally, in this embodiment, the aforementioned non-volatile storage medium can be used to store the program code executed by the device fault diagnosis method provided in the above embodiments.

[0214] Optionally, in this embodiment, the non-volatile storage medium may be located in any computer terminal in a group of computer terminals in a computer network, or in any mobile terminal in a group of mobile terminals.

[0215] Optionally, in this embodiment, the non-volatile storage medium is configured to store program code for performing the following steps: acquiring multimodal data of the device to be diagnosed; using a pre-trained multimodal model to extract multimodal features from the multimodal data and fusing the multimodal features into fused features; if it is determined that the device to be diagnosed has a fault based on the fused features, using the multimodal model to perform attribution analysis on the multimodal features to obtain an attribution evidence chain, wherein the attribution analysis includes at least: performing attribution analysis on the temporal features in the multimodal features to obtain an attribution evidence chain, and supplementing the attribution evidence chain with at least one modal feature other than the temporal features in the multimodal features; using the multimodal model to perform causal reasoning on the multimodal features based on the guiding information generated by the attribution evidence chain to obtain the root cause of the fault of the device to be diagnosed.

[0216] Optionally, in this embodiment, the non-volatile storage medium is configured to store program code for performing the following steps: using a multimodal model to perform predictive reasoning on multimodal features to obtain predicted data of the device to be diagnosed at a target time, wherein the target time is later than the acquisition time of the multimodal data, the predictive reasoning is based at least on the temporal and process features in the multimodal features, and combined with the image and semantic features in the multimodal features to infer the predicted features of the device to be diagnosed at the target time, and to determine the predicted data based on the predicted features, wherein the predicted features include: the multimodal features and fusion features of the device to be diagnosed at the target time, and the predicted data includes: multimodal data reconstructed based on the multimodal features at the target time.

[0217] Optionally, in this embodiment, the non-volatile storage medium is configured to store program code for performing the following steps: acquiring multimodal sample data, wherein the multimodal sample data includes single-modal sample data of multiple modalities; using a preset multimodal model to perform representation encoding learning on the single-modal sample data of different modalities to obtain a first model, wherein the preset multimodal model is the initial model before training the multimodal model, and the representation encoding learning is used to enable the first model to learn to extract multimodal features from the multimodal data; using the first model to perform fusion alignment learning on the single-modal sample features of different modalities to obtain a second model, wherein the single-modal sample features are extracted from the single-modal sample data using the first model, and the fusion alignment learning is used to enable the second model to learn to align and fuse the modal features of different modalities; using the second model to perform feature mask reconstruction on the multimodal sample data to obtain a multimodal model, wherein the feature mask reconstruction is used to enable the multimodal model to learn the dependencies between the modal data of different modalities, and the dependencies are used at least for causal inference and predictive inference.

[0218] Optionally, in this embodiment, the non-volatile storage medium is configured to store program code for performing the following steps: First-level judgment is performed on multimodal data using pre-defined judgment conditions, wherein the judgment conditions are generated using a pre-defined language model based on judgment statements described in natural language; if the result of the first-level judgment indicates that the multimodal data does not meet the judgment conditions, the device to be diagnosed is determined to be faulty; Second-level judgment is performed on the reconstruction error of multimodal data using a pre-defined error threshold, wherein the reconstruction error is the difference between the reconstructed multimodal data based on multimodal features and the multimodal data; if the result of the second-level judgment indicates that the reconstruction error exceeds the error threshold, the device to be diagnosed is determined to be faulty; Third-level judgment is performed on fused features using a pre-defined feature library, wherein the feature library includes: a first database and a second database, the first database containing positive sample features of multiple non-faulty devices, and the second database containing negative sample features of multiple faulty devices; if the result of the third-level judgment indicates that the fused features do not have matching positive sample features, or that the fused features have matching negative sample features, the device to be diagnosed is determined to be faulty.

[0219] Optionally, in this embodiment, the non-volatile storage medium is configured to store program code for performing the following steps: acquiring multimodal historical data of the device during its historical operation phase, wherein the multimodal historical data includes: multimodal positive sample data of non-faulty devices and multimodal negative sample data of faulty devices; performing multimodal fusion on the multimodal historical data using a multimodal model to obtain multimodal historical features, wherein the multimodal historical features include: multimodal positive sample features determined based on multimodal positive sample data and multimodal negative sample features obtained based on multimodal negative sample data; and performing multimodal fusion on the multimodal historical data. Modal history features are clustered to obtain multiple cluster feature clusters. The cluster feature clusters include: positive sample feature clusters obtained by clustering multimodal positive sample features, and negative sample clusters obtained by clustering multimodal negative sample features. Feature mapping aggregation is performed on each cluster feature cluster to obtain sample features, and the sample features are stored in the feature library. Feature mapping aggregation is used to compress multiple multimodal history features in the same cluster feature cluster into a single sample feature. The sample features include: positive sample features obtained based on positive sample feature clusters, and negative sample features obtained based on negative sample feature clusters.

[0220] Optionally, in this embodiment, the non-volatile storage medium is configured to store program code for performing the following steps: performing attention analysis on temporal features in multimodal features to obtain an attention map, wherein the attention map represents the dependencies between different temporal features; performing gradient attribution analysis on the attention map to obtain an attribution score for each temporal feature in the attention map, wherein the attribution score is positively correlated with the feature contribution of the temporal feature; determining an attribution evidence chain based on the attribution score, wherein the attribution evidence chain is determined based on temporal features with attribution scores higher than a preset score threshold and the dependencies between temporal features; selecting a target historical case that matches the attribution evidence chain from multiple historical cases recorded in the historical case library for supplementation, wherein each historical case records a corresponding historical evidence chain, and the target historical case is a historical case whose historical evidence chain matches the attribution evidence chain.

[0221] Optionally, in this embodiment, the multimodal data includes at least: time-series data, image data, text data, and process data. The non-volatile storage medium is configured to store program code for performing the following steps: using a first feature extraction model to extract time-series features from the time-series data, wherein the first feature extraction model employs a Transformer network architecture; using a second feature extraction model to extract image features from the image data, wherein the second feature extraction model employs a CNN network architecture or a ViT network architecture; using a third feature extraction model to extract semantic features from the text data, wherein the third feature extraction model employs a BERT network architecture or a GPT network architecture; using a fourth feature extraction model to extract process features from the process data, wherein the fourth feature extraction model employs a GNN network architecture; and fusing the time-series features, image features, semantic features, and process features to obtain fused features.

[0222] Embodiments of the present invention also provide a computer program product, including a computer program. Optionally, in this embodiment, when the computer program is executed by a processor, it implements the steps of the device fault diagnosis method provided in the above embodiments.

[0223] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0224] In the above embodiments of the present invention, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0225] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For instance, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.

[0226] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0227] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0228] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a non-volatile storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a non-volatile storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned non-volatile storage medium includes various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0229] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for diagnosing equipment faults, characterized in that, include: Acquire multimodal data of the device to be diagnosed; A pre-trained multimodal model is used to extract multimodal features from the multimodal data, and the multimodal features are fused into a fused feature. When it is determined that the device to be diagnosed is faulty based on the fusion features, the multimodal model is used to perform attribution analysis on the multimodal features to obtain an attribution evidence chain. The attribution analysis includes at least: performing attribution analysis on the temporal features in the multimodal features to obtain the attribution evidence chain, and supplementing the attribution evidence chain with at least one modal feature in the multimodal features other than the temporal features. Using the multimodal model, causal reasoning is performed on the multimodal features based on the guiding information generated by the attribution evidence chain to obtain the root cause of the failure of the device to be diagnosed.

2. The method according to claim 1, characterized in that, After extracting multimodal features from the multimodal data using a pre-trained multimodal model and fusing the multimodal features into the fused features, the method further includes: Using the multimodal model, predictive reasoning is performed on the multimodal features to obtain predicted data of the device to be diagnosed at a target time, wherein the target time is later than the acquisition time of the multimodal data. The predictive reasoning is based at least on the temporal and process features in the multimodal features, and combines the image and semantic features in the multimodal features to infer the predicted features of the device to be diagnosed at the target time, and determines the predicted data based on the predicted features. The predicted features include: the multimodal features and the fusion features of the device to be diagnosed at the target time, and the predicted data includes: the multimodal data reconstructed based on the multimodal features at the target time.

3. The method according to claim 1, characterized in that, Before extracting multimodal features from the multimodal data using a pre-trained multimodal model and fusing the multimodal features into the fused features, the method further includes: Acquire multimodal sample data, wherein the multimodal sample data includes single-modal sample data of multiple modalities; A first model is obtained by performing representation encoding learning on the single-modal sample data of different modalities using a preset multimodal model, wherein the preset multimodal model is the initial model of the multimodal model before training, and the representation encoding learning is used to enable the first model to learn to extract the multimodal features from the multimodal data; The first model is used to perform fusion alignment learning on the single-modal sample features of different modalities to obtain a second model, wherein the single-modal sample features are extracted from the single-modal sample data using the first model, and the fusion alignment learning is used to enable the second model to learn to align and fuse the modal features of different modalities; The second model is used to reconstruct the multimodal sample data using feature masks to obtain the multimodal model. The feature mask reconstruction is used to enable the multimodal model to learn the dependencies between different modal data, and the dependencies are used at least for causal reasoning and predictive reasoning.

4. The method according to claim 1, characterized in that, Before determining that the device to be diagnosed is faulty based on the fusion features, and before using the multimodal model to perform attribution analysis on the multimodal features to obtain the attribution evidence chain, the method further includes at least one of the following: Using pre-set judgment conditions, a first-level judgment is performed on the multimodal data. The judgment conditions are generated using a pre-set language model based on judgment statements described in natural language. If the result of the first-level judgment indicates that the multimodal data does not meet the judgment conditions, it is determined that the device to be diagnosed has a fault. Using a pre-set error threshold, a secondary determination is made on the reconstruction error of the multimodal data. The reconstruction error is the difference between the multimodal reconstructed data based on the multimodal features and the original multimodal data. If the result of the secondary determination indicates that the reconstruction error exceeds the error threshold, it is determined that the device to be diagnosed has a fault. Using a pre-defined feature library, the fused features are subjected to a three-level determination. The feature library includes a first database and a second database. The first database contains positive sample features of multiple non-faulty devices, and the second database contains negative sample features of multiple faulty devices. If the result of the three-level determination indicates that the fused features do not have matching positive sample features, or that the fused features have matching negative sample features, then the device to be diagnosed is determined to be faulty.

5. The method according to claim 4, characterized in that, Before performing a three-level determination on the fused features using a pre-defined feature library, the method further includes: Acquire multimodal historical data of the equipment during its historical operation phase, wherein the multimodal historical data includes: multimodal positive sample data of non-faulty equipment and multimodal negative sample data of faulty equipment; The multimodal model is used to perform multimodal fusion on the multimodal historical data to obtain multimodal historical features, wherein the multimodal historical features include: multimodal positive sample features determined based on the multimodal positive sample data, and multimodal negative sample features obtained based on the multimodal negative sample data; Clustering the multimodal historical features yields multiple cluster feature clusters, wherein the cluster feature clusters include: positive sample feature clusters obtained by clustering the multimodal positive sample features, and negative sample clusters obtained by clustering the multimodal negative sample features; For each cluster feature cluster, feature mapping aggregation is performed to obtain sample features, and the sample features are stored in the feature library. The feature mapping aggregation is used to compress multiple multimodal historical features in the same cluster feature cluster into a single sample feature. The sample feature includes: positive sample features obtained based on the positive sample feature cluster, and negative sample features obtained based on the negative sample feature cluster.

6. The method according to claim 1, characterized in that, If the fusion features indicate that the device to be diagnosed is faulty, the multimodal model is used to perform attribution analysis on the multimodal features, resulting in an attribution evidence chain including: Attention analysis is performed on the temporal features in the multimodal features to obtain an attention map, wherein the attention map represents the dependency relationship between different temporal features; Gradient attribution analysis is performed on the attention map to obtain the attribution score of each temporal feature in the attention map, wherein the attribution score is positively correlated with the feature contribution of the temporal feature; Based on the attribution score, the attribution evidence chain is determined, wherein the attribution evidence chain is determined based on the temporal features whose attribution scores are higher than a preset score threshold, and the dependencies between the temporal features; Among the multiple historical cases recorded in the historical case database, a target historical case that matches the attribution evidence chain is selected for supplementation. Each historical case records a corresponding historical evidence chain, and the target historical case is the historical case whose historical evidence chain matches the attribution evidence chain.

7. The method according to claim 1, characterized in that, The multimodal data includes at least: time-series data, image data, text data, and process data. A pre-trained multimodal model is used to extract multimodal features from the multimodal data, and the multimodal features are fused into a fused feature, including: A first feature extraction model is used to extract time-series features from the time-series data, wherein the first feature extraction model adopts a Transformer network architecture; A second feature extraction model is used to extract image features from the image data, wherein the second feature extraction model adopts a CNN network architecture or a ViT network architecture; A third feature extraction model is used to extract semantic features from the text data, wherein the third feature extraction model adopts the BERT network architecture or the GPT network architecture. A fourth feature extraction model is used to extract process features from the process data, wherein the fourth feature extraction model adopts a GNN network architecture; The temporal features, image features, semantic features, and process features are fused to obtain the fused features.

8. A diagnostic device for equipment malfunctions, characterized in that, include: The acquisition module is used to acquire multimodal data of the device to be diagnosed; The feature fusion module is used to extract multimodal features from the multimodal data using a pre-trained multimodal model, and fuse the multimodal features into a fused feature; The attribution analysis module is used to perform attribution analysis on the multimodal features based on the fusion features when it is determined that the device to be diagnosed has a fault, using the multimodal model to obtain an attribution evidence chain. The attribution analysis includes at least: performing attribution analysis on the temporal features in the multimodal features to obtain the attribution evidence chain, and supplementing the attribution evidence chain with at least one modal feature other than the temporal features in the multimodal features. The root cause reasoning module is used to perform causal reasoning on the multimodal features based on the guidance information generated by the attribution evidence chain using the multimodal model, so as to obtain the root cause of the failure of the device to be diagnosed.

9. An electronic device comprising a memory and a processor, characterized in that, The memory stores a computer program, and the processor is configured to execute the diagnostic method for any one of claims 1 to 7 through the computer program.

10. A computer program product comprising computer instructions, characterized in that, When the computer instructions are executed by the processor, they implement the steps of the diagnostic method for device faults according to any one of claims 1 to 7.

Images