Fault prediction method for multi-modal cross-attention enhancement graph neural network

Abstract

The invention relates to the technical field of fault prediction, and provides a fault prediction method for a multi-modal cross-attention enhancement graph neural network, and the method comprises the steps: collecting the data of equipment; performing adaptive enhancement and normalization processing on the image data, performing sliding window segmentation, standardization and noise suppression on a time sequence numerical signal, and performing semantic vectorization coding on a maintenance log text; extracting low-dimensional spatial features of image data by using the pruned lightweight convolutional neural network, connecting time sequence features of modeling time sequence numerical signals in series, extracting context semantic expressions of maintenance log texts, integrating the features into multi-modal data, alternately taking each modal feature as Query and the other modal features as Key and Value, and obtaining multi-modal data; calculating attention weight and performing weighted fusion; constructing a modal node weighted graph, and performing inter-node feature propagation through a multi-layer graph attention network; and a residual service life regression prediction module and a degradation level classification module are deployed in parallel, and fault early warning is completed through multi-task joint optimization.

Description

Technical Field

[0001] This invention relates to the field of fault prediction technology, and in particular to a fault prediction method using a multimodal cross-attention enhanced graph neural network. Background Technology

[0002] Existing multimodal fault prediction technologies mainly rely on simple feature concatenation or cascade fusion methods, directly superimposing multi-source data such as images, time-series signals, and text before inputting them into a single model for training. However, due to the fundamental differences in dimensionality, resolution, statistical distribution, and semantic representation among the various modal data, this "one-size-fits-all" fusion strategy fails to take into account their respective characteristics, resulting in the accumulation of a large amount of useless information and the dilution of critical signals. For example, high-dimensional spatial features extracted from the image branch often interfere with the semantic vectors of the text branch in the same feature space, causing minor vibration anomalies or extremely critical fault descriptions in maintenance logs to be submerged in redundant noise after fusion. Furthermore, existing methods statically assign fusion weights to each modality during the training phase and fix them long-term after the model goes live, lacking the ability to dynamically adjust the focus based on real-time operating conditions or sudden fault signals. As a result, when a sensor drift, camera obstruction, or text input error occurs in a certain path signal, the model cannot automatically reduce the weight of the failed mode, leading to a sharp decline in overall prediction performance and severely limiting the reliability and robustness of the system in complex and dynamic industrial environments.

[0003] Meanwhile, most current research only linearly combines different modalities at the feature or decision level, neglecting the potential structured dependencies and causal relationships between multimodal data. In real-world industrial scenarios, there is often a high degree of coupling between the image features of equipment microcracks, the high-frequency components of vibration signals, and fault descriptions in maintenance logs: "bearing wear" in the log often corresponds to surface cracks in the image, which is then corroborated by specific peaks in the vibration spectrum. However, existing methods lack tools for systematically modeling such non-Euclidean relationships, failing to capture cross-modal spatial and semantic associations, resulting in insufficient model interpretability and making it difficult for maintenance personnel to intuitively understand the causes of faults from the prediction results. Furthermore, industrial field fault samples are often scarce, and the dependence of deep models on large-scale labeled data makes them prone to severe overfitting under new fault types or new equipment scenarios, leading to a significant drop in accuracy during the testing phase. Once operating conditions change, existing models often require time-consuming and labor-intensive retraining or manual parameter tuning. The lack of model transfer and online fine-tuning mechanisms further exacerbates the deployment difficulty and maintenance costs in small-sample scenarios. In addition, field data is often accompanied by impulse noise, light fluctuations and input errors. Traditional preprocessing methods rely on manually defined filtering, interpolation or thresholding rules, which lack the ability to adaptively identify and complete noise types and missing patterns. This leads to drift or distortion of features after preprocessing, resulting in significant fluctuations in prediction results, which makes it difficult to meet the requirements of high reliability and low jitter for industrial online monitoring. Summary of the Invention

[0004] To address the aforementioned shortcomings, the present invention aims to propose a fault prediction method based on a multimodal cross-attention enhanced graph neural network. This method seeks to solve the problems of poor adaptability of multi-source heterogeneous data fusion, lack of structured relationships between modalities, and difficulty in small sample transfer in industrial scenarios by using multimodal cross-attention dynamic fusion and graph neural network modeling. This will achieve a synergistic improvement in fault prediction accuracy, robustness, and generalization ability.

[0005] To achieve this objective, the present invention adopts the following technical solution:

[0006] A fault prediction method across a multimodal attention-enhanced graph neural network, the fault prediction method comprising the following steps:

[0007] S1: Acquire image data, timing numerical signals, and maintenance log text from the acquisition device, and add spatiotemporal identifiers;

[0008] S2: Perform adaptive enhancement and normalization on image data, perform sliding window segmentation, standardization and noise suppression on time-series numerical signals, and perform semantic vectorization encoding on maintenance log text;

[0009] S3: Use a pruned lightweight convolutional neural network to extract low-dimensional spatial features of image data, use 1D-CNN and GRU in series to model the time series features of time series numerical signals, and extract the contextual semantic expression of maintenance log text based on Transformer and BERT architecture.

[0010] S4: Integrate low-dimensional spatial features, time series features, and contextual semantic representations into multimodal data. Alternately use each modal feature as the Query and the other modal features as the Key and Value. Calculate attention weights and perform weighted fusion to output cross-modal fusion features.

[0011] S5: Based on the spatial similarity, semantic association and prior working condition rules of the cross-modal fusion features, construct a modal node weighted graph, propagate features between nodes through a multi-layer graph attention network, automatically learn the coupling relationship between modalities, perform fast gradient updates on a small number of support samples based on the meta-learning framework, and back-optimize meta-parameters on the query set, automatically divide the support set and query set, update parameters through the inner loop and evaluate generalization ability through the outer loop;

[0012] S6: Parallel deployment of the remaining useful life regression prediction module and the degradation level classification module, and completion of fault early warning through multi-task joint optimization;

[0013] The lifetime regression prediction module and the degradation level classification module share the same fusion feature layer, and are optimized by weighted regression loss function and weighted classification loss function, respectively.

[0014] Preferably, the image data processing in step S2 includes:

[0015] The local contrast of the image data is enhanced by adaptive histogram equalization while preserving edge details. Then, random cropping and random rotation are performed on the image data to generate geometric transformation samples.

[0016] The image data is normalized for illumination based on the mean and variance standardization of the brightness channel, and the processed image is uniformly scaled to the input tensor of a preset size.

[0017] Step S3, the processing of image data, includes:

[0018] Image features are extracted using a pruned and optimized MobileNetV2 backbone network, which outputs a 256-dimensional spatial feature vector as the input to the image modality. The MobileNetV2 backbone network includes depthwise separable convolutional and linear bottleneck modules.

[0019] Preferably, the processing of the time-series numerical signal in step S2 includes:

[0020] Based on a configurable sliding window mechanism, the time-series numerical signal is divided into equal-length data segments according to a preset window length and step size;

[0021] Perform Z-score normalization, use a multimodal noise filtering gate to automatically identify and suppress short-term spikes and slow changing trends, and output the first sequence features;

[0022] Step S3, the processing of the time-series numerical signal, includes:

[0023] 1D-CNN uses two layers of convolution to extract local trend patterns and edge mutation features of the first sequence features, and connects to a two-layer GRU module to perform global temporal modeling of the first sequence features, extract long-term dependent features of device operating status, and map them to a unified 256-dimensional second temporal feature through a fully connected layer.

[0024] Preferably, the processing of the maintenance log text in step S2 includes:

[0025] The text is segmented into words using Chinese word segmentation tools, and pre-defined redundant information is removed. The pre-trained model BERT is used in conjunction with the maintenance corpus of the same domain for transfer learning, and each text is encoded into a semantic vector of fixed length.

[0026] Step S3, the processing of the maintenance log text, includes:

[0027] Semantic context modeling is performed using a BERT-Base model with a 12-layer Transformer encoder containing 768 hidden units. The BERT-Base model is fine-tuned based on a specific industrial equipment maintenance corpus. The 768-dimensional semantic representation is reduced to a 256-dimensional semantic embedding vector through a fully connected layer, while preserving the global semantic information of the location.

[0028] Preferably, step S4 includes:

[0029] A linear mapping module is constructed, which includes a low-dimensional spatial feature mapping layer W for image data. I b I Time-series feature mapping layer W for time-series numerical signals N ,b N and the context semantic representation mapping layer W for maintaining log text T ,b T ;

[0030] Input low-dimensional space features Time series features and contextual semantic expression Projected onto the same dimensional space R d middle:

[0031]

[0032] A cross-modal attention submodule is constructed, comprising h parallel attention heads, each containing a query linear mapping W. Q , bond linear mapping W K And value linear mapping W V All mappings reduce the input feature dimension to d. h =d / h;

[0033] In the i-th attention head, if the Query is a low-dimensional feature... Then time series features and contextual semantic representation Simultaneously, as Key / Value pairs, attention weights are calculated and weighted fused to satisfy the following relation:

[0034]

[0035] If the query is a time series feature Then low-dimensional space features and contextual semantic representation Simultaneously, as Key / Value pairs, attention weights are calculated and weighted fused to satisfy the following relation:

[0036]

[0037] If the query is a contextual semantic expression This combines low-dimensional spatial features and time series features. Simultaneously, as Key / Value pairs, attention weights are calculated and weighted fused to satisfy the following relation:

[0038]

[0039] in, This represents the weight of low-dimensional spatial features on time series features. This represents the weight of low-dimensional spatial features to the semantic representation of the context. This represents the feature fusion output regarding low-dimensional spatial features. This represents the weight of time series features on low-dimensional spatial features. This represents the weight of time series features on the semantic expression of the context. This represents the feature fusion output regarding time series features. This represents the weight of the contextual semantic representation on the features in the low-dimensional space. This represents the weight of the contextual semantic representation on the time series features. This represents the feature fusion output that expresses semantic meaning in context. and These represent the feature dimensions of low-dimensional spatial features, time-series features, and contextual semantic expressions, respectively.

[0040] Preferably, step S4 includes:

[0041] The output of each attention head i Concatenate along the feature dimension and pass through a linear mapping layer W. O Restored to d-dimensional, the multi-head fused output is obtained:

[0042]

[0043] Residual connections and layer normalization add the multi-head fusion output to the original mapped features and apply layer normalization (LayerNorm) to prevent gradient vanishing and stabilize the training process.

[0044]

[0045] The multimodal dynamic fusion unit ultimately outputs a three-way fused feature F′. I ,F′ N N,F′ T Each feature in the path contains both original modal information and complementary information from other modalities.

[0046] Preferably, step S5 includes:

[0047] F′ I ,F′N ,F′ T Abstracted into a set of nodes {v I ,v N ,v T Based on the Euclidean distance and cosine similarity between the three-way fused features, the edge weight w between any two nodes is calculated. ij The following relation is satisfied:

[0048]

[0049] Where α and β represent adjustable weight parameters used to control the relative influence of spatial similarity and semantic similarity in edge weights, F′ i T F′ j Let |F′| represent the fused feature vectors of the i-th and j-th modalities, respectively. i -F′ j || 2 The squared Euclidean distance between two modal features is represented by... It represents the cosine similarity of two feature vectors in the direction.

[0050] Preferably, step S5 includes:

[0051] Deploy a two-layer graph attention network to calculate the attention score between node pairs.

[0052]

[0053] in W is the representation of node i at layer l. (l) Here, a is a learnable weight matrix, and 'a' is the attention vector, which is then normalized to the attention weights:

[0054]

[0055] Where k represents the number of independent attention heads in each layer of the graph attention network;

[0056] The final representation of node i in the next layer is obtained by weighted summation of the information of its neighboring nodes:

[0057]

[0058] Where σ represents the activation function;

[0059] After processing through a two-layer graph attention network, the final set of node feature representations is obtained. The structured representations of different modalities are further integrated into a unified global feature representation. Satisfying the relation:

[0060]

[0061] Where λ i These are modal importance weights that can be trained or adaptively calculated by another attention mechanism based on the current operating conditions.

[0062] Preferably, step S5 includes:

[0063] Samples are extracted from the real-time multimodal fusion feature database and divided into a support set and a query set with high confidence labels according to the time sequence and the fault mode balance principle.

[0064] Perform inner gradient update: Let the current model parameters be θ, and calculate the loss function on the support set. And θ′ is obtained by updating the parameters through gradient descent: Where μ is the inner learning rate. This represents the gradient of the loss function with respect to the model parameters θ;

[0065] Perform outer gradient update: compute the loss function on the query set. And calculate The gradient of the initial parameter θ is used to update the parameter θ: Where ω is the outer learning rate. This represents the second-order gradient of the model parameter θ;

[0066] Repeat the inner and outer gradient updates multiple times to obtain the optimized parameters.

[0067] Add a mode consistency regularization term to the final loss function to prevent mode feature drift: Where λ is the regularization weight. This is a function that measures the consistency of modal distribution.

[0068] One of the above technical solutions has the following advantages or beneficial effects:

[0069] This invention designs a multi-head cross-modal dynamic attention fusion mechanism that can perform attention calculations on three modalities—image, time-series signal, and text—alternating between these modalities as queries. This achieves dynamic weighted fusion of features from different modalities. Residual connections and layer normalization ensure training stability, effectively improving the quality of feature fusion and the model's training performance. A relationship graph construction method based on spatial and semantic similarity automatically constructs a modal node weighted graph, fully exploring deep-level structural dependencies between modalities. A two-layer multi-head graph attention network is then used for deep fusion, with each layer containing multiple attention heads and employing skip connections to prevent "oversmoothing," significantly enhancing the model's ability to model complex multimodal coupling information and improving the accuracy of fault prediction. Based on few-sample adaptive meta-learning fine-tuning, it can quickly complete online fine-tuning of model parameters with only a very small number of new samples while maintaining high generalization ability. This system effectively addresses the scarcity of labeled samples in industrial settings, enhancing the model's rapid adaptability to new equipment and operating conditions. Through cross-modal noise filtering and missing feature completion modules, it automatically identifies and suppresses impulse noise and long-term drift noise, while adaptively completing missing data, improving the reliability of input features and enhancing the model's robustness in complex industrial environments. A lightweight multi-branch encoder design efficiently extracts features from different modal data and unifies the output dimension, facilitating subsequent fusion processing while reducing computational and storage costs. This enables the system to achieve real-time prediction and online early warning on resource-constrained edge devices. By employing parallel multi-task prediction heads and joint optimization strategies, the system deploys a remaining useful life regression prediction module and a degradation level classification module in parallel, achieving synergistic enhancement of quantitative prediction and qualitative assessment of equipment failures. This provides maintenance personnel with more comprehensive fault information, assisting in the development of more scientific and reasonable maintenance plans. Attached Figure Description

[0070] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0071] Figure 1 This is a flowchart of the fault prediction method for multimodal cross-attention enhanced graph neural networks provided in the first embodiment of the present invention;

[0072] Figure 2 This is a flowchart of the fault prediction method for multimodal cross-attention enhanced graph neural networks provided in the second embodiment of the present invention;

[0073] Figure 3This is a flowchart of step S4 of the fault prediction method for multimodal cross-attention enhanced graph neural networks provided in this embodiment of the invention;

[0074] Figure 4 This is a schematic diagram comparing the prediction accuracy of the multimodal cross-attention enhanced graph neural network fault prediction method provided in this embodiment of the invention with that of other models;

[0075] Figure 5 This is a schematic diagram illustrating the energy decay of the multimodal cross-attention enhanced graph neural network fault prediction method provided in this embodiment of the invention compared with other models under different small sample ratios.

[0076] Figure 6 This is a schematic diagram illustrating the changing trends of the multimodal cross-attention enhanced graph neural network fault prediction method provided in this embodiment of the invention and other models under different proportions of small samples. Detailed Implementation

[0077] Embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0078] In this invention, the terms "comprising," "including," or any other variations thereof are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0079] A fault prediction method using a multimodal attention-enhanced graph neural network, such as Figure 1 and Figure 2 As shown, the fault prediction method includes the following steps:

[0080] S1: Acquire image data, timing numerical signals, and maintenance log text from the acquisition device, and add spatiotemporal identifiers;

[0081] Image data refers to images of the equipment's appearance or thermal state taken using industrial cameras or thermal imagers, which can intuitively reflect the equipment's physical condition; time-series numerical signals are numerical information such as vibration, temperature, and pressure collected by sensors during the equipment's operation, reflecting the equipment's operating parameters that change over time; maintenance log text is textual information such as equipment maintenance status and fault descriptions recorded by inspection personnel, providing a basis for equipment maintenance history and human judgment; spatiotemporal identification refers to adding a Coordinated Universal Time (UTC) timestamp and equipment ID to the collected data. The timestamp is used to determine the time of data collection, and the equipment ID is used to distinguish data from different devices, thereby ensuring accurate data correspondence and synchronized processing.

[0082] In step S1, multimodal data of the equipment is comprehensively collected and spatiotemporally labeled to provide a rich information foundation for subsequent fault prediction. By simultaneously collecting different types of data, the operating status of the equipment can be reflected from multiple perspectives, while spatiotemporal labeling ensures the synchronization and traceability of the data, which is crucial for accurately analyzing the occurrence and development process of equipment faults.

[0083] S2: Perform adaptive enhancement and normalization on image data, perform sliding window segmentation, standardization and noise suppression on time-series numerical signals, and perform semantic vectorization encoding on maintenance log text;

[0084] Among them, adaptive enhancement is an image processing technique that automatically adjusts enhancement parameters based on local image features to improve image contrast and clarity, making key components and fault features of equipment more prominent; normalization is the process of mapping image pixel values ​​to a specific range (such as 0-1 or 0-255) to reduce the difference in illumination intensity between different images and ensure the consistency of image data; sliding window segmentation is a method of dividing time-series numerical signals into multiple segments according to a set time window length and step size, which can convert long-term continuous signals into multiple short-term segments, facilitating subsequent feature extraction and analysis; standardization is to perform Z-score transformation on each signal segment, that is, subtract the mean and divide by the standard deviation, so that the mean of the signal segment is 0 and the standard deviation is 1, eliminating the dimensional differences between different signals; noise suppression is to use specific algorithms (such as wavelet transform, filters, etc.) to remove noise components from the signal while retaining the main features of the signal; semantic vectorization coding is the process of converting text information into fixed-length numerical vectors using natural language processing technology, which can quantify the semantic information in the text, facilitating subsequent calculation and analysis.

[0085] Step S2 involves targeted preprocessing of the collected data of different types to improve data quality and applicability, making it suitable for subsequent feature extraction and fusion. Adaptive enhancement and normalization of image data can improve image quality, enhance target features, and eliminate the influence of factors such as illumination; sliding window segmentation, standardization, and noise suppression of time-series numerical signals can discretize continuous signals into equal-length segments, eliminating dimensional differences and noise interference, and highlighting the main features of the signal; semantic vectorization encoding of maintenance log text converts textual information into numerical feature vectors, facilitating computer processing and analysis, thus laying the foundation for subsequent feature extraction and fusion, such as... Figure 2 In step S2, image enhancement (adaptive histogram equalization - random enhancement - illumination correction), temporal numerical filtering (sliding window segmentation - Z-score normalization - noise filtering gate) and text cleaning and embedding (word segmentation and cleaning - BERT embedding - PCA dimensionality reduction) are included.

[0086] S3: Use a pruned lightweight convolutional neural network to extract low-dimensional spatial features of image data, use 1D-CNN and GRU in series to model the time series features of time series numerical signals, and extract the contextual semantic expression of maintenance log text based on Transformer and BERT architecture.

[0087] like Figure 2 As shown, step S3 corresponds to Figure 2 The parallel inference part of the model includes: The pruned lightweight convolutional neural network (CNN) refers to a CNN obtained by removing redundant connections and unimportant neurons through pruning techniques. It reduces computation and model parameter size while maintaining high feature extraction capabilities, making it suitable for deployment on resource-constrained edge devices; 1D-CNN is a one-dimensional convolutional neural network mainly used to process one-dimensional data such as time-series signals. It can extract local feature patterns, such as signal edges and abrupt changes, by sliding convolutional kernels along the time axis; GRU (Gated Recurrent Unit) is a simplified version of LSTM (Long Short-Term Memory) network, which can effectively alleviate the gradient vanishing problem when processing long sequence data and capture long-term dependencies in time-series signals; Transformer architecture is a deep learning model based on self-attention mechanism, which can dynamically adjust feature weights according to the positional relationship and semantic association of words in the text, highlighting the semantic contribution of important words; BERT (Bidirectional Encoder Representations from Transformers) is a pre-trained language model that learns a general semantic representation of language through unsupervised pre-training on large-scale text data, and can be fine-tuned to adapt to text processing tasks in specific domains.

[0088] Step S3 utilizes deep learning technology to extract features based on the characteristics of different modalities of data, in order to obtain key features that can characterize the operating status of the device. The pruned lightweight convolutional neural network can efficiently extract low-dimensional spatial features from image data, remove redundant information, and retain key targets and texture features in the image; the cascaded structure of 1D-CNN and GRU can respectively perform local feature extraction and global temporal modeling of time-series numerical signals, capturing short-term trends and long-term dependencies in the signal; the text processing method based on the Transformer and BERT architecture can fully mine and maintain the contextual semantic information of log text, converting semantic features such as words and phrases in the text into numerical vector representations, thereby providing expressive feature representations for subsequent multimodal fusion.

[0089] S4: Integrate low-dimensional spatial features, time series features, and contextual semantic representations into multimodal data. Alternately use each modal feature as the Query and the other modal features as the Key and Value. Calculate attention weights and perform weighted fusion to output cross-modal fusion features.

[0090] like Figure 3 As shown, Figure 3 This is a schematic diagram of the implementation process of step S4 in one embodiment. The principle of step S4 is based on a multi-head cross-modal attention mechanism. By alternately using each modal feature as the Query and the remaining modal features as the Key and Value, attention weights between modalities are calculated, thereby achieving dynamic weighted fusion of different modal features. This approach can adaptively adjust their fusion weights according to the importance and relevance of each modal feature under the current operating conditions, highlighting the contribution of key modal features while suppressing the influence of irrelevant or noisy modalities. This generates cross-modal fusion features with greater information value and discriminative power, providing a comprehensive feature representation for subsequent fault prediction.

[0091] Multimodal data refers to a collection that simultaneously contains heterogeneous information such as numerical values ​​(sensor time-series signals), images (equipment appearance or infrared thermal images), and text (inspection records, logs). The multi-head cross-modal attention mechanism refers to calculating attention weights between different modalities in multiple parallel attention heads, and then concatenating and fusing the outputs of each attention head to capture the intermodal relationships within different subspaces. Query refers to the modal features used to generate the query vector in the attention calculation, which are used for matching and interaction with other modal features (Key and Value). Key and Value refer to the modal features matched and weighted with the Query in the attention calculation. The Key is used to calculate the attention score, and the Value is obtained by weighted summation based on the attention weights to obtain the fused features. Attention weights represent the relative importance of each modal feature in the current fusion process, obtained by calculating the similarity between the Query and Key (such as dot product, cosine similarity, etc.), and are used to determine the weighting coefficients of the Value. The cross-modal fusion feature is a comprehensive feature vector obtained after weighted fusion through the attention mechanism. It integrates key information from different modalities and can more comprehensively reflect the operating status of the device.

[0092] S5: Based on the spatial similarity, semantic association and prior working condition rules of the cross-modal fusion features, construct a modal node weighted graph, propagate features between nodes through a multi-layer graph attention network, automatically learn the coupling relationship between modalities, perform fast gradient updates on a small number of support samples based on the meta-learning framework, and back-optimize meta-parameters on the query set, automatically divide the support set and query set, update parameters through the inner loop and evaluate generalization ability through the outer loop;

[0093] Among them, the modal node weighted graph refers to a graph structure constructed using different modal features as nodes, and calculating edge weights based on their spatial similarity, semantic association, and prior working condition rules, used to represent the complex relationships and interactions between modalities; the multi-layer graph attention network (GAT) is a neural network composed of multiple graph attention layers, where each layer calculates the weights between nodes and aggregates neighbor node information through an attention mechanism to achieve feature propagation and updating; inter-node feature propagation refers to the process in the graph structure where each node receives and integrates the feature information of its neighbor nodes according to the attention weights, thereby enriching its own feature representation; inter-modal coupling refers to the inherent correlation and interdependence between different modal data, such as the relationship between nodes in an image. The correspondence between fault features and fault phenomena described in the text; the meta-learning framework is a training method designed to improve the model's rapid learning ability. It learns a general model parameter initialization on multiple tasks, enabling the model to quickly adapt to new tasks with the support of a small number of new samples; support samples refer to small sample data used for rapid fine-tuning of the model, which represent the feature patterns of the new task; fast gradient update refers to optimization operations such as gradient descent performed on the support set to quickly adjust the model parameters to adapt to the new task; the query set is used to evaluate the model's generalization ability on new tasks, and backward optimization of meta-parameters is to adjust the initial parameters in the meta-learning process based on the loss on the query set to improve the model's rapid learning effect.

[0094] In step S5, on the one hand, a modal node weighted graph is constructed based on cross-modal fusion features, and a multi-layer graph attention network is used for feature propagation between nodes. By learning the coupling relationship between modalities, the expressive power and global consistency of the features are further enhanced. The graph structure can effectively model the complex relationships between modalities, while the graph attention network achieves efficient information propagation and fusion by adaptively learning the attention weights between nodes. On the other hand, based on the meta-learning framework, fast gradient updates are performed on a small number of support samples, and meta-parameters are optimized in reverse on the query set. By automatically dividing the support set and query set, and using an inner loop to update parameters and an outer loop to evaluate generalization ability, the model can quickly adapt to fault prediction tasks under new equipment and new working conditions, alleviate the overfitting problem in small sample scenarios, and improve the model's generalization ability and learning efficiency.

[0095] S6: Parallel deployment of the remaining useful life regression prediction module and the degradation level classification module, and completion of fault early warning through multi-task joint optimization;

[0096] The lifetime regression prediction module and the degradation level classification module share the same fusion feature layer, and are optimized by weighted regression loss function and weighted classification loss function, respectively.

[0097] The remaining useful life regression prediction module is a model used to predict the remaining usable time or cycle of equipment before the next maintenance. It estimates the degradation trend and remaining life of the equipment based on fused features and a regression algorithm. The degradation level classification module is a model used to classify the health status of equipment into different levels (such as normal, mild degradation, severe degradation, etc.). It achieves a qualitative assessment of the health status of the equipment through a classification algorithm. Multi-task joint optimization refers to the optimization process of training multiple related tasks (regression and classification) simultaneously. By sharing feature layers and joint loss functions, different tasks can influence and promote each other. The fused feature layer is a set of feature vectors obtained after multimodal data fusion. It serves as the common input for different task models and contains the equipment operating status information after multimodal data integration. The weighted regression loss function is a loss function that calculates the regression prediction error by weighting it. It can assign different weights to different error terms according to different business needs or data distribution characteristics to highlight the accuracy of key prediction intervals. The weighted classification loss function is a loss function that calculates the classification prediction error by weighting it. It can also set weights according to class imbalance or the importance of different fault levels to strengthen the model's ability to identify high-risk faults.

[0098] Step S6 involves deploying the remaining useful life regression prediction module and the degradation level classification module in parallel. A multi-task joint optimization strategy is used to ensure both modules share the same fusion feature layer. Simultaneously, optimization is performed using weighted regression loss functions and weighted classification loss functions, respectively. This achieves quantitative prediction (remaining useful life) and qualitative assessment (degradation level) of equipment failures, and the synergistic effect of both improves the accuracy and comprehensiveness of fault warnings. Multi-task learning, through shared feature representations, allows different tasks to complement and constrain each other, fully utilizing information from the data and improving the overall performance of the model.

[0099] During the validation process, a series of experimental evaluations were conducted based on the degradation simulation data of the steam generator. The experimental dataset consisted of 50 subfolders, each representing a complete operating cycle, including data in three modalities: image, text, and numerical. The image data were 224×224 .tiff format files, the text data recorded maintenance work order content, and the numerical data contained four operational features. The target variable was to predict the degradation level of the equipment at the next maintenance. The data was divided into a 70% training set, a 20% test set, and 10% for small-sample adaptive evaluation.

[0100] In terms of comparative method design, single-modal methods (CNN-LSTM: using only the image modality, extracting features through CNN and then connecting LSTM for time series modeling; BERT-MLP: using only the text modality, using pre-trained BERT to encode maintenance records, followed by a fully connected network; GRU-Regression: using only the numerical modality, using GRU to model multi-dimensional time series and perform regression prediction) and multi-modal fusion methods (MM-Transformer: traditional multi-modal Transformer fusing image, text, and numerical features) were set as comparison groups. Figure 4 As shown, the comparison aspects include MAE (Mean Absolute Error, lower is better); RMSE (Root Mean Square Error, lower is better); F1-Score (used to uniformly evaluate the boundary tasks of classification and regression, higher is better); and Accuracy (classification accuracy, higher is better). The MCA-GNN of this invention outperforms the comparison methods in all metrics on the full dataset, especially in degradation level regression prediction, exhibiting lower error and higher consistency. Specifically, the MAE of CNN-LSTM is 0.142, the MAE of MM-Transformer is 0.119, while the MAE of MCA-GNN is only 0.102; in terms of accuracy, MCA-GNN reaches 89.6%, significantly higher than other methods.

[0101] In small-sample adaptability testing, the proportion of the training set was gradually reduced (100%, 50%, 20%, 10%), and the MAE performance and performance degradation of different models were compared. Figure 5 and Figure 6 The results show that MCA-GNN exhibits the smallest performance degradation when the proportion of the training set decreases. For example, when the proportion of the training set is 10%, the performance degradation of MCA-GNN is only 4.9%, while that of CNN-LSTM and MM-Transformer is 30.3% and 40.3%, respectively. This result demonstrates that MCA-GNN has a significant advantage in small sample situations and can effectively adapt to insufficient sample conditions.

[0102] In summary, this invention demonstrates superior performance in multimodal fusion prediction tasks through a multimodal cross-attention augmented graph neural network method. It not only leads in prediction accuracy, stability, and interpretability but also exhibits good small-sample adaptability and industrial deployment feasibility. This method is suitable for health status modeling and early warning prediction of critical equipment, effectively improving equipment reliability, safety, and maintenance efficiency.

[0103] Preferably, the image data processing in step S2 includes:

[0104] The local contrast of the image data is enhanced by adaptive histogram equalization while preserving edge details. Then, random cropping and random rotation are performed on the image data to generate geometric transformation samples.

[0105] The image data is normalized for illumination based on the mean and variance standardization of the brightness channel, and the processed image is uniformly scaled to the input tensor of a preset size.

[0106] Step S3, the processing of image data, includes:

[0107] Image features are extracted using a pruned and optimized MobileNetV2 backbone network, which outputs a 256-dimensional spatial feature vector as the input to the image modality. The MobileNetV2 backbone network includes depthwise separable convolutional and linear bottleneck modules.

[0108] In step S2, adaptive histogram equalization enhances the local contrast of the image data while preserving edge details. This improves the visibility of target features in the image, making key components or potential fault features of the equipment more prominent. Preserving edge information also aids in the accurate identification of the equipment structure or damaged areas. Subsequently, random cropping and rotation of the image data generate geometric transformation samples, increasing data diversity. This exposes the model to image features from more different perspectives and local regions during training, thereby improving the model's robustness and generalization ability to image variations. It also avoids overfitting to specific image acquisition angles or positions, enhancing the model's adaptability to various image conditions in practical applications.

[0109] Normalizing image data based on the mean and variance of the luminance channels avoids inconsistencies in brightness and contrast between different images caused by differences in lighting conditions, ensuring the consistency of image data within a numerical range and providing a stable data foundation for subsequent feature extraction. After processing, the images are uniformly scaled to a preset input tensor size (reducing the image size to a uniform 224×224×3 RGB input tensor), ensuring all image data have a uniform size and format, meeting the dimensionality requirements of deep learning models for input data, and guaranteeing the standardization of data processing and the efficiency of model training.

[0110] In step S3, a pruned and optimized MobileNetV2 backbone network is used to extract image features. This backbone network includes depthwise separable convolutions and linear bottleneck modules, which can significantly reduce model parameters and computational cost while maintaining high feature extraction capabilities. Pruning optimization further removes redundant connections in the network, making the model more lightweight and suitable for deployment in resource-constrained industrial environments. The output 256-dimensional spatial feature vector is used as the image modality input. This dimension retains the key spatial feature information of the image while avoiding the computational burden and overfitting risk associated with high-dimensional data, providing an efficient and compact image feature representation for subsequent multimodal fusion.

[0111] Preferably, the processing of the time-series numerical signal in step S2 includes:

[0112] Based on a configurable sliding window mechanism, the time-series numerical signal is divided into equal-length data segments according to a preset window length and step size;

[0113] Perform Z-score normalization, use a multimodal noise filtering gate to automatically identify and suppress short-term spikes and slow changing trends, and output the first sequence features;

[0114] Step S3, the processing of the time-series numerical signal, includes:

[0115] 1D-CNN uses two layers of convolution to extract local trend patterns and edge mutation features of the first sequence features, and connects to a two-layer GRU module to perform global temporal modeling of the first sequence features, extract long-term dependent features of device operating status, and map them to a unified 256-dimensional second temporal feature through a fully connected layer.

[0116] In step S2, based on a configurable sliding window mechanism, the time-series numerical signal is divided into equal-length data segments according to a preset window length and step size (e.g., window length T = 60, step size S = 10). This step discretizes the continuous time-series signal into multiple segments with temporal locality, ensuring that each segment contains dynamic information of the signal within a certain time range. This facilitates subsequent analysis of the signal's short-term characteristics and provides the model with local feature patterns in the time dimension, helping to capture transient changes and periodic features in the signal. Performing Z-score normalization eliminates dimensional differences between signals from different sensors, unifying signal values ​​to a standard scale and enhancing the model's ability to compare and analyze different signals. Utilizing a multimodal noise filtering gate to automatically identify and suppress short-term spikes and slow changing trends effectively removes noise components from the signal, retains the signal's true characteristics, and the output first sequence features more accurately reflect the actual operating status of the equipment.

[0117] Step S3 employs a two-layer convolutional structure of 1D-CNN (in this embodiment, the kernel sizes are 5 and 3, and the number of channels are 64 and 128, respectively), which can extract local features from the first sequence features, capturing local trend patterns and edge abrupt changes in the signal, such as key features like transient impacts and periodic vibrations during equipment operation. These local features are of great significance for early fault diagnosis. A dual-layer GRU module (128 hidden units per layer) is then used to perform global temporal modeling of the first sequence features, capturing long-term dependent features of the equipment's operating state, such as long-term degradation trends and cumulative damage. These global features help predict the remaining service life and long-term health status of the equipment. The extracted features are mapped to a unified 256-dimensional second temporal feature through a fully connected layer. This not only unifies the feature dimensions, facilitating subsequent fusion with other modal features, but also further integrates and condenses feature information during the mapping process, highlighting the most valuable feature components for fault prediction.

[0118] Preferably, the processing of the maintenance log text in step S2 includes:

[0119] The text is segmented into words using Chinese word segmentation tools, and pre-defined redundant information is removed. The pre-trained model BERT is used in conjunction with the maintenance corpus of the same domain for transfer learning, and each text is encoded into a semantic vector of fixed length.

[0120] Step S3, the processing of the maintenance log text, includes:

[0121] Semantic context modeling is performed using a BERT-Base model with a 12-layer Transformer encoder containing 768 hidden units. The BERT-Base model is fine-tuned based on a specific industrial equipment maintenance corpus. The 768-dimensional semantic representation is reduced to a 256-dimensional semantic embedding vector through a fully connected layer, while preserving the global semantic information of the location.

[0122] In step S2, a Chinese word segmentation tool (such as Jieba) is used to segment the text into lexical units and remove pre-defined redundant information. This decomposes the original text into meaningful lexical units, removes irrelevant stop words, numbers, and punctuation marks, reduces noise interference, and retains the key semantic components of the text. Combining the pre-trained BERT model with maintenance corpora from the same domain for transfer learning fully leverages BERT's powerful semantic understanding capabilities and adapts it to the text characteristics of a specific industrial maintenance domain. Each text is encoded into a fixed-length semantic vector, achieving a numerical and semantic representation of text information, laying the foundation for subsequent feature extraction and fusion.

[0123] Step S3 further employs a BERT-Base model containing a 12-layer Transformer encoder for semantic context modeling. This model, through a multi-layer self-attention mechanism, can capture long-distance dependencies and contextual semantic information of words in the text. Based on a specific industrial equipment maintenance corpus, the model is fine-tuned to more accurately understand the semantic content of professional terminology, fault descriptions, and operating procedures in industrial equipment maintenance texts. A fully connected layer reduces the 768-dimensional semantic representation to a 256-dimensional semantic embedding vector. While compressing feature dimensions and reducing computational complexity, it retains the text's positional and global semantic information as a unified representation of text modality. This makes the generated semantic embedding vector more representative, effectively representing the deep semantic features of maintenance log texts and improving its information contribution in multimodal fault prediction.

[0124] Preferably, step S4 includes:

[0125] A linear mapping module is constructed, which includes a low-dimensional spatial feature mapping layer W for image data. I b l Time-series feature mapping layer W for time-series numerical signals N ,b N and the context semantic representation mapping layer W for maintaining log text T ,b T ;

[0126] Input low-dimensional space features Time series features and contextual semantic expression Projected onto the same dimensional space R d middle:

[0127]

[0128] A cross-modal attention submodule is constructed, comprising h parallel attention heads, each containing a query linear mapping W. Q , bond linear mapping W K And value linear mapping W V All mappings reduce the input feature dimension to d. h =d / h;

[0129] In the i-th attention head, if the Query is a low-dimensional feature... Then time series features and contextual semantic representation Simultaneously, as Key / Value pairs, attention weights are calculated and weighted fused to satisfy the following relation:

[0130]

[0131] If the query is a time series feature Then low-dimensional space features and contextual semantic representation Simultaneously, as Key / Value pairs, attention weights are calculated and weighted fused to satisfy the following relation:

[0132]

[0133] If the query is a contextual semantic expression This combines low-dimensional spatial features and time series features. Simultaneously, as Key / Value pairs, attention weights are calculated and weighted fused to satisfy the following relation:

[0134]

[0135] in, This represents the weight of low-dimensional spatial features on time series features. This represents the weight of low-dimensional spatial features to the semantic representation of the context. This represents the feature fusion output regarding low-dimensional spatial features. This represents the weight of time series features on low-dimensional spatial features. This represents the weight of time series features on the semantic expression of the context. This represents the feature fusion output regarding time series features. This represents the weight of the contextual semantic representation on the features in the low-dimensional space. This represents the weight of the contextual semantic representation on the time series features. This represents the feature fusion output that expresses semantic meaning in context. and These represent the feature dimensions of low-dimensional spatial features, time-series features, and contextual semantic expressions, respectively.

[0136] By constructing a linear mapping module and a cross-modal attention submodule, this system achieves dimensional unification and feature fusion based on an attention mechanism for features from different modalities. The linear mapping module projects image, temporal, and text features onto the same dimensional space, ensuring features are in the same space for effective comparison and fusion. The cross-modal attention submodule uses multiple parallel attention heads to calculate the correlation weights between different modalities, thereby achieving weighted feature fusion. This mechanism dynamically adjusts the contribution of different modalities to the final fusion result, highlighting key information and suppressing redundant information, thus improving the model's comprehensive understanding of multimodal data and the accuracy of fault prediction.

[0137] The linear mapping module ensures that features from different modalities reside in the same dimensional space, providing a foundation for subsequent feature interactions. Next, the cross-modal attention submodule calculates the correlation weights between different modalities using multiple attention heads, achieving dynamic weighted fusion of features. This mechanism automatically adjusts the contribution of each modality based on the correlation between features, highlighting key information and suppressing noise. For example, when image features are used as the query, the model automatically calculates their correlation weights with time-series and text features, and fuses the information from these features accordingly. This dynamic adjustment capability allows the model to adapt to different operating conditions and fault modes, improving the accuracy and robustness of fault prediction. Furthermore, through reasonable weight initialization, adjusting the number of attention heads, and optimizing the scaling factor, the model's fusion effect and training stability are further enhanced.

[0138] Preferably, step S4 includes:

[0139] The output of each attention head i Concatenate along the feature dimension and pass through a linear mapping layer W. O Restored to d-dimensional, the multi-head fused output is obtained:

[0140]

[0141] Residual connections and layer normalization add the multi-head fusion output to the original mapped features and apply layer normalization (LayerNorm) to prevent gradient vanishing and stabilize the training process.

[0142]

[0143] The multimodal dynamic fusion unit ultimately outputs a three-way fused feature F′. I ,F′ N N,F′ T Each feature in the path contains both original modal information and complementary information from other modalities.

[0144] The concatenation operation merges the output feature vectors of multiple attention heads along the feature dimension into a single vector. Its purpose is to integrate the diverse feature information captured by different attention heads, enabling the fused feature vector to contain a more comprehensive feature representation. Linear mapping layer W O This is a weight matrix used to transform the concatenated high-dimensional feature vectors back to the original dimension d. This helps reduce the feature dimensionality, prevent overfitting, and ensure consistency in subsequent processing. (Multi-head fusion output) It is a comprehensive representation of the outputs of all attention heads, aggregating feature information from different modalities and providing a foundation for subsequent feature fusion. Residual connections, by adding the multi-head fused output to the original mapped features, help the model retain key information from the original features while accelerating gradient flow during backpropagation, effectively mitigating the vanishing gradient problem. Layer normalization (LayerNorm) is used to standardize the feature vectors, giving them zero mean and unit variance. This helps stabilize model training, accelerates convergence, and improves the model's generalization ability. Original mapped features This is the feature vector after linear mapping, providing the foundation for the input of the cross-modal attention mechanism. The final output is the three-way fused feature F′. I ,F′ N ,F′ T Each path integrates original modal information and complementary features from other modalities, making the features more representative and informative.

[0145] Preferably, after completing the cross-attention fusion of the aforementioned multimodal features, in order to further explore the deep-level structural dependencies between multiple source modalities such as images, numerical values, and text, and to improve the semantic expression integrity and global consistency of the fused features, F′ is added during the modal node weighted graph construction stage. I ,F′ N ,F′ T Abstracted into a set of nodes {v I ,v N ,v T Based on the Euclidean distance and cosine similarity between the three-way fused features, the edge weight w between any two nodes is calculated. ij The following relation is satisfied:

[0146]

[0147] Where α and β represent adjustable weight parameters used to control the relative influence of spatial similarity and semantic similarity on edge weights. Let |F′| represent the fused feature vectors of the i-th and j-th modalities, respectively. i -F′ j || 2 The squared Euclidean distance between two modal features is represented by... It represents the cosine similarity of two feature vectors in the direction.

[0148] This edge weight calculation method not only preserves the correlation of modal features at the numerical and semantic levels, but also further strengthens the rationality of the coupling relationship by combining prior domain knowledge, thereby constructing a more expressive weighted undirected graph and laying a structural foundation for subsequent graph neural network modeling.

[0149] Preferably, step S5 includes:

[0150] In the graph neural network information fusion stage, a two-layer graph attention network is deployed to calculate the attention score between node pairs.

[0151]

[0152] in W is the representation of node i at layer l. (l) Here, a is a learnable weight matrix, and 'a' is the attention vector, which is then normalized to the attention weights:

[0153]

[0154] Where k represents the number of independent attention heads in each layer of the graph attention network;

[0155] The final representation of node i in the next layer is obtained by weighted summation of the information of its neighboring nodes:

[0156]

[0157] Here, σ represents the activation function; through multi-layer stacking, each node can absorb contextual information from a larger graph domain, achieving feature enhancement from local to global. In addition, to avoid the "over-smoothing" problem caused by multi-layer graph convolution, a skip connection mechanism is introduced to ensure that shallow information is not forgotten in deep networks, enhancing the inter-layer differences and expressive diversity of the final features.

[0158] After processing through a two-layer graph attention network, the final set of node feature representations is obtained. The structured representations of different modalities are further integrated into a unified global feature representation. Satisfying the relation:

[0159]

[0160] Where λ i These are modal importance weights that can be trained or adaptively calculated by another attention mechanism based on the current working conditions, thereby enabling more flexible modal fusion.

[0161] Among them, skip connection mechanism is a deep learning technique that directly connects the output of the previous layer to the next layer, allowing gradients to flow more smoothly during backpropagation, thereby alleviating the gradient vanishing problem. LeakyReLU (Leaky Rectified Linear Unit) is an activation function designed to solve the "neuron death" problem in the standard ReLU activation function.

[0162] Preferably, step S5 includes:

[0163] Samples are extracted from the real-time multimodal fusion feature database and divided into a support set and a query set with high confidence labels according to the time sequence and the fault mode balance principle.

[0164] Perform inner gradient update: Let the current model parameters be θ, and calculate the loss function on the support set. And θ′ is obtained by updating the parameters through gradient descent: Where μ is the inner learning rate. This represents the gradient of the loss function with respect to the model parameters θ;

[0165] Perform outer gradient update: compute the loss function on the query set. And calculate The gradient of the initial parameter θ is used to update the parameter θ: Where ω is the outer learning rate. This represents the second-order gradient of the model parameter θ;

[0166] Repeat the inner and outer gradient updates multiple times to obtain the optimized parameters.

[0167] Add a mode consistency regularization term to the final loss function to prevent mode feature drift: Where λ is the regularization weight. This is a function that measures the consistency of modal distribution.

[0168] Specifically, based on the Model-Independent Meta-Learning (MAML) framework, samples are extracted from a real-time multimodal fusion feature database. The support set and query set are divided according to chronological order and fault mode balance principles to ensure the representativeness and reliability of the samples. Inner gradient updates utilize the support set for rapid parameter adjustment, capturing fault features of new samples while preserving the original model structure. Outer gradient updates evaluate and further optimize the model's generalization ability based on the query set. After multiple iterations, the optimized parameters possess strong transferability, enabling rapid adaptation to new tasks. A modality consistency regularization term is added to the final loss function to prevent modal feature drift, ensuring the model maintains the stability of intermodal collaborative expression while converging rapidly. The entire unit runs in parallel with the main prediction process, without affecting online inference, and automatically switches to parameter updates after fine-tuning. This achieves the goal of fault prediction with few samples, fast response, and high accuracy, aiming to solve the problem of scarce labeled samples in industrial settings caused by equipment updates, sensor additions or removals, or sudden changes in operating conditions.

[0169] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "illustrative embodiment," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0170] Although embodiments of the invention have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the claims and their equivalents.

Claims

1. A fault prediction method using a multimodal attention-enhanced graph neural network, characterized in that, The fault prediction method includes the following steps: S1: Acquire image data, timing numerical signals, and maintenance log text from the acquisition device, and add spatiotemporal identifiers; S2: Perform adaptive enhancement and normalization on image data, perform sliding window segmentation, standardization and noise suppression on time-series numerical signals, and perform semantic vectorization encoding on maintenance log text; S3: Use a pruned lightweight convolutional neural network to extract low-dimensional spatial features of image data, use 1D-CNN and GRU in series to model the time series features of time series numerical signals, and extract the contextual semantic expression of maintenance log text based on Transformer and BERT architecture. S4: Integrate low-dimensional spatial features, time series features, and contextual semantic representations into multimodal data. Alternately use each modal feature as the Query and the other modal features as the Key and Value. Calculate attention weights and perform weighted fusion to output cross-modal fusion features. S5: Based on the spatial similarity, semantic association and prior working condition rules of the cross-modal fusion features, construct a modal node weighted graph, propagate features between nodes through a multi-layer graph attention network, automatically learn the coupling relationship between modalities, perform fast gradient updates on a small number of support samples based on the meta-learning framework, and back-optimize meta-parameters on the query set, automatically divide the support set and query set, update parameters through the inner loop and evaluate generalization ability through the outer loop; S6: Parallel deployment of the remaining useful life regression prediction module and the degradation level classification module, and completion of fault early warning through multi-task joint optimization; The lifetime regression prediction module and the degradation level classification module share the same fusion feature layer, and are optimized by weighted regression loss function and weighted classification loss function, respectively.

2. The fault prediction method of multimodal cross-attention augmented graph neural network according to claim 1, characterized in that, The image data processing in step S2 includes: The local contrast of the image data is enhanced by adaptive histogram equalization while preserving edge details. Then, random cropping and random rotation are performed on the image data to generate geometric transformation samples. The image data is normalized for illumination based on the mean and variance standardization of the brightness channel, and the processed image is uniformly scaled to the input tensor of a preset size. Step S3, the processing of image data, includes: Image features are extracted using a pruned and optimized MobileNetV2 backbone network, which outputs a 256-dimensional spatial feature vector as the input to the image modality. The MobileNetV2 backbone network includes depthwise separable convolutional and linear bottleneck modules.

3. The fault prediction method of multimodal cross-attention augmented graph neural network according to claim 1, characterized in that, The processing of the time-series numerical signal in step S2 includes: Based on a configurable sliding window mechanism, the time-series numerical signal is divided into equal-length data segments according to a preset window length and step size; Perform Z-score normalization, use a multimodal noise filtering gate to automatically identify and suppress short-term spikes and slow changing trends, and output the first sequence features; Step S3, the processing of the time-series numerical signal, includes: 1D-CNN uses two layers of convolution to extract local trend patterns and edge mutation features of the first sequence features, and connects to a two-layer GRU module to perform global temporal modeling of the first sequence features, extract long-term dependent features of device operating status, and map them to a unified 256-dimensional second temporal feature through a fully connected layer.

4. The fault prediction method of multimodal cross-attention augmented graph neural network according to claim 1, characterized in that, The processing of the maintenance log text in step S2 includes: The text is segmented into words using Chinese word segmentation tools, and pre-defined redundant information is removed. The pre-trained model BERT is used in conjunction with the maintenance corpus of the same domain for transfer learning, and each text is encoded into a semantic vector of fixed length. Step S3, the processing of the maintenance log text, includes: Semantic context modeling is performed using a BERT-Base model with a 12-layer Transformer encoder containing 768 hidden units. The BERT-Base model is fine-tuned based on a specific industrial equipment maintenance corpus. The 768-dimensional semantic representation is reduced to a 256-dimensional semantic embedding vector through a fully connected layer, while preserving the global semantic information of the location.

5. The fault prediction method of multimodal cross-attention augmented graph neural network according to claim 1, characterized in that, Step S4 includes: A linear mapping module is constructed, which includes a low-dimensional spatial feature mapping layer W for image data. I b I Time-series feature mapping layer W for time-series numerical signals N ,b N and the context semantic representation mapping layer W for maintaining log text T ,b T ; Input low-dimensional space features Time series features and contextual semantic expression Projected onto the same dimensional space R d middle: A cross-modal attention submodule is constructed, comprising h parallel attention heads, each containing a query linear mapping W. Q , bond linear mapping W K And value linear mapping W V All mappings reduce the input feature dimension to d. h =d / h; In the i-th attention head, if the Query is a low-dimensional feature... Then time series features and contextual semantic representation Simultaneously, as Key / Value pairs, attention weights are calculated and weighted fused to satisfy the following relation: If the query is a time series feature Then low-dimensional space features and contextual semantic representation Simultaneously, as Key / Value pairs, attention weights are calculated and weighted fused to satisfy the following relation: If the query is a contextual semantic expression This combines low-dimensional spatial features and time series features. Simultaneously, as Key / Value pairs, attention weights are calculated and weighted fused to satisfy the following relation: in, This represents the weight of low-dimensional spatial features on time series features. This represents the weight of low-dimensional spatial features to the semantic representation of the context. This represents the feature fusion output regarding low-dimensional spatial features. This represents the weight of time series features on low-dimensional spatial features. This represents the weight of time series features on the semantic expression of the context. This represents the feature fusion output regarding time series features. This represents the weight of the contextual semantic representation on the features in the low-dimensional space. This represents the weight of the contextual semantic representation on the time series features. This represents the feature fusion output that expresses semantic meaning in context. and These represent the feature dimensions of low-dimensional spatial features, time-series features, and contextual semantic expressions, respectively.

6. The fault prediction method of multimodal cross-attention augmented graph neural network according to claim 5, characterized in that, Step S4 includes: The output of each attention head i Concatenate along the feature dimension and pass through a linear mapping layer W. O Restored to d-dimensional, the multi-head fused output is obtained: Residual connections and layer normalization add the multi-head fusion output to the original mapped features and apply layer normalization (LayerNorm) to prevent gradient vanishing and stabilize the training process. The multimodal dynamic fusion unit ultimately outputs a three-way fused feature F′. I ,F′ N ,F′ T Each feature in the path contains both original modal information and complementary information from other modalities.

7. The fault prediction method of multimodal cross-attention augmented graph neural network according to claim 6, characterized in that, Step S5 includes: F I ′ ,F N ′ ,F T ′ Abstracted into a set of nodes {v I ,v N ,v T Based on the Euclidean distance and cosine similarity between the three-way fused features, the edge weight w between any two nodes is calculated. ij The following relation is satisfied: Where α and β represent adjustable weight parameters used to control the relative influence of spatial similarity and semantic similarity on edge weights. Let |F′| represent the fused feature vectors of the i-th and j-th modalities, respectively. i -F′ j || 2 The squared Euclidean distance between two modal features is represented by the following expression: It represents the cosine similarity of two feature vectors in the direction.

8. The fault prediction method of multimodal cross-attention augmented graph neural network according to claim 7, characterized in that, Step S5 includes: Deploy a two-layer graph attention network to calculate the attention score between node pairs. in W is the representation of node i at layer l. (l) Here, a is a learnable weight matrix, and 'a' is the attention vector, which is then normalized to the attention weights: Where k represents the number of independent attention heads in each layer of the graph attention network; The final representation of node i in the next layer is obtained by weighted summation of the information of its neighboring nodes: Where σ represents the activation function; After processing through a two-layer graph attention network, the final set of node feature representations is obtained. The structured representations of different modalities are further integrated into a unified global feature representation. Satisfying the relation: Where λ i These are modal importance weights that can be trained or adaptively calculated by another attention mechanism based on the current operating conditions.

9. The fault prediction method of multimodal cross-attention augmented graph neural network according to claim 8, characterized in that, Step S5 includes: Samples are extracted from the real-time multimodal fusion feature database and divided into a support set and a query set with high confidence labels according to the time sequence and the fault mode balance principle. Perform inner gradient update: Let the current model parameters be θ, and calculate the loss function on the support set. And obtain θ by updating the parameters through gradient descent. ′ : Where μ is the inner learning rate. This represents the gradient of the loss function with respect to the model parameters θ; Perform outer gradient update: compute the loss function on the query set. And calculate The gradient of the initial parameter θ is used to update the parameter θ: Where ω is the outer learning rate. This represents the second-order gradient of the model parameter θ; Repeat the inner and outer gradient updates multiple times to obtain the optimized parameters. Add a mode consistency regularization term to the final loss function to prevent mode feature drift: Where λ is the regularization weight. This is a function used to measure the consistency of modal distribution.

Images