Dual-mode GAF encoding and cross-modal fusion fault diagnosis method and system

Abstract

This invention relates to the field of fault diagnosis technology, specifically to a fault diagnosis method and system based on dual-modal GAF coding and cross-modal fusion. The method includes the following steps: acquiring vibration signals; generating GASF and GADF images with complementary physical features using GASF and FFT-based GADF methods respectively; extracting deep feature sequences using two independent feature extraction networks, and then enhancing and fusing them. During fusion, a monotonically decreasing constraint is used to progressively evolve the model from shallow global exploration to deep local focus; outputting fault diagnosis results based on the fused features, and simultaneously outputting three dimensions of visual evidence: a Grad-CAM heatmap of the coding layer and its correspondence with the signal's physical features; a heatmap of attention weights in the feature layer and CKA values; and the distribution of modal fusion weights in the decision layer. This invention, through multi-level interpretability analysis, provides complete and traceable evidence for the diagnostic results, making it applicable to safety-critical scenarios such as aero-engines and wind turbine gearboxes.

Description

Technical Field

[0001] This invention relates to the field of fault diagnosis technology, specifically to a fault diagnosis method and system that combines bimodal GAF coding with cross-modal fusion. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] Rotating machinery is core equipment in critical fields such as aerospace and energy, and its fault diagnosis is of great significance to ensuring operational safety. Currently, deep learning methods are widely used in the field of rotating machinery fault diagnosis due to their powerful automatic feature learning capabilities. However, existing deep learning diagnostic models generally suffer from the "black box" problem: the internal feature extraction and decision-making processes of the model lack interpretability, and technicians cannot understand the basis for the model's judgment or which features play a key role.

[0004] This problem is particularly prominent in safety-critical scenarios. For example, in equipment used in aircraft engines and nuclear power plants, if the model only outputs a fault category label without providing traceable diagnostic evidence, technicians find it difficult to trust the model's judgment and cannot review or confirm decisions. Existing interpretability studies mostly present single-level feature visualizations, such as classifier activation heatmaps, lacking a systematic explanation of the contribution mechanisms of each mode in the multimodal fusion process, and failing to establish a complete causal reasoning chain from the original vibration signal to the final classification result. Summary of the Invention

[0005] This invention provides a fault diagnosis method and system that combines bimodal GAF coding with cross-modal fusion. By assigning physical meaning to the input through bimodal GAF coding, it uses graded temperature-controlled cross-modal attention to simulate the cognitive process of "global exploration → local focus" and simultaneously outputs three-dimensional interpretable evidence of the coding layer, feature layer, and decision layer, forming a complete and traceable decision chain from signal to diagnosis.

[0006] To achieve the above objectives, the present invention adopts the following technical solution:

[0007] The first aspect of this invention provides a fault diagnosis method based on bimodal GAF coding and cross-modal fusion, comprising the following steps:

[0008] Vibration signals of rotating machinery are acquired, and time-domain encoding is performed using the GASF method and frequency-domain encoding using the FFT-based GADF method, respectively, to generate GASF and GADF images with complementary physical features.

[0009] Two feature extraction networks with identical structures but independent parameters are used to extract deep feature sequences from GASF and GADF images, respectively. The extracted deep feature sequences are then enhanced and fused. During the enhancement process, a monotonically decreasing constraint is used to make the model evolve progressively from shallow global exploration to deep local focus.

[0010] Based on the fused features, the fault diagnosis results are output, along with three dimensions of visual evidence, including:

[0011] In the coding layer, Grad-CAM activation heatmaps are generated for GASF and GADF images respectively. The decision-making basis area is highlighted on the GAF image, and the physical correspondence between the activation area of ​​the heatmap and the time-domain periodicity and frequency-domain fault characteristic frequency of the vibration signal is established.

[0012] At the feature layer, a heatmap of the attention weight matrix during deep feature sequence fusion is output, and the center kernel aligned CKA is used to quantitatively evaluate the feature space similarity between modalities. The output CKA values ​​are used to prove that substantial feature interaction is achieved between modalities.

[0013] At the decision level, the modality fusion weight distribution during deep feature sequence fusion is output;

[0014] The fault diagnosis results, together with the visual evidence from the three dimensions, constitute a complete diagnostic report and are output.

[0015] Furthermore, the GASF method includes: normalizing the time-domain vibration signal, obtaining the time-domain angle parameters through inverse cosine transform, and constructing matrix elements to capture the time-domain periodic evolution law and global steady-state correlation of the signal.

[0016] The GADF method includes: performing a fast Fourier transform on the original vibration signal to extract the amplitude spectrum of the positive frequency component, normalizing it to obtain the frequency domain angle parameters, constructing matrix elements, and using them to enhance the local changes and modulation sidebands at the fault characteristic frequencies.

[0017] Furthermore, the feature extraction network adopts a ResNet structure, including an initial convolutional layer, four residual layer groups, and an adaptive average pooling layer. The initial convolutional layer uses a 7×7 convolutional kernel with a stride of 2. The number of channels in the four residual layer groups are 32, 64, 128, and 256, respectively, and each group contains 2 residual blocks. After adaptive average pooling, the output is a deep feature sequence flattened to 49×256.

[0018] Furthermore, the model is progressively evolved from shallow global exploration to deep local focus through monotonically decreasing constraints. This is achieved by employing a three-layer serial bidirectional cross-modal attention mechanism, specifically by setting three temperature parameters that correspond to the three-layer cross-modal attention module and satisfy monotonically decreasing constraints. , ;in, To make the first layer of attention distribution more uniform in order to cover a wide range of cross-modal global correlations, This sharpens the third layer of attention distribution to accurately locate cross-modal local features. A balance is struck between the two; the temperature parameters of each layer are fixed hyperparameters preset before training and are not updated during the training process.

[0019] Furthermore, the extracted deep feature sequences are enhanced and fused, including a category-aware adaptive fusion step, which involves performing global average pooling on the enhanced dual-modal features to obtain global feature vectors, and inputting the concatenated vectors into a category predictor to obtain the fault category probability distribution. ;Utilizing a learnable class weight matrix The fusion weights are obtained by multiplying the weights by the probability distribution. The fusion coefficients of the GASF and GADF modes were obtained by softmax normalization. Ultimately, adaptive fusion is achieved through weighted summation; the category weight matrix is ​​automatically learned through backpropagation, and the modality fusion weight distribution is the fusion coefficient. .

[0020] Furthermore, the training phase employs the following strategies: Mixup data augmentation is introduced into the current batch with a 50% probability, and virtual training samples are generated by linear interpolation of samples within the batch; standard cross-entropy loss or Mixup loss is randomly and equally selected as the optimization objective; the AdamW optimizer is used in conjunction with a cosine annealing learning rate strategy, while automatic mixed precision training is employed; and the optimal weights of the validation set are saved as the final model.

[0021] Furthermore, the fault diagnosis results, together with the three dimensions of visualized evidence, constitute a complete diagnostic report and are output. Specifically, the fault category prediction results, Grad-CAM activation heatmap, attention weight matrix heatmap, CKA values, and modal fusion weight distribution are output to the user interface or storage medium, forming a complete and traceable diagnostic basis from the original vibration signal to the final fault classification.

[0022] A second aspect of the present invention provides a system for implementing the above-described method, comprising:

[0023] The dual-modal coding module is configured to: acquire the vibration signal of the rotating machinery, perform time-domain coding using the GASF method and frequency-domain coding using the FFT-based GADF method, respectively, to generate GASF and GADF images with complementary physical features;

[0024] The feature extraction and fusion module is configured to: use two feature extraction networks with the same structure but independent parameters to extract deep feature sequences from GASF and GADF images respectively, and enhance and fuse the extracted deep feature sequences; during the enhancement, the model is progressively evolved from shallow global exploration to deep local focus through monotonically decreasing constraints;

[0025] The interpretability analysis module is configured to output fault diagnosis results based on the fused features, and simultaneously output three dimensions of visual evidence, including:

[0026] In the coding layer, Grad-CAM activation heatmaps are generated for GASF and GADF images respectively. The decision-making basis area is highlighted on the GAF image, and the physical correspondence between the activation area of ​​the heatmap and the time-domain periodicity and frequency-domain fault characteristic frequency of the vibration signal is established.

[0027] At the feature layer, a heatmap of the attention weight matrix during deep feature sequence fusion is output, and the center kernel aligned CKA is used to quantitatively evaluate the feature space similarity between modalities. The output CKA values ​​are used to prove that substantial feature interaction is achieved between modalities.

[0028] At the decision level, the modality fusion weight distribution during deep feature sequence fusion is output;

[0029] The results output module is configured to output a complete diagnostic report consisting of fault diagnosis results and three dimensions of visual evidence.

[0030] A third aspect of the present invention provides a computer-readable storage medium.

[0031] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps in the fault diagnosis method of bimodal GAF coding and cross-modal fusion as described above.

[0032] A fourth aspect of the present invention provides a computer device.

[0033] A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps in the fault diagnosis method of bimodal GAF coding and cross-modal fusion as described above.

[0034] A fifth aspect of the present invention provides a computer program product, including a computer program that, when executed by a processor, implements the steps in the above-described fault diagnosis method of bimodal GAF encoding and cross-modal fusion.

[0035] Compared with existing technologies, one or more of the above technical solutions have the following beneficial effects:

[0036] By employing GASF and GADF dual-modal complementary coding, the grid-like texture in the GASF image corresponds to the time-domain periodic impact features, and the mirrored stripes in the GADF image correspond to the frequency-domain fault feature frequencies, thus giving the image a clear physical meaning from the input end. In the feature fusion stage, a monotonically decreasing constraint is introduced to drive the model to evolve progressively from shallow global exploration to deep local focus, and the attention weight matrix heatmap and CKA values ​​are output simultaneously, so that the modal interaction intensity and similarity changes during the fusion process can be quantified and visualized. The output modal fusion weight distribution clearly shows the contribution ratio of the time-domain mode and the frequency-domain mode to the current diagnostic results. The Grad-CAM heatmap in the coding layer establishes the physical correspondence between the image activation region and the time-domain periodicity and frequency-domain fault feature frequencies of the vibration signal, so that the decision basis can be traced back to the original physical quantity. By integrating interpretability as an integral part of the diagnostic process rather than an afterthought, and through a three-tiered progressive design of "physically perceptible input representation - progressively traceable fusion process - multi-level visualized decision evidence," a complete and traceable diagnostic chain from raw vibration signals to final fault classification is fundamentally constructed. This significantly improves the engineering credibility and feasibility of deep learning models in safety-critical scenarios. Attached Figure Description

[0037] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0038] Figure 1 A schematic diagram of the dual-modal GAF encoding process provided in one or more embodiments of the present invention;

[0039] Figure 2 This is a diagram illustrating the overall fault diagnosis framework architecture provided in one or more embodiments of the present invention.

[0040] Figure 3 A schematic diagram of temperature parameter control provided for one or more embodiments of the present invention;

[0041] Figure 4 A ResNet feature extraction network structure diagram provided for one or more embodiments of the present invention;

[0042] Figure 5 A structural diagram of a cross-modal attention mechanism provided in one or more embodiments of the present invention;

[0043] Figure 6 A comparison chart (including error bars) of the accuracy of various algorithms provided for one or more embodiments of the present invention under different training data sizes on the JNU bearing dataset (Case 1).

[0044] Figure 7 A visual comparison of the t-SNE features of the algorithms provided in one or more embodiments of the present invention on the WT planetary gearbox dataset (Case 2);

[0045] Figure 8 A schematic diagram of a multi-level interpretability analysis system provided for one or more embodiments of the present invention. Detailed Implementation

[0046] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0047] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0048] Terminology Explanation:

[0049] Gramian Angular Field (GAF) is an encoding method that converts one-dimensional time series data into two-dimensional images. Its core idea is to map normalized time-series data to polar coordinate angle space through an inverse cosine transform, and then construct a Gram matrix using the trigonometric relationships between angles, thereby generating an image representation while preserving the original signal's temporal dependencies. GAF mainly includes two forms: Gramian Angular Sum Field (GASF), which uses cosine angle sum operations and excels at capturing the global periodicity and steady-state correlation of signals; and Gramian Angular Difference Field (GADF), which uses sinusoidal angle difference operations and is more sensitive to local changes and transient shocks in signals.

[0050] Grad-CAM: An interpretability technique for visualizing deep neural networks. It generates heatmaps by analyzing the gradient of classification scores with respect to feature maps to locate the image regions that contribute most to the model's decisions.

[0051] CKA (Centered Kernel Alignment): A quantitative metric for measuring the similarity between two feature spaces. This scheme is used to evaluate whether cross-modal attention mechanisms achieve substantial intermodal feature interaction.

[0052] As described in the background section, deep learning fault diagnosis models suffer from a lack of interpretability. Since most existing methods treat fault diagnosis as an end-to-end classification task, the optimization objective focuses solely on improving classification accuracy. Driven by this, models tend to learn highly abstract, nonlinearly transformed deep features. While these features are beneficial for classification, they gradually become decoupled from the physical meaning of the original vibration signal (such as the time-domain impact period and frequency-domain characteristic frequencies). When the model outputs a fault category (e.g., "inner ring fault"), technicians cannot trace back to which segment of the signal or which physical feature the model based its judgment. More critically, existing multimodal fusion methods typically employ feature concatenation or fixed-weight summation, effectively "black-boxing" the contributions of different modes to the final decision. This makes it impossible to quantify the individual roles of the time-domain and frequency-domain modes, or explain why the model assigns different fusion ratios for different fault types.

[0053] To address the aforementioned issues, existing methods that directly use the original vibration signal or spectrum as input can learn effective features, but these features lack intuitive physical mapping relationships. If image encoding (such as time-spectrum mapping) is used, the correspondence between image textures and fault physical features presented by different encoding methods is often unclear. In other words, designing an encoding method that establishes a stable and interpretable correspondence between each type of texture structure in the image and specific fault physical features (such as time-domain periodic impacts and frequency-domain characteristic peaks) is the primary challenge.

[0054] In addition, the fusion process is opaque. The core of multimodal fusion lies in the reasons for fusion, the fusion method, and how the fusion results are used, but existing methods cannot quantify and visualize these aspects.

[0055] Therefore, this scheme systematically constructs a complete, traceable, and auditable diagnostic decision chain from the original vibration signal to the final fault classification by using complementary bimodal coding with clear physical meaning, hierarchical temperature-controlled attention that mimics the cognitive process, and an interpretable analysis system with three-dimensional synchronous output. By assigning physical meaning to the input through bimodal GAF coding, and simulating the cognitive process of "global exploration → local focus" using hierarchical temperature-controlled cross-modal attention, the scheme simultaneously outputs three-dimensional interpretable evidence at the coding layer, feature layer, and decision layer, forming a complete and traceable decision chain from signal to diagnosis.

[0056] A fault diagnosis method based on bimodal GAF coding and cross-modal fusion includes the following steps:

[0057] Vibration signals of rotating machinery are acquired, and time-domain encoding is performed using the GASF method and frequency-domain encoding using the FFT-based GADF method, respectively, to generate two two-dimensional images, namely GASF images and GADF images. Since the textures in the images correspond to the physical features, the physical features in the two images are complementary.

[0058] GASF and GADF images are input into two independent ResNet networks to extract features, and then fused through a three-layer sequential bidirectional attention mechanism; each layer is set with a decreasing "temperature parameter". This step simulates the human cognitive process of "looking at the whole before looking at the details". By outputting the attention weight matrix of each layer, engineers can quantitatively observe how the model gradually identifies fault features.

[0059] The model first "guesses" a fault probability distribution based on current features (e.g., a higher probability of faults in the inner ring), and then uses this probability to dynamically calculate the final fusion weights of GASF and GADF features. For example, when judging "inner ring fault," the model automatically assigns higher weights to GASF (capturing temporal impulses); when judging "gear tooth breakage," it assigns higher weights to GADF (capturing frequency-domain modulation sidebands). Engineers can see "why the model trusts a certain mode more."

[0060] While the model outputs diagnostic results (such as "rolling element failure"), it also outputs three dimensions of visual evidence in parallel:

[0061] Encoding layer: Grad-CAM heatmap, which highlights the decision-making basis area on the GAF image;

[0062] Feature layer: Attention weight matrix heatmap + CKA values ​​(proving that modal interactions did indeed occur);

[0063] Decision-making level: Modality fusion weight distribution (e.g., "In this diagnosis, time domain information contributes 70%, and frequency domain information contributes 30%");

[0064] This step ensures that the conclusion is no longer an isolated label, but a complete and traceable chain of diagnostic evidence that includes "original signal → encoded features → fusion basis → final decision".

[0065] The specific scheme is explained with reference to the attached diagram below, in which:

[0066] Figure 1 (a) shows the time series before and after normalization and in polar coordinates; (b) shows the GASF encoded image; (c) shows the GADF encoded image.

[0067] Figure 2The fault diagnosis framework architecture diagram shown includes a time-frequency dual-modal coding module, a feature extraction and cross-modal fusion module, an adaptive fusion and decision output module, and a multi-level interpretability analysis system.

[0068] Figure 3 The temperature parameter control diagram shows: (a) a self-attention structure diagram with temperature scaling; (b) a comparison of the attention distribution effect under different temperature parameter control. ).

[0069] Figure 4 The ResNet feature extraction network structure diagram shows the connection relationship between the initial convolutional layer, four residual layer groups, and the adaptive average pooling layer;

[0070] Figure 5 The provided cross-modal attention mechanism structure diagram shows the organization of three layers of bidirectional cross-modal attention, FFN, and layer normalization.

[0071] A fault diagnosis method based on bimodal GAF coding and cross-modal fusion includes the following steps:

[0072] Step 1: Dual-modal GAF encoding. After preprocessing the original vibration signal, the time-domain vibration signal is encoded using the GASF method. Simultaneously, the frequency domain amplitude spectrum is extracted using FFT and then encoded using the GADF method to generate a complementary dual-modal image representation.

[0073] Step 2: ResNet Feature Extraction. The GASF and GADF images are input into two ResNet feature extraction networks with identical structures but independent parameters to extract deep feature sequences for each modality.

[0074] Step 3: Graded Temperature Control Cross-Modal Attention Fusion. A three-layer serial bidirectional cross-modal attention mechanism is used to achieve layer-by-layer deep interaction of bimodal features. The temperature parameters of each layer are pre-set fixed hyperparameters, corresponding one-to-one with the three attention modules and must satisfy a monotonically decreasing constraint (as shown in the figure). (As a preferred embodiment), the driving model evolves progressively from shallow global exploration to deep local focus;

[0075] Step 4: Category-Aware Adaptive Fusion. Based on the predicted fault category probability distribution, the fusion weights of each modality are dynamically calculated using a learnable category weight matrix, and the enhanced bimodal features are adaptively weighted and fused.

[0076] Step 5: Classification Output and Interpretability Analysis. The fused features are processed by a fully connected classifier to output the fault category prediction results. Simultaneously, a multi-level interpretability analysis module runs concurrently as part of the diagnostic process. At the encoding layer, it outputs Grad-CAM activation heatmaps for each mode and establishes a correspondence with the physical features of the signal. At the feature layer, it outputs cross-modal attention weight matrices and CKA quantitative evaluation results to track the feature interaction process. At the decision layer, it outputs key feature localization maps and modal fusion weight distributions to support the credibility assessment of the diagnostic results. This provides engineers with a complete and traceable diagnostic basis from the original vibration signal to the final fault classification.

[0077] As a further implementation, GASF coding transforms the time-domain vibration signal into a two-dimensional image reflecting the temporal correlation through cosine angle sum operations: after normalizing the time-domain signal, the time-domain angle parameters are obtained through inverse cosine transform. Construct matrix elements ,in These are the row and column indices of the time-domain angular parameter sequence, respectively, and the matrix's first... Each element encodes the angle and cosine value at time i and time j, which are used to capture the periodic evolution of the signal in the time domain and the global steady-state correlation.

[0078] As a further implementation, GADF encoding transforms the frequency domain amplitude spectrum into a two-dimensional image reflecting frequency differences through sinusoidal angle difference operations: the positive frequency component amplitude spectrum is extracted by performing an FFT transform on the time-domain signal, and then normalized to obtain the frequency domain angle parameters. Construct matrix elements ,in These are the row and column indices of the frequency domain angular parameter sequence, respectively, and the matrix's first... Each element encodes the sine value of the angle difference between the i-th and j-th frequency components. This is used to enhance local variations and modulation sidebands at fault characteristic frequencies.

[0079] As a further implementation, the ResNet feature extraction network includes an initial convolutional layer (7×7 convolutional kernel, stride 2), four residual layer groups (number of channels...). Each group contains two residual blocks and an adaptive average pooling layer; the residual blocks effectively alleviate the gradient vanishing problem in deep networks through skip connections; after adaptive average pooling, a feature sequence flattened to 49×256 is obtained.

[0080] As a further implementation, the formula for calculating the cross-modal attention weight of the l-th layer in the hierarchical temperature control cross-modal attention mechanism is as follows:

[0081] ;

[0082] Among them temperature parameters To keep the hyperparameters fixed, they are pre-set before training and do not update during training. The three temperature values ​​correspond strictly one-to-one with the three-layer cross-modal attention modules. Corresponding to the first layer, Corresponding to the second layer, (Corresponding to the 3rd layer).

[0083] High temperature value ( This makes the attention distribution more uniform, which facilitates the coverage of a wide range of cross-modal global associations to support global exploration;

[0084] Low temperature value ( This sharpens the distribution of attention, which is beneficial for accurately locating the most discriminative cross-modal local features;

[0085] when It degenerates into standard scaled dot product attention.

[0086] The temperature parameters of the three layers must satisfy a monotonically decreasing constraint. ,by For the preferred embodiment, a progressive feature refinement from global exploration to local focus is achieved; the actual optimal configuration should be determined on the target dataset through ablation experiments.

[0087] As a further implementation method, the essential difference between the category-aware adaptive fusion module and the conventional fixed-weight fusion and feature stitching method is that the conventional method uses static fusion weights that are independent of the category for all fault types, implicitly assuming that the time domain and frequency domain modes contribute equally to any fault type, ignoring the inherent differences in the time-frequency information dependence of different faults.

[0088] This module introduces a learnable class weight matrix. The model uses the fault category prediction probability distribution of the current sample as a dynamic bridge, allowing the fusion weights to adaptively adjust according to the fault type of the input sample. This ensures that both time-domain dominant faults (more dependent on GASF mode) and frequency-domain dominant faults (more dependent on GADF mode) can achieve a fusion ratio matching their feature dependencies. Specifically, this is achieved by extracting dual-modality global features through global average pooling, and then obtaining the probability distribution through a category predictor. Using a learnable class weight matrix Calculate fusion weights The fusion coefficients were obtained by softmax normalization. Ultimately, the weighted summation is used. To achieve adaptive fusion, among which and These are bimodal features enhanced with cross-modal attention; the category weight matrix is ​​automatically learned through backpropagation and does not require manual specification.

[0089] As a further implementation, training strategies include: introducing Mixup data augmentation with a 50% probability for linear interpolation of samples; randomly selecting standard cross-entropy or Mixup loss; using the AdamW optimizer with a cosine annealing learning rate; and automatic mixed precision (AMP) training.

[0090] In this scheme, the dual-modal GAF encoding part adopts a differentiated encoding strategy, transforming the one-dimensional vibration signal into two complementary two-dimensional image representations. GASF encoding directly applies to the time-domain vibration signal: firstly, the signal... Normalization to The time-domain angular parameters are obtained by using an inverse cosine transform within the interval. Finally, construct the matrix .

[0091] This encoding uses angle and cosine operations to encode the amplitude correlation between adjacent time points in the time domain, presenting a characteristic "grid-like texture structure" in the image. The corner lines and their neighborhoods form a continuous band-like structure, and the grid size and density of different fault states (inner ring, outer ring, ball fault) are significantly different, which has clear physical interpretability.

[0092] Unlike GASF, GADF encoding operates on the frequency domain amplitude spectrum: it first performs a Fast Fourier Transform (FFT) on the original vibration signal to extract the amplitude spectrum of the positive frequency components, and then normalizes the amplitude spectrum to obtain the frequency domain angular parameters. Constructing a matrix .

[0093] This matrix is ​​antisymmetric, with mirrored color distributions on both sides of the image. Significant peaks at fault feature frequencies in the frequency domain are transformed into distinct vertical or horizontal stripes, accurately marking the fault feature frequency locations and significantly enhancing the visual saliency of the fault features. GASF and GADF characterize vibration signals from two complementary dimensions: time-domain periodicity and frequency-domain local variation, respectively, providing a rich foundation of fault information for subsequent cross-modal feature fusion (e.g., Figure 1 (As shown).

[0094] The feature extraction part consists of two independent ResNet feature extraction networks, which are used to extract deep feature representations from GASF and GADF images respectively (e.g., Figure 4 (As shown).

[0095] Each ResNet network consists of an initial convolutional layer (7×7 kernel, stride 2), four residual layer groups (with channels of 32, 64, 128, and 256 respectively, each group containing two residual blocks), and an adaptive average pooling layer. Each residual block consists of two 3×3 convolutional layers, with skip connections directly adding the input to the convolutional output, effectively mitigating the gradient vanishing problem in deep networks. After progressive feature abstraction through four residual layers, the feature map space is gradually compressed to 7×7, and then flattened into a 49×256 feature sequence after adaptive average pooling, serving as the input to the subsequent cross-modal fusion module. The parameters of the two ResNet networks are completely independent, ensuring the independence and complementarity of features from each modality.

[0096] To fully leverage the complementarity of bimodal features, this scheme constructs a three-layer bidirectional cross-modal attention fusion module (e.g., Figure 5 (As shown). The three-layer modules are connected in series, and the output feature of the l-th layer is used as the... The input from each layer is used to complete the deep interaction and refinement of cross-modal features layer by layer.

[0097] For the l-th layer GASF features output from the previous layer and GADF features As input, the query matrix Q, key matrix K, and value matrix V are generated through their respective independent linear transformation matrices.

[0098] In cross-modal attention along the GASF→GADF direction, the query matrix The key matrix is ​​obtained by the GASF characteristic linear transformation. AND-value matrix All are obtained by the GADF characteristic linear transformation, specifically:

[0099] .

[0100] By temperature parameter Calculate cross-modal attention weights using scaled dot product attention: Then, update the GASF features through residual connections: The GADF→GASF direction proceeds symmetrically, that is... Features derived from GADF and Update GADF features from GASF features: This bidirectional attention mechanism enables each modality to selectively extract complementary information from the other modality, achieving deep feature interaction across modalities.

[0101] Temperature parameters The hierarchical setting strategy is one of the core innovations of this solution, and its design concept originates from the simulation of the human diagnostic cognitive process "from the whole to the part" (such as...). Figure 3 (As shown). In the three-layer cross-modal attention module, the temperature parameter To keep the hyperparameters constant, they are manually set before training and are not updated during training. The three temperature values ​​correspond one-to-one with the three cross-modal attention modules: Acts on layer 1. Acts on the second layer. It operates on the third layer of the cross-modal attention module and is independent of the residual layer.

[0102] The temperature parameter should be set according to the following rule: as the network depth increases, the temperature parameter must decrease monotonically, that is... .

[0103] Specifically, the first layer uses a high temperature value greater than 1 (in the form of...). (In the preferred embodiment) the attention distribution tends to be uniform, covering a wide range of cross-modal associations to support global exploration;

[0104] The second layer uses a transition temperature (with) In this preferred embodiment, it degenerates into standard scaled dot product attention, achieving a balance between global context and local details;

[0105] The third layer uses a low temperature value less than 1 (in the form of...). (As a preferred embodiment), the attention distribution tends to be sharp, and the most discriminative cross-modal local features are accurately located to achieve local focusing.

[0106] when At this point, it degenerates into standard scaled dot product attention. The temperature difference between adjacent layers should not be too small (otherwise the difference between layers is not significant, and the grading effect degenerates into a uniform temperature setting), nor should it be too large (otherwise the shallow attention is overly diffused, introducing irrelevant noise that interferes with subsequent feature refinement). The actual optimal configuration should be determined on the target dataset through ablation experiments.

[0107] Compared to simply stacking cross-modal attention, this hierarchical temperature scheduling strategy enables feature layers of different depths to adaptively adjust the intensity and manner of cross-modal information interaction, driving the model to complete progressive feature refinement from global exploration to local focus, thereby fully leveraging the complementary advantages of GASF and GADF. Each layer of cross-modal attention is followed by a feedforward network (FFN, with hidden layer dimensions four times the embedding dimension) and layer normalization to further enhance the features.

[0108] After three layers of cross-modal attention fusion, the dual-modal features are integrated into a unified fault diagnosis representation through a category-aware adaptive fusion module.

[0109] This module first processes the enhanced bimodal features. and Global average pooling is performed separately to obtain the global feature vector. and , concatenate vectors The input category predictor yields the fault category probability distribution. Furthermore, the learnable class weight matrix is ​​utilized. The fusion weight is obtained by multiplying it by the category probability distribution. The fusion coefficient is obtained after softmax normalization. ;

[0110] Finally, by weighted summation Adaptive fusion is achieved. This mechanism automatically adapts to the different dependencies of different fault types on time-domain and frequency-domain information: fault categories characterized by specific frequency impulses will have their weights more biased towards the GADF side, while fault categories characterized by time-domain amplitude modulation will have their weights more biased towards the GASF side. This makes the fusion weights of each fault category exhibit significant intra-class clustering, greatly enhancing the interpretability of model decisions.

[0111] The category weight matrix is ​​learned automatically through backpropagation, eliminating the need for manual specification. Feature fusion. After global average pooling, the result is input into a fully connected classifier to obtain the final fault category prediction.

[0112] In terms of training strategy, this scheme employs a joint optimization approach to enhance the model's generalization ability and robustness. Mixup data augmentation is introduced into the current batch with a 50% probability, and virtual training samples are generated by linear interpolation of samples within the batch. Standard cross-entropy loss (with added label smoothing) or Mixup loss is randomly and equally selected as the optimization objective. The AdamW optimizer (initial learning rate 1×10⁻⁻⁻⁶) is used. 4 The learning rate is smoothly decayed by using a cosine annealing learning rate strategy with a weight decay of 0.01. At the same time, automatic mixed precision (AMP) training is used to accelerate convergence and reduce memory consumption, and the optimal weights of the validation set are saved as the final model.

[0113] This solution integrates a multi-level interpretability analysis system as an integral part of the fault diagnosis method, running synchronously with the diagnosis process during the inference phase. It outputs traceable diagnostic evidence from three dimensions: the coding layer, the feature layer, and the decision layer (e.g., ...). Figure 8 As shown in the figure, this enables technicians to understand and review the model's diagnostic decisions for each input sample.

[0114] At the coding layer, the interpretability analysis module automatically outputs Grad-CAM activation heatmaps of GASF and GADF images for each input sample. By locating the image region that contributes the most to the current diagnostic result in each coding mode, it establishes a physical correspondence between the coding features and the temporal periodicity of the vibration signal (GASF grid-like texture region) and the frequency of the fault feature (GADF antisymmetric stripe region), so that the feature extraction process of the coding layer has a traceable physical basis.

[0115] At the feature layer, the interpretability analysis module outputs a three-layer cross-modal attention weight matrix in real time, intuitively presenting the interaction intensity distribution between GASF and GADF features in each layer. Technicians can use this to confirm whether the gradual evolution of attention from global exploration to local focus as the temperature parameter decreases is in line with expectations. At the same time, CKA is used to quantitatively evaluate the feature space similarity between each layer. When the cross-modal feature similarity increases significantly from a low level at the input (e.g., 0.2417) as the fusion layer deepens (e.g., to 0.9444), it can be quantitatively confirmed that the cross-modal attention mechanism has achieved substantial intermodal feature interaction, rather than simple superposition.

[0116] At the decision-making level, the interpretability analysis module outputs a true Grad-CAM activation map for each diagnostic sample, accurately locating the key feature regions driving the final classification decision. It also outputs the modal fusion weights and prediction confidence for that sample. Technicians can observe whether the fusion weights are consistent with the historical weight distribution (intra-class clustering interval) of the fault category to help determine the reliability of the diagnostic results. These three layers of output together constitute a complete interpretable diagnostic decision chain from the original vibration signal to the final fault classification, ensuring that every step of the model's reasoning logic is verifiable.

[0117] To verify the effectiveness of the proposed method, comprehensive experiments were conducted on two typical datasets: the JNU bearing dataset (Case 1) and the WT planetary gearbox dataset (Case 2). The experimental data are as follows: Figures 6-8 As shown.

[0118] Figure 6 A comparison chart (including error bars) showing the accuracy of each algorithm on different training data sizes of the JNU bearing dataset (Case 1).

[0119] Figure 7 A visualization comparison of t-SNE features of each algorithm on the WT planetary gearbox dataset (Case 2) is provided, showing the distribution characteristics of the five types of fault features in two-dimensional space.

[0120] Figure 8It provides a schematic diagram of a multi-level interpretable analysis system, which systematically demonstrates the diagnostic decision-making mechanism from three dimensions: the coding layer (GASF / GADF original images and simulated Grad-CAM), the feature layer (residual features, cross-modal attention weights and fusion feature evolution), and the decision layer (real Grad-CAM and confidence-fusion weight distribution).

[0121] The JNU bearing dataset was collected from a bearing test bench at a university. The sampling frequency was 50kHz. It included four health states: normal (N), inner ring fault (I), outer ring fault (O), and ball fault (B). A 2048-point sliding window was used to extract samples, and each experiment was repeated 5 times and the average value was taken.

[0122] Comparative experiments on encoding methods show that the dual-modal GASF+GADF fusion (99.00%±0.31%) is significantly better than GASF alone (77.20%±1.04%), GADF alone (97.05%±0.72%), and the STFT+CWT joint scheme (98.25%±0.47%), verifying the superiority of the dual-modal complementary encoding strategy.

[0123] Ablation experiments showed that the baseline stitching fusion without attention mechanisms (A1) achieved 99.00%; the result with the introduction of uniform temperature cross-modal attention (A2) slightly decreased to 98.80%, indicating that the lack of temperature differentiation introduces optimization instability; the addition of hierarchical temperature scheduling... (A3) The accuracy was restored and improved to 99.10%, proving the necessity of progressive temperature annealing for effective cross-modal feature interaction; further, category-aware adaptive fusion (A4) was introduced, achieving the highest accuracy of 99.50% ± 0.25%, verifying the effectiveness of dynamic modal weight adjustment.

[0124] Temperature parameter sensitivity analysis systematically validated the optimal configuration within the experimental space evaluated on the JNU bearing dataset across 13 temperature configurations (optimal values ​​may differ across datasets; ablation experiments are recommended to determine the optimal configuration). Reverse scheduling was also demonstrated. The accuracy rate was only 99.10%, validating the design principle that temperature must monotonically decrease with network depth. Regarding small-sample generalization performance, under a 60% test scale (minimum training samples), this method still maintained an accuracy of 97.55% ± 0.29%, with a standard deviation consistently below 0.6%, far lower than the fluctuation range of CNNs (2.36%~4.65%).

[0125] The WT planetary gearbox dataset was collected from a wind power transmission system test bench, with a sampling frequency of 48kHz. It includes five health states: broken tooth, healthy, missing tooth, root crack, and wear, mainly focusing on scenarios with few samples.

[0126] This method achieved the highest diagnostic accuracy across all test data scales (10%–60%): at 60% test scale, it reached 92.87% ± 1.45%, surpassing ResNet and WDCNN (both 90.93%) by approximately 2 percentage points, and surpassing traditional CNN (45.47% ± 2.56%), MLFNet (46.33% ± 3.39%), and MSACNN (42.07% ± 1.44%) by over 46 percentage points; at 10% test scale, it reached 98.40% ± 0.89%, surpassing all comparative methods.

[0127] The t-SNE feature visualization qualitatively verified the feature learning ability of the method: the five types of fault features of this method present a clearly separated cluster structure in two-dimensional space, with large inter-class distance and compact intra-class structure; the confusion matrix analysis shows that the diagonal elements are between 0.96 and 1.00 when the split is 90 / 10, and are still higher than 0.92 even when the split is 40 / 60, which fully verifies the strong generalization and robustness of the method in the case of few samples.

[0128] By constructing a time-frequency dual-modal complementary coding strategy, the limitation of incomplete information representation in existing single-modal GAF methods is overcome. GASF captures the time-domain periodicity of the signal through angle and cosine operations, while GADF enhances the local changes at fault characteristic frequencies by performing angle difference sine operations on the frequency-domain amplitude spectrum. The two form a naturally complementary feature representation. Ablation experiments show that the diagnostic accuracy of the dual-modal GASF+GADF fusion scheme is significantly better than that of GASF alone, GADF alone, and the STFT+CWT combined scheme. Detailed experimental data can be found in the implementation details.

[0129] The hierarchical temperature-controlled cross-modal attention mechanism overcomes the limitations of conventional single-temperature cross-modal attention. Conventional approaches use uniform temperature parameters for all fusion layers, degenerating into standard scaled dot product attention with indiscriminate attention distribution across layers. Shallow layers lose complementary information due to their inability to broadly explore global cross-modal associations, while deeper layers lack sufficient attention concentration, weakening their ability to extract discriminative local features. The core design of this scheme mimics the human cognitive pattern of "from shallow to deep," assigning monotonically decreasing temperature parameters to each cross-modal attention layer: high temperatures ensure uniform attention distribution, covering a wide range of cross-modal association regions for global exploration; low temperatures sharpen attention distribution, precisely locating the most discriminative cross-modal feature regions for local focus. Compared to uniform temperature settings, this progressive temperature scheduling strategy fully leverages the hierarchical representation advantages of multi-layer networks; compared to reverse temperature scheduling, the monotonically decreasing temperature design principle aligns with the feature evolution of deep neural networks from low-level texture to high-level semantics, thereby driving GASF and GADF modal features to achieve layer-by-layer deep interaction and refinement from coarse to fine granular.

[0130] The category-aware adaptive fusion mechanism addresses the shortcomings of conventional fixed-weight fusion, which treats different fault types "equally." Conventional multimodal fusion (such as feature concatenation or fixed-weight summation) implicitly assumes that time-domain and frequency-domain modes contribute equally to any fault type, ignoring the inherent differences in the time-frequency information dependence of different faults: faults with significant time-domain periodic impact characteristics rely more on the time-domain correlation carried by the GASF mode, while faults with specific frequency components as distinguishing features rely more on the frequency-domain local changes enhanced by the GADF mode. The essential difference of this scheme lies in introducing a learnable category weight matrix, using the model's fault category prediction probability distribution for the current sample as a bridge, dynamically mapping it to the fusion weights of each mode, thus deeply coupling the fusion decision with the fault category characteristics, rather than using static weights unrelated to the fault category. This mechanism enables the model to adaptively adjust the fusion ratio of time-frequency modes for different fault types. While improving diagnostic accuracy, it also makes the fusion weights of samples of each fault category show obvious intra-class clustering, making the decision logic interpretable and transparent. It can simultaneously address the two core problems of insufficient multimodal complementary representation and "black box".

[0131] This scheme incorporates a multi-level interpretability analysis system as an integral part of the fault diagnosis method. During the inference stage, it simultaneously outputs traceable diagnostic evidence in three dimensions: the coding layer, the feature layer, and the decision layer, forming a complete diagnostic decision chain from the original vibration signal to the final fault classification. Furthermore, it uses CKA to quantitatively evaluate the feature interaction effect between each layer, which can confirm in real time that the cross-modal attention mechanism has achieved substantial intermodal feature interaction. For detailed analysis results, please refer to the implementation method.

[0132] It exhibits excellent generalization ability and robustness in small sample scenarios: Under different training sample sizes in the JNU bearing dataset and the WT planetary gearbox dataset, this method maintains the highest diagnostic accuracy, and the standard deviation is consistently significantly lower than all the comparison methods, indicating that the method has high stability to changes in the amount of training data; for specific experimental data, please refer to the implementation method.

[0133] Correspondingly, the fault diagnosis system combining bimodal GAF coding and cross-modal fusion includes:

[0134] The dual-modal coding unit is configured to perform GASF time-domain coding and FFT-based GADF frequency-domain coding on the vibration signal to generate complementary dual-modal image representations.

[0135] The feature extraction unit is configured to use two independent ResNet feature extraction networks to extract deep feature sequences from GASF and GADF images, respectively.

[0136] The cross-modal fusion unit is configured to achieve progressive deep interaction and enhancement of dual-modal features through a three-layer serial hierarchical temperature-controlled cross-modal attention mechanism;

[0137] The three-layer temperature parameters To ensure that the hyperparameters are fixed and correspond one-to-one with the three-layer cross-modal attention modules, they must satisfy the monotonically decreasing constraint. ,in This allows for a more uniform distribution of shallow attention, enabling global exploration. To sharpen deep attention to achieve localized focus, This is a preferred embodiment;

[0138] The adaptive fusion unit is configured to dynamically calculate the dual-modal fusion weights based on the predicted fault category probability and perform adaptive weighted feature fusion.

[0139] The classification output unit is configured to output the final fault category diagnosis result by passing the fused features through a fully connected classifier.

[0140] The interpretability analysis unit is configured to run synchronously with the diagnostic process during the inference phase, output Grad-CAM activation heatmaps of each modality and their correspondence with the physical features of the signal from the coding layer, output cross-modal attention weight matrices and CKA quantitative evaluation results from the feature layer, and output key feature localization maps and modal fusion weights from the decision layer, providing technicians with a traceable and complete diagnostic basis.

[0141] By employing GASF and GADF dual-modal complementary encoding, the grid-like texture in the GASF image corresponds to the time-domain periodic impact features, and the mirrored stripes in the GADF image correspond to the frequency-domain fault feature frequencies. This endows the image with a clear physical meaning from the input end, solving the problem of "what the model is looking at" being unknown. By introducing a monotonically decreasing constraint in the feature fusion stage, the model is driven to evolve progressively from shallow global exploration to deep local focus, and the attention weight matrix heatmap and CKA values ​​are output simultaneously, making the changes in modal interaction intensity and similarity during the fusion process quantifiable and visualized, solving the problem of "how the model fuses" being invisible. Furthermore, the modal fusion weight distribution is output, clearly showing the contribution ratio of the time-domain mode and the frequency-domain mode to the current diagnostic results, solving the unsolvable problem of "why the model trusts a certain mode more". Finally, the Grad-CAM heatmap in the encoding layer establishes a physical correspondence between the image activation region and the time-domain periodicity and frequency-domain fault feature frequencies of the vibration signal, making the decision basis traceable to the original physical quantities. The reason for the above design is that existing methods only focus on classification accuracy while ignoring the transparency of the reasoning process. This solution regards interpretability as an organic part of the diagnostic process rather than an afterthought module. Through a three-layer progressive design of "physically perceptible input representation - progressively traceable fusion process - multi-level visualized decision evidence", it fundamentally constructs a complete traceable diagnostic chain from the original vibration signal to the final fault classification, which significantly improves the engineering credibility and implementation feasibility of deep learning models in safety-critical scenarios.

[0142] Correspondingly, a computer program product includes computer-readable instructions that, when executed on an electronic device, enable the electronic device to implement the aforementioned fault diagnosis method of bimodal GAF encoding and cross-modal fusion.

[0143] Accordingly, an electronic device includes at least one processor and a memory connected to the processor, the memory being used to store a computer program; the processor is used to execute the computer program, enabling the electronic device to implement the aforementioned fault diagnosis method of bimodal GAF coding and cross-modal fusion.

[0144] Those skilled in the art will understand that embodiments of this solution can be embodied as a method, system, or computer program product. Therefore, this solution can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this solution can also take the form of a computer program product implemented on one or more computer-readable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0145] This solution is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this solution. It should be understood that each block in the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to create a machine such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, can implement one or more blocks in the flowchart and / or one or more blocks in the block diagram, specifying the function.

[0146] These computer program instructions may also be stored in a computer-readable storage medium capable of directing a computer or other programmable data processing device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium generate an article of manufacture containing instruction means. These instruction means are used to implement the functions specified in one or more flowcharts and / or one or more blocks of a block diagram.

[0147] Furthermore, these computer program instructions can be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to generate a computer-implemented processing procedure. Thus, the instructions that execute on the computer or other programmable equipment will provide steps for implementing one or more processes of the flowchart and / or one or more blocks of the block diagram that specify the function.

[0148] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A fault diagnosis method based on dual-modal GAF coding and cross-modal fusion, characterized in that, Includes the following steps: Vibration signals of rotating machinery are acquired, and time-domain encoding is performed using the GASF method and frequency-domain encoding using the FFT-based GADF method, respectively, to generate GASF and GADF images with complementary physical features. Two feature extraction networks with identical structures but independent parameters are used to extract deep feature sequences from GASF and GADF images, respectively. The extracted deep feature sequences are then enhanced and fused. During the enhancement process, a monotonically decreasing constraint is used to make the model evolve progressively from shallow global exploration to deep local focus. Based on the fused features, the fault diagnosis results are output, along with three dimensions of visual evidence, including: In the coding layer, Grad-CAM activation heatmaps are generated for GASF and GADF images respectively. The decision-making basis area is highlighted on the GAF image, and the physical correspondence between the activation area of ​​the heatmap and the time-domain periodicity and frequency-domain fault characteristic frequency of the vibration signal is established. At the feature layer, a heatmap of the attention weight matrix during deep feature sequence fusion is output, and the center kernel aligned CKA is used to quantitatively evaluate the feature space similarity between modalities. The output CKA values ​​are used to prove that substantial feature interaction is achieved between modalities. At the decision level, the modality fusion weight distribution during deep feature sequence fusion is output; The fault diagnosis results, together with the visual evidence from the three dimensions, constitute a complete diagnostic report and are output.

2. The fault diagnosis method based on dual-modal GAF encoding and cross-modal fusion as described in claim 1, characterized in that, The GASF method includes: normalizing the time-domain vibration signal, obtaining the time-domain angle parameters through inverse cosine transform, constructing matrix elements, and using them to capture the time-domain periodic evolution law and global steady-state correlation of the signal. The GADF method includes: performing a fast Fourier transform on the original vibration signal to extract the amplitude spectrum of the positive frequency component, normalizing it to obtain the frequency domain angle parameters, constructing matrix elements, and using them to enhance the local changes and modulation sidebands at the fault characteristic frequencies.

3. The fault diagnosis method based on dual-modal GAF encoding and cross-modal fusion as described in claim 1, characterized in that, The feature extraction network adopts the ResNet structure, which includes an initial convolutional layer, four residual layer groups, and an adaptive average pooling layer. The initial convolutional layer uses a 7×7 convolutional kernel with a stride of 2. The number of channels in the four residual layer groups are 32, 64, 128, and 256, respectively, and each group contains 2 residual blocks. After adaptive average pooling, the output is a deep feature sequence flattened to 49×256.

4. The fault diagnosis method based on dual-modal GAF encoding and cross-modal fusion as described in claim 1, characterized in that, The model evolves progressively from shallow global exploration to deep local focus by using monotonically decreasing constraints. This is achieved through a three-layer serial bidirectional cross-modal attention mechanism, specifically by setting three temperature parameters that correspond to the three-layer cross-modal attention module and satisfy the monotonically decreasing constraints. , ;in, To make the first layer of attention distribution more uniform in order to cover a wide range of cross-modal global correlations, This sharpens the third layer of attention distribution to accurately locate cross-modal local features. A balance is struck between the two; the temperature parameters of each layer are fixed hyperparameters preset before training and are not updated during the training process.

5. The fault diagnosis method based on dual-modal GAF encoding and cross-modal fusion as described in claim 1, characterized in that, The extracted deep feature sequences are enhanced and fused, including a category-aware adaptive fusion step, which involves performing global average pooling on the enhanced dual-modal features to obtain global feature vectors, and inputting the concatenated vectors into a category predictor to obtain the fault category probability distribution. ; Utilizing a learnable class weight matrix The fusion weights are obtained by multiplying the weights by the probability distribution. The fusion coefficients of the GASF and GADF modes were obtained by softmax normalization. Ultimately, adaptive fusion is achieved through weighted summation; the category weight matrix is ​​automatically learned through backpropagation, and the modality fusion weight distribution is the fusion coefficient. .

6. The fault diagnosis method based on dual-modal GAF encoding and cross-modal fusion as described in claim 1, characterized in that, The following strategies are adopted during the training phase: Mixup data augmentation is introduced into the current batch with a 50% probability, and virtual training samples are generated by linear interpolation of samples within the batch; standard cross-entropy loss or Mixup loss is randomly selected with equal probability as the optimization objective; AdamW optimizer is used in conjunction with cosine annealing learning rate strategy, and automatic mixed precision training is used at the same time; the optimal weights of the validation set are saved as the final model.

7. The fault diagnosis method based on dual-modal GAF encoding and cross-modal fusion as described in claim 1, characterized in that, The fault diagnosis results, together with the three dimensions of visual evidence, constitute a complete diagnostic report and are output. Specifically, the fault category prediction results, Grad-CAM activation heatmap, attention weight matrix heatmap, CKA value, and modal fusion weight distribution are output to the user interface or storage medium, forming a complete and traceable diagnostic basis from the original vibration signal to the final fault classification.

8. A fault diagnosis system based on dual-modal GAF encoding and cross-modal fusion, used to implement the fault diagnosis method as described in any one of claims 1-7, characterized in that, include: The dual-modal coding module is configured to: acquire the vibration signal of the rotating machinery, perform time-domain coding using the GASF method and frequency-domain coding using the FFT-based GADF method, respectively, to generate GASF and GADF images with complementary physical features; The feature extraction and fusion module is configured to: use two feature extraction networks with the same structure but independent parameters to extract deep feature sequences from GASF and GADF images respectively, and enhance and fuse the extracted deep feature sequences; during the enhancement, the model is progressively evolved from shallow global exploration to deep local focus through monotonically decreasing constraints; The interpretability analysis module is configured to output fault diagnosis results based on the fused features, and simultaneously output three dimensions of visual evidence, including: In the coding layer, Grad-CAM activation heatmaps are generated for GASF and GADF images respectively. The decision-making basis area is highlighted on the GAF image, and the physical correspondence between the activation area of ​​the heatmap and the time-domain periodicity and frequency-domain fault characteristic frequency of the vibration signal is established. At the feature layer, a heatmap of the attention weight matrix during deep feature sequence fusion is output, and the center kernel aligned CKA is used to quantitatively evaluate the feature space similarity between modalities. The output CKA values ​​are used to prove that substantial feature interaction is achieved between modalities. At the decision level, the modality fusion weight distribution during deep feature sequence fusion is output; The results output module is configured to output a complete diagnostic report consisting of fault diagnosis results and visual evidence from three dimensions.

9. A computer-readable storage medium, characterized in that, It stores a computer program that, when executed by a processor, implements the steps in the fault diagnosis method of bimodal GAF coding and cross-modal fusion as described in any one of claims 1-7.

10. A computer device, characterized in that, It includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps in the fault diagnosis method of bimodal GAF coding and cross-modal fusion as described in any one of claims 1-7.

Images