Multimodal feature fusion domain adaptive diagnosis method and system, and storage medium

Abstract

The invention is applied to the technical field of mechanical equipment state monitoring and intelligent fault diagnosis, and discloses a multi-modal feature fusion domain self-adaptive diagnosis method and system and a storage medium, through a feature correction module (FCM), a cross-modal guide mechanism is utilized to carry out bidirectional self-adaptive calibration on depth features of vibration and current signals, and the depth features of the vibration and current signals are corrected. Feature distribution is aligned, and modal specific noise is suppressed; then, the feature fusion module uses a cross attention mechanism to realize deep interaction and selective information enhancement based on the corrected features; finally, the generated high-quality fusion features are sent into a conditional domain adversarial network, domain invariant features are learned through class-aware adversarial training, and accurate diagnosis is achieved. According to the method, heterogeneity and noise interference among modals are effectively overcome, deep interaction and complementary information enhancement of multi-modal features are realized, and the fault diagnosis accuracy and robustness of the model on a cross-working-condition migration task with relatively large inter-domain difference are remarkably improved.

Description

Technical Field

[0001] This invention relates to the field of mechanical equipment condition monitoring and intelligent fault diagnosis technology, specifically to a domain adaptive diagnosis method, system, and storage medium based on multimodal feature fusion. Background Technology

[0002] Rotating machinery is a core piece of equipment in modern industrial systems. Its stability and reliability directly affect the safety and efficiency of the production system. In rotating machinery, key components such as rolling bearings often work in harsh environments with high speed, heavy load and variable conditions, making them one of the weakest links most prone to failure. Therefore, developing precise and efficient intelligent fault diagnosis technology has significant practical engineering implications. In recent years, data-driven methods, represented by deep learning, have achieved remarkable results in the field of fault diagnosis. Their powerful nonlinear modeling capabilities can automatically extract features from data. However, in actual industrial scenarios, the working conditions of rotating machinery (such as speed and load) often change, resulting in significant differences in the distribution of the collected monitoring signal data, i.e., the "domain shift" phenomenon. At this time, the diagnostic performance of the model trained on the source working condition (source domain) data will drop sharply on the target working condition (target domain), and the generalization ability will be greatly limited. To address the "domain offset" problem, domain adaptation technology has been introduced into cross-condition fault diagnosis. Its core idea is to learn domain-invariant features by reducing the data distribution difference between the source domain and the target domain, thereby transferring knowledge from the source domain to the target domain. For example, some methods use maximum mean square difference (MMSD) to measure distribution difference, while others use adversarial training to achieve feature invariant learning. Existing domain adaptation research mostly focuses on single-mode sensing signals (such as single vibration or current signals), ignoring the redundancy and complementarity between multi-source sensing information. In fact, signals from different physical sources can characterize the health status of equipment from different dimensions: for example, vibration signals are extremely sensitive to transient impact responses caused by defects; while motor current signals can indirectly reflect load fluctuations caused by mechanical faults. The combination of the two can form cross-validation and complementarity of information, providing a more comprehensive and robust fault description than a single signal. Summary of the Invention

[0003] The purpose of this invention is to provide a domain adaptive diagnostic method, apparatus, and storage medium for multimodal feature fusion, in order to solve the problems mentioned in the background art, such as shallow fusion strategy, insufficient modal information interaction, and failure to effectively deal with modal differences and noise interference when processing rotating machinery under varying operating conditions and diagnosing multimodal faults, which result in limited diagnostic performance and generalization ability.

[0004] To achieve the above objectives, the present invention provides the following technical solution: a domain adaptive diagnostic method based on multimodal feature fusion, comprising the following steps: S1. Acquire vibration and current signals of rotating machinery simultaneously under source conditions (source domain) and target conditions (target domain); preprocess the source and target domain signals and construct them into multi-channel samples, and divide them into their respective training and validation sets; S2. In a training loop, perform the following operations on the data sampled from the source domain training set Ds and the target domain training set Dt: a. The samples are processed through a parallel feature extraction network, using the original input samples. They were respectively mapped to vibration feature sequences and current sequence , where B is the batch size, C is the number of channels (128), and T is the sequence length (4). b. Input the preliminary feature sequence into the feature correction module (FCM) and perform bidirectional adaptive calibration through a cross-modal guided mechanism to obtain the corrected feature sequence. , ; c. Input the corrected feature sequence into the feature fusion module (FFM), and perform deep interaction through a cross-attention mechanism to obtain fused features. ; d. Based on the fusion features The source domain classification loss Lcls is calculated based on the output of the fault classifier and the fused features. Constructing conditional features from category probability vector p ,Will Input domain discriminant and calculate conditional adversarial loss ; S3, based on the Lcls and The combined total loss is used to adversarially update the parameters of the feature extraction network, FCM, FFM, fault classifier, and domain discriminator through a gradient inversion layer (GRL) and backpropagation algorithm. S4. Repeat steps S2 and S3, monitor performance using the validation set, and save the optimal parameters.

[0005] Preferably, the data preprocessing in S1 specifically includes: a. For vibration signals and current signals, based on their minimum common length Perform truncation to achieve time alignment; b. Slicing is performed using a non-overlapping sliding window method with a window length W=1024; c. Stack vibration signal segments and current signal segments from the same window to construct a single... Dual-channel matrix samples ; ; d. For the matrix sample Min-MaxNormalization is performed to map the data range to the interval [0, 1].

[0006] Preferably, the feature extraction network in S2(a) is a dual-path parallel, weight-sharing four-level one-dimensional convolutional neural network (1D-CNN).

[0007] Preferably, the feature correction module (FCM) in S2(b) includes a feature sequence correction stage and a feature channel correction stage, and is balanced by a learnable balancing parameter. To automatically weigh the contributions of the two stages; The feature sequence correction stage (time-series correction) includes: Will and splicing along the channel dimension; ; A time-series correction weight map is learned through a lightweight multilayer perceptron (MLP) consisting of one-dimensional convolutions. This weight map acts like a dynamic time gating mechanism, capturing the local contribution or correlation of one mode to another at a specific time step. ; in, For weighted graphs, Output the weight map of the two channels, after... After activation, it decomposes into and ; According to the formula: ); ); in, Represents element-wise multiplication. It is a learnable balancing parameter used to control the strength of timing correction; Calculate the final corrected features; The feature channel correction stage includes: Features after timing correction and To splice; Global information summarization is performed on the concatenated features using global average pooling (torch.mean), global max pooling (torch.max), and global standard deviation pooling (torch.std), and then concatenated into a global statistical vector. Will Input a second multilayer perceptron (channel_mlp24) consisting of fully connected layers and a sigmoid activation function, and learn channel correction weights. ; ; ; According to the formula ); ; Calculate the final corrected features.

[0008] Preferably, the feature fusion module (FFM) in S2(c) specifically includes: a. Apply the corrected features through a linear layer (nn.Linear). and Generate query (Q), key (K), and value (V) vectors respectively; ; ; b. Bidirectional information enhancement is achieved through the multi-head attention mechanism (nn.MultiheadAttention), the calculation formula of which is shown below: ; ; Where Softmax is the normalization function. , The query vector and key vector are derived from the vibration signal. , The query vector and key vector are derived from the current signal. This is the scaling factor; c. The enhanced features are updated using residual connections and layer normalization (LayerNorm), calculated as follows: ; ; d. Enhance the features of the two paths and The concatenation is performed along the channel dimension (torch.cat), and dimensionality reduction and information integration are performed through a 1×1 Conv fusion bottleneck layer (fusion_bottleneck32) with 32 output channels to generate the fused features. .

[0009] Preferably, the loss calculation in S2(d) and the parameter optimization in S3 are achieved through a min-max adversarial game: a. The source domain classification loss Lcls uses the cross-entropy loss function (nn.CrossEntropyLoss). b. Conditional adversarial loss first converts the classifier output into a class probability vector using softmax. Then, by using tensor outer product, the class probability vector p and the feature vector are combined. By combining these, conditional features are constructed. : ; in, , D is the feature dimension, representing the generated conditional features. It is a matrix that encodes both the semantic information of the features themselves and the model's judgment on their category classification; Domain discriminator The input becomes a conditional feature It needs to determine the source domain of a sample in a space that combines features and class predictions. The total loss of the conditional adversarial domain is: ; in, X represents all samples from the source and target domains, and D is the corresponding domain label; c. The training objective of the model mainly consists of two parts: source domain classification loss. Combating loss against conditional domains The overall optimization objective of the model is achieved through a mini-maximum adversarial process, as shown in the equation: ; Among them, L cls To ensure the model's classification accuracy in the source domain, L adv The feature distributions of the source and target domains are aligned, and λ is a regularization coefficient used to balance the two parts of the loss. These represent feature extractors G and G, respectively. feat Bottleneck layer G bottle Classifier Gclf And the network parameters of the conditional domain discriminator D, in order to achieve domain adaptation, the feature extractor G feat The objective is the opposite of that of the domain discriminator D, namely, maximizing the loss L of the domain discriminator. adv This confuses the domain discriminator, making it unable to determine the source of the features.

[0010] Preferably, the parameter optimization in S3 uses the Adam optimizer; The training loop of S2 executes max_epoch rounds, and the batch size (batch_size) is set to a range of 16-512. The Calculation begins after the middle_epoch cycle; The GRL The coefficient dynamically increases from 0 to 1.

[0011] Domain-adaptive diagnostic systems based on multimodal feature fusion include: a. Data acquisition and segmentation module, used to acquire vibration and current signals from the source and target domains and construct them into a multi-channel sample set; b. Model training module, which performs a training loop, which includes a feature extraction module (parallel 1D-CNN). A feature correction module (FCM) is used to perform bidirectional adaptive calibration on the preliminary feature sequence; A Feature Fusion Module (FFM) is used to generate fused features through a cross-attention mechanism; A loss calculation module for calculating classification loss. and construct conditional features Calculate the conditions to counteract losses ; c. A parameter optimization module for adversarially updating the parameters of the other modules via a gradient inversion layer (GRL) and a backpropagation algorithm; d. Model saving module, used to save the optimal parameters during training.

[0012] A computer-readable storage medium on which a computer program is stored, wherein when the computer program is executed by a processor, the steps of the domain adaptive diagnostic method for multimodal feature fusion are implemented.

[0013] Compared with the prior art, the beneficial effects of the present invention are: the domain adaptive diagnostic method of multimodal feature fusion: To address the shortcomings of existing technologies in multimodal fault diagnosis of rotating machinery, such as shallow fusion strategies, modal noise interference, insufficient information exchange, and loss of discriminative information due to forced alignment, this paper proposes a cascaded feature cross-fusion domain adaptive fault diagnosis method and system. Through a feature correction module (FCM), a cross-modal guidance mechanism is employed to perform bidirectional adaptive calibration of the deep features of vibration and current signals. This design not only effectively suppresses noise interference specific to each mode but also aligns their feature distributions, providing a high-quality and more consistent feature foundation for subsequent deep fusion. The feature fusion module (FFM) further facilitates the calibration process. Based on the positive features, a cross-attention mechanism is introduced for deep interaction. It can adaptively and selectively enhance the most critical features for the diagnostic task and effectively integrate complementary information, thereby avoiding information annihilation caused by shallow splicing and fusion and loss of key discriminative information caused by forced modality alignment. Combined with Conditional Domain Adversarial Network (ACDANN) for category-aware accurate feature alignment, it can learn domain-invariant features that are insensitive to changes in working conditions and have stronger discriminative power. Experimental results show that the present invention can fully extract bimodal complementary features and significantly improve the diagnostic performance and robustness of cross-working-condition transfer tasks with large inter-domain differences. Attached Figure Description

[0014] Figure 1 This is a schematic diagram of the overall framework of the diagnostic network (FCFMAC-TL) of this invention; Figure 2 This is a schematic diagram of the feature correction module (FCM) of the present invention; Figure 3 This is a schematic diagram of the feature fusion module (FFM) of the present invention; Figure 4 This is a visualization comparison of the feature t-SNE dimensionality reduction of the method of this invention and several comparative methods on the transfer task T4; Figure 5 This is a comparison matrix of the diagnostic results confusion between the method of this invention and various comparative methods on migration task T4. Detailed Implementation

[0015] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0016] Please see Figures 1-5 This invention provides a technical solution: a domain adaptive diagnostic method based on multimodal feature fusion.

[0017] Example 1: A domain adaptive fault diagnosis method based on multimodal feature cross-fusion, comprising: S1. Acquire vibration and current signals of rotating machinery simultaneously under source conditions (source domain) and target conditions (target domain); The source and target domain signals are preprocessed and constructed into multi-channel samples, which are then divided into their respective training and validation sets. The source domain dataset is represented as:

[0018] The target domain dataset is represented as follows:

[0019] in, Represents the source domain. Indicates the first One source domain sample, Represents the set of all source domain samples. Indicates the first The labels of the source domain samples, where Indicates the number of fault categories. This represents the set of all source domain sample labels. Indicates the number of samples in the source domain; Indicates the target domain. Indicates the first One target domain sample, Represents the set of all samples in the target domain. Indicates the number of samples in the target domain; The vibration and current signals were acquired synchronously at the same sampling frequency of 64kHz, and each .mat file for both modes had the same sampling length.

[0020] The experimental dataset was used to validate the method in six cross-condition experiments, selecting one healthy category and six real damage categories. The seven categories selected were K001 (healthy state), KA16 (outer ring fatigue pitting), KA30 (outer ring plastic indentation), KB23 (multiple damage to inner and outer rings), KI14 (inner ring fatigue pitting), KI16 (inner ring fatigue pitting), and KI17 (inner ring repetitive fatigue pitting). Subsequently, the data of each category in each subset was divided into 1000 samples, each containing 1024 data points. When the dataset was used as the target domain, it was randomly divided into two parts: 80% of the unlabeled samples were used as the training set, and 20% of the unlabeled samples were used as the test set. The dataset was divided into four conditions according to different torque, load, and speed, as shown in Table 1.

[0021] Table 1

[0022] Based on the differences between the source and target domains, the migration tasks are divided into six different tasks, as shown in Table 2. Table 2

[0023] S2. In a training loop, perform the following operations on the data sampled from the source domain training set Ds and the target domain training set Dt: a. The samples are processed through a parallel feature extraction network, using the original input samples. They were respectively mapped to vibration feature sequences and current sequence , where B is the batch size, C is the number of channels (128), and T is the sequence length (4); b. Input the preliminary feature sequence into the feature correction module (FCM) and perform bidirectional adaptive calibration through a cross-modal guided mechanism to obtain the corrected feature sequence. , ; c. Input the corrected feature sequence into the feature fusion module (FFM), and perform deep interaction through a cross-attention mechanism to obtain fused features. ; d. Based on the fusion features The source domain classification loss Lcls is calculated based on the output of the fault classifier and the fused features. Constructing conditional features from category probability vector p ,Will Input domain discriminant and calculate conditional adversarial loss ; First, the original vibration and current signal time series were extracted separately. The dataset used in this study provides synchronously acquired multimodal sensing signals, ensuring that the vibration and current signals are precisely aligned in timestamps within any time window. Let the original signal be denoted as... and M and N represent the number of sampling points for the two signals, respectively. Since the sampling lengths of the two signals may have slight differences, they are first aligned in length, and both signals are truncated to their minimum common length. .

[0024] Next, a non-overlapping sliding window method is used to segment the aligned continuous signal into independent samples. The window length W is fixed at 1024. For the i-th sample, its corresponding vibration signal segment... and It can be represented as: ; ; in and These represent the amplitudes of the vibration and current signals related to the climate at time t, respectively.

[0025] Vibration and current signal segments acquired within the same time window are stacked to construct a multi-channel input sample. This sample is The matrix is ​​given, where the first row represents the vibration signal and the second row represents the current signal. Therefore, any sample input into the model... The mathematical form is as follows: ; Finally, for each constructed sample Min-MaxNormalization is performed to linearly map the data range of each sample to the interval [0, 1].

[0026] A dual-path parallel one-dimensional convolutional network (1D-CNN) is used to process the input data of vibration and current modes. Specifically, the feature extractor of each mode adopts a four-level one-dimensional convolutional network with shared weights. This network progressively increases the number of feature channels from 1 to 128 through stacked convolutional modules. The network integrates max pooling and adaptive max pooling operations to reduce the sequence dimension and finally output a feature sequence of fixed length 4 for use by subsequent modules. All convolutional layers are connected to batch normalization layers and ReLU activation functions.

[0027] After passing through this feature extraction network, the original input sample They were respectively mapped to vibration feature sequences and current sequence Where B is the batch size, C is the number of channels (128), and T is the sequence length (4).

[0028] The core objective of the feature correction module is to use the features of one modality to dynamically and adaptively adjust the features of another modality, thereby achieving bidirectional feature calibration. While retaining the advantages of each modality, it effectively overcomes the inherent differences and noise interference between modalities.

[0029] S3, based on the Lcls and The combined total loss is used to adversarially update the parameters of the feature extraction network, FCM, FFM, fault classifier, and domain discriminator through a gradient inversion layer (GRL) and backpropagation algorithm. S4. Repeat steps S2 and S3, monitor performance using the validation set, and save the optimal parameters.

[0030] In S1, the original vibration and current signal time series are extracted respectively. The PU dataset used in this study provides synchronously acquired multimodal sensing signals, ensuring that the vibration and current signals are precisely aligned in timestamps within any time window. Let the original signal be denoted as... and M and N represent the number of sampling points for the two signals, respectively. Since the sampling lengths of the two signals may have slight differences, they are first aligned in length, and both signals are truncated to their minimum common length. .

[0031] In step S1, a non-overlapping sliding window method is used to segment the aligned continuous signal into independent samples. The window length W is fixed at 1024. For the i-th sample, its corresponding vibration signal segment and It can be represented as: ; ; in and These represent the amplitudes of the vibration and current signals related to the climate at time t, respectively. Vibration and current signal segments collected within the same time window are stacked to construct a multi-channel input sample. This sample is The matrix is ​​given, where the first row represents the vibration signal and the second row represents the current signal. Therefore, any sample input into the model... The mathematical form is as follows: ; Finally, for each constructed sample Min-MaxNormalization is performed to linearly map the data range of each sample to the interval [0, 1].

[0032] S1 uses a dual-path parallel one-dimensional convolutional network. A 1D-CNN is used to process input data for both vibration and current modes. Specifically, the feature extractor for each mode employs a four-level 1D convolutional network with shared weights. This network progressively increases the number of feature channels from 1 to 128 through stacked convolutional modules. The network integrates max pooling and adaptive max pooling operations to reduce sequence dimensionality and ultimately output a fixed-length feature sequence of 4 for subsequent modules. All convolutional layers are followed by batch normalization layers and ReLU activation functions. After passing through this feature extraction network, the original input samples... They were respectively mapped to vibration feature sequences and current sequence Where B is the batch size, C is the number of channels (128), and T is the sequence length (4).

[0033] The process is designed with two complementary stages: timing correction that focuses on local dynamics and channel correction that focuses on global information. The contributions of the two stages are automatically balanced by a learnable parameter.

[0034] For the temporal correction stage, this stage aims to perform fine-grained local calibration at each time step of the feature sequence. To achieve this, the features of the two modalities are first concatenated in the channel dimension, and then a lightweight multilayer perceptron (MLP) composed of one-dimensional convolutions is used to learn the temporal correction weight map. This weight map acts like a dynamic temporal gating mechanism, which can capture the local contribution or correlation of one modality to another at a specific time step.

[0035] ; ; in, For weighted graphs, Output the weight map of the two channels, after... After activation, it decomposes into and Features after time-correction and The calculation is as follows: ); ); in, Represents element-wise multiplication. It is a learnable balancing parameter used to control the strength of timing correction.

[0036] In the channel feature correction stage, the aim is to calibrate the channel information of features from a global perspective to capture the macroscopic statistical dependencies between modes. First, a global information summary is performed on the time-corrected features. To obtain a more comprehensive description of the feature distribution, global average pooling, max pooling, and standard deviation pooling are used. These three pooling methods capture the average response strength (central tendency), peak response strength (salient features), and response dispersion (stability) of the feature channels, respectively, thus forming a more informative global statistical descriptor. .

[0037] ; ; Where A, M, and S represent average pooling, max pooling, and standard deviation pooling, respectively.

[0038] global statistical vector Input an MLP consisting of fully connected layers, and learn the correction weights at the channel level. .

[0039] ; Weight Decomposed into and The final corrected features , The calculation is as follows: ); ; The core of the feature fusion module is to introduce an efficient multi-head cross-attention mechanism that allows features from two modalities to interact globally, thereby making the most effective use of their complementary information. It allows a feature from one modality (as a query) to actively and selectively "extract" the most useful information from features from another modality (as keys and values), thus achieving bidirectional information enhancement. The multi-head cross-attention mechanism enables the interaction of global information and fully captures the complex nonlinear relationships between different modal data.

[0040] The input is already two one-dimensional feature sequences after FCM correction. and The two sequences are processed through a linear layer to generate Query(Q), Key(K), and Value(V) vectors respectively.

[0041] ; ; The Linear layer is a linear transformation layer that performs a linear mapping on the input feature sequence to generate the three vectors (K, Q, V) in the attention mechanism. The core of cross-attention is that the Q of one mode is calculated together with the K and V of another mode. Taking the enhancement of vibration features as an example, this paper uses the vibration query vector. bond vector of current The attention score is calculated by normalizing the matrix using Softmax. Each element of this matrix represents a position in the vibration feature sequence and the degree of attention paid to each position in the current feature sequence.

[0042] ; ; This process is executed in parallel for multiple attention heads, allowing the model to capture dependencies at different levels in different representation subspaces. The same applies to the current, which is executed multiple times for each head. The inputs K, Q, and V will each be processed through h independent linear projection layers to generate h different sets of... , , A vector, where i = 1, 2, ..., h.

[0043] ; in, , , It is the learnable projection matrix of the i-th attention head.

[0044] By combining the attention weight matrix with the current value vector Multiplication yields a summary of information most relevant to the "query" of vibration modes extracted from the current modes. This summary is then used to update vibration features, and the enhancement process for current features is similar, thus achieving bidirectional information complementarity.

[0045] The enhanced features are residually connected to the original input features, and training is stabilized by layer normalization (LayerNorm).

[0046] ; ; Finally, the two bidirectionally enhanced feature sequences are concatenated along the channel dimension, and then a bottleneck convolutional layer (1×1 Conv) is used for final dimensionality reduction and information integration. This 1×1 convolutional layer acts like a cross-channel fully connected network, which can efficiently fuse the information of the two enhanced modalities and generate a final high-quality single fused feature sequence. This is for use by downstream conditional domain adversarial modules.

[0047] ;

[0048] The conditional adversarial module incorporates category information into adversarial training to achieve more refined category-aware domain alignment. Its core idea is to match the conditional distribution of features. and This ensures that the alignment process revolves around the category structure.

[0049] First, the classifier output is converted into a class probability vector using softmax. Then, by using tensor outer product, the class probability vector p and the feature vector are combined. By combining these, conditional features are constructed. : ; in, , D is the feature dimension, representing the generated conditional features. It is a matrix that encodes both the semantic information of the features themselves and the model's judgment on their category classification.

[0050] For step 3, the domain discriminator The input becomes a conditional feature c, it needs to determine the source domain of the sample in a space that combines features and class predictions. The total loss of the conditional adversarial domain is: ; in, X represents all samples from the source and target domains, and D is the corresponding domain label.

[0051] The training objective of the model mainly consists of two parts: source domain classification loss. Combating loss against conditional domains The overall optimization objective of the model is achieved through a mini-maximum adversarial process, as shown in the equation: ; in, Ensure the model's classification accuracy in the source domain. Used to align the feature distributions of the source and target domains. It is a regularization coefficient used to balance the two parts of the loss. These represent feature extractors G and G, respectively. feat Bottleneck layer G bottle Classifier G clf And the network parameters of the conditional domain discriminator D.

[0052] To achieve domain adaptation, the feature extractor G... feat The objective is the opposite of that of the domain discriminator D, namely, to maximize the loss of the domain discriminator. This confuses the domain discriminator, preventing it from determining the source of the features. Therefore, in the conditional adversarial total loss formula, the feature extractor G... feat The optimization objective is to be in L adv A negative sign is added to indicate that it maximizes the objective. In addition, by adopting a conditional domain adversarial strategy, the model considers class information when aligning feature distributions, so that features of the same type of samples in different domains can be aligned more accurately, thereby improving the model's generalization ability in the target domain. The update process of each parameter is shown in the following formula: ; in, Indicates the learning rate. The partial derivative operator is used to adopt a conditional domain adversarial strategy. When aligning feature distributions, the model will consider class information, so that the features of the same type of samples in different domains can be more accurately aligned, thereby improving the model's generalization ability in the target domain. Furthermore, the specific process of step 4 is to select the learning rate, optimizer, training batch size and number of training rounds of the network, input the obtained training dataset into the set fault diagnosis model, and obtain the best fault diagnosis model. All modules are trained using the Adam optimizer. To ensure the stability of the adversarial training process and the convergence of the model, this paper adopts a dynamically adjusted learning rate strategy instead of a fixed step-down approach. The initial learning rate η is set to 0.001 and gradually decays according to the training progress. In each training epoch p, the learning rate is updated according to the following formula: ; Where p is the current epoch, P is the total epochs, the training epochs are set to 200, the batch size is 64, and the parameters α, β, and ... The values ​​are set to 0.1, 0.05, and 0.001 respectively. The dual-path parallel 1D-CNN output channels of the feature extractor are set to 128. After feature fusion, a fully connected layer with a dimension of 256 is used as the bottleneck layer. The number of heads in the multi-head cross attention in FFM is set to 8. The hidden layer dimensions of the domain discriminator in CDA are 1024 and 256 respectively. The tradeoff coefficient λ in GRL dynamically increases from 0 to 1 as the training progresses. Domain adaptive training is started after the 50th epoch.

[0053] This invention selects diagnostic accuracy, which is the most representative indicator in classification tasks, as the core evaluation metric. This metric directly reflects the model's ability to correctly classify all fault categories on the target domain test set. Its calculation formula is as follows: ; Where TP (TruePositives) is the number of correctly predicted positive samples, TN (TrueNegatives) is the number of correctly predicted negative samples, and FP (FalsePositives) and FN (FalseNegatives) are the number of incorrectly predicted positive and negative samples, respectively.

[0054] This paper selects four methods for comparative experiments, including a classification method without domain adaptation, a baseline method with implicit domain adaptation, a generative graph adversarial network method for unimodal datasets, and an advanced fault diagnosis method for bimodal datasets of PU fault diagnosis datasets. Through comparative analysis on the same task, the advantages and potential limitations of the proposed methods are demonstrated.

[0055] (1) 1D-CNN: It consists of a feature extractor and a classifier. It uses a one-dimensional convolutional neural network (1DCNN) as the feature extractor and compares it with the method proposed in this paper. This method does not involve knowledge transfer, but only uses source domain data for model training and is directly applied to the diagnostic task of the target domain. It uses single-mode vibration signals for experiments.

[0056] (2) DANN: DANN is a closed-domain adaptive method based on adversarial learning. The relevant details have been introduced in Section 1.2, and the method has been verified using simple concatenation of bimodal data.

[0057] (3) DAGCN: DAGCN is an adaptive method for fault diagnosis of single-mode rotating machinery. This method constructs feature map data with connection information by feature mapping and classifies it by using graph convolutional network as the backbone network.

[0058] (3) DS-HDA: DS-HDA is a recent advanced method for bimodal hybridization based on graph construction and bimodal adversarial fusion. It uses a domain adversarial network to confuse the cross features of the two modalities and uses a graph network to connect the nodes of the hybrid features to generate features that are easier to transfer.

[0059] (4) FCFMAC-TL: The complete model proposed in this paper.

[0060] The experimental results are the average diagnostic accuracy (%) after 5 independent repeated experiments to ensure the stability and reliability of the conclusions.

[0061] The results of each comparative experiment are shown in Table 3: Table 3

[0062] As shown in Table 3, the method proposed in this invention achieved the best accuracy in all five different tasks, and the effectiveness of the method proposed in this invention is demonstrated through the examples.

[0063] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A domain-adaptive diagnostic method based on multimodal feature fusion, comprising the following steps: S1. Acquire vibration and current signals of rotating machinery synchronously under source and target operating conditions; preprocess the source and target signals and construct them into multi-channel samples, and divide them into their respective training and validation sets; S2. In a training loop, the training set D from the source domain... s and the target domain training set D t Perform the following operations on the sampled data: a. The samples are processed through a parallel feature extraction network, using the original input samples. They were respectively mapped to vibration feature sequences and current sequence Where B is the batch size, C is the number of channels, and T is the sequence length; b. Input the preliminary feature sequence into the feature correction module FCM, and perform bidirectional adaptive calibration through a cross-modal guidance mechanism to obtain the corrected feature sequence. , ; c. Input the corrected feature sequence into the feature fusion module FFM, and perform deep interaction through a cross-attention mechanism to obtain fused features. ; d. Based on the fusion features Calculate the source domain classification loss L based on the output of the fault classifier. cls And based on the fusion features Constructing conditional features from category probability vector p ,Will Input domain discriminant and calculate conditional adversarial loss ; S3, based on the L cls and The combined total loss is used to adversarially update the parameters of the feature extraction network, FCM, FFM, fault classifier, and domain discriminator through the gradient inversion layer GRL and the backpropagation algorithm; S4. Repeat steps S2 and S3, monitor performance using the validation set, and save the optimal parameters.

2. The domain adaptive diagnostic method for multimodal feature fusion according to claim 1, characterized in that: The data preprocessing in S1 specifically includes: a. For vibration signals and current signals, based on their minimum common length To achieve time alignment, the data is truncated. b. Slicing is performed using a non-overlapping sliding window method with a window length of W; c. Stack vibration signal segments and current signal segments from the same window to construct a single... Dual-channel matrix samples ; ； d. For the matrix sample Perform min-max normalization to map the data range to the interval [0, 1].

3. The domain adaptive diagnostic method for multimodal feature fusion according to claim 1, characterized in that: The feature extraction network in S2(a) is a dual-path parallel, weight-sharing four-level one-dimensional convolutional neural network (1D-CNN).

4. The domain adaptive diagnostic method for multimodal feature fusion according to claim 1, characterized in that: The feature correction module FCM in S2(b) includes a feature sequence correction stage and a feature channel correction stage, and is achieved through a learnable balance parameter. To automatically weigh the contributions of the two stages; The feature sequence correction stage includes: Will and splicing along the channel dimension; ； A time-series correction weight map is learned through a lightweight multilayer perceptron (MLP) consisting of one-dimensional convolutions. This weight map acts like a dynamic time gating mechanism, capturing the local contribution or correlation of one mode to another at a specific time step. ； in, For weighted graphs, Output the weight map of the two channels, after... After activation, it decomposes into and ; According to the formula: )； )； in, Represents element-wise multiplication. It is a learnable balancing parameter used to control the strength of timing correction; Calculate the final corrected features; The feature channel correction stage includes: Features after timing correction and To splice; Global average pooling, global max pooling, and global standard deviation pooling are used to perform global information summarization on the concatenated features, and then concatenated into a global statistical vector. Will Input a second multilayer perceptron consisting of fully connected layers and a sigmoid activation function, and learn channel correction weights. ; ； ； According to the formula )； ； Calculate the final corrected features.

5. The domain adaptive diagnostic method for multimodal feature fusion according to claim 1, characterized in that: The feature fusion module FFM in S2(c) specifically includes: a. The corrected features are processed through a linear layer. and Generate vectors for query Q, key K, and value V respectively; ； ； b. Bidirectional information enhancement is achieved through a multi-head cross-attention mechanism, the calculation formula of which is shown below: ； ； Where Softmax is the normalization function. , The query vector and key vector are derived from the vibration signal. , The query vector and key vector are derived from the current signal. This is the scaling factor; c. The enhanced features are updated through residual connections and layer normalization, and the calculation formula is as follows: ； ； d. Enhance the features of the two paths and The concatenation is performed along the channel dimension (torch.cat), and dimensionality reduction and information integration are performed through a 1×1 Conv fusion bottleneck layer with 32 output channels to generate the fused features. .

6. The domain adaptive diagnostic method for multimodal feature fusion according to claim 1, characterized in that: The overall optimization objective of the loss calculation in S2(d) and the parameter optimization in S3 is achieved through a mini-maximum adversarial game: a. Source domain classification loss Lcls uses the cross-entropy loss function. b. Conditional adversarial loss first converts the classifier output into a class probability vector using softmax. Then, by using tensor outer product, the class probability vector p and the feature vector are combined. By combining these, conditional features are constructed. : ； in, , D is the feature dimension, representing the generated conditional features. It is a matrix that encodes both the semantic information of the features themselves and the model's judgment on their category classification; Domain discriminator The input becomes conditional features It needs to determine the source domain of a sample in a space that combines features and class predictions. The total loss of the conditional adversarial domain is: ； in, X represents all samples from the source and target domains, and D is the corresponding domain label; c. The training objective of the model mainly consists of two parts: source domain classification loss. Combating loss against conditional domains The overall optimization objective of the model is achieved through a mini-maximum adversarial process, as shown in the equation: ； Among them, L cls To ensure the model's classification accuracy in the source domain, L adv The feature distributions of the source and target domains are aligned, and λ is a regularization coefficient used to balance the two parts of the loss. These represent feature extractors. Bottleneck layer Classifier And the network parameters of the conditional domain discriminator D, in order to achieve domain adaptation, feature extractor The objective is the opposite of that of the domain discriminator D, namely, maximizing the loss of the domain discriminator. This confuses the domain discriminator, making it unable to determine the source of the features.

7. The domain adaptive diagnostic method for multimodal feature fusion according to claim 1, characterized in that: The parameter optimization in S3 uses the Adam optimizer; The training loop of S2 executes max_epoch rounds, and the batch size is set in the range of 16-512. The Calculation begins after the middle_epoch cycle; The GRL The coefficient dynamically increases from 0 to 1.

8. A domain adaptive diagnostic system based on multimodal feature fusion, characterized in that, The diagnostic system is used to execute the domain adaptive diagnostic method based on multimodal feature fusion as described in any one of claims 1-7, including: a. Data acquisition and segmentation module, used to acquire vibration and current signals from the source and target domains and construct them into a multi-channel sample set; b. Model training module, which executes a training loop, which includes a feature extraction module; A feature correction module (FCM) is used to perform bidirectional adaptive calibration on the preliminary feature sequence; A feature fusion module (FFM) is used to generate fused features through a cross-attention mechanism; A loss calculation module for calculating classification loss. and construct conditional features Calculate the conditions to counteract losses ; c. A parameter optimization module for adversarially updating the parameters of the other modules using a gradient inversion layer (GRL) and a backpropagation algorithm; d. Model saving module, used to save the optimal parameters during training.

9. A computer-readable storage medium, characterized in that: A computer program is stored on the computer-readable storage medium, which, when executed by a processor, implements the steps of the domain adaptive diagnostic method for multimodal feature fusion as described in any one of claims 1 to 7.

Images