Modality missing external gear pump fault diagnosis method based on multi-modal fusion

Abstract

The present application relates to the technical field of artificial intelligence and fault diagnosis, and particularly relates to a modal missing external gear pump fault diagnosis method based on multi-modal fusion.The model structure provided by the present application comprises a modal learning module, a physical inspiration module and a feature fusion module;the modal learning module encodes modal signals, and divides the encoded features into modal unique features and shared features of two modes, and the modal learning module constructs final shared features based on the obtained shared features;the physical inspiration module fuses the encoded feature coefficients obtained by encoding the pressure signals and the frequency domain feature coefficients obtained by frequency domain extraction to obtain physical features;the feature fusion module fuses the final shared features, the modal unique features and the physical features to obtain fused features;and then, fault diagnosis is performed based on the fused features.Through joint learning of modal shared features and modal exclusive features, the fault information of each mode is fully utilized, the modal missing problem is solved, and the diagnosis accuracy is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of artificial intelligence and fault diagnosis, and in particular to a modal missing external gear pump fault diagnosis method based on multi-modal fusion. BACKGROUND

[0002] The external gear pump is a kind of rotating machinery, and its key function in the system is highlighted as the core component of the aircraft engine fuel system. However, due to the harsh working conditions such as high pressure, strong disturbance and long time running in rotating mode, the key components of the gear pump are prone to various faults, which is a common challenge faced by rotating machinery in harsh environments.

[0003] Multi-modal learning has attracted extensive research interest in the field of fault diagnosis, as it can effectively utilize the diversified data collected by multiple sensors of equipment. Diagnosis experts often integrate multi-modal information for mechanical system fault diagnosis.

[0004] Multi-modal learning that integrates fault information captured by different sensors is crucial for effectively utilizing multi-modal data for fault diagnosis. However, despite recent progress, the application of these methods in real-world scenarios still faces challenges, as they often require complete modal data in both training and testing phases.

[0005] In the real world, various factors make it difficult to obtain complete multi-modal data. For example, limited space constraints, sensor reliability and deployment costs pose significant challenges to comprehensive monitoring of aircraft systems. In addition, strong external excitation during long flights can accelerate sensor wear and failure, resulting in modal missing. Therefore, unlike controlled experimental environments, real-world applications often face the dilemma of multi-source data missing. Taking the gear pump as an example, it is easy to collect multi-modal data such as pressure, vibration and sound in the laboratory environment, providing rich information for fault diagnosis model development; but in actual working conditions, hydraulic gear pumps usually only install anti-interference pressure sensors, and vibration sensors are rarely configured. How to ensure diagnostic accuracy while preserving effective information in multi-modal data under the condition of missing data modalities constitutes a major challenge. SUMMARY

[0006] In order to overcome the problem of multi-modal learning in fault diagnosis facing modal missing in the prior art, the present application proposes a modal missing external gear pump fault diagnosis method based on multi-modal fusion, which can ensure the accuracy of fault diagnosis under the condition of modal missing.

[0007] The application provides a modal missing external gear pump fault diagnosis method based on multi-modal fusion, which comprises the following steps: firstly, training a basic model on a labeled mixed data set containing multi-modal and single-modal; the multi-modal comprises pressure signals and vibration signals, and the single-modal is the pressure signal; the basic model comprises a modal learning module, a physical inspiration module, a feature fusion module and five classifiers;

[0008] The modal learning module encodes the modal signals and divides the encoded features into modal-specific features and shared features of the two modalities, and the modal learning module constructs final shared features based on the obtained shared features; the physical inspiration module fuses the encoded feature coefficients obtained by encoding the pressure signals and the frequency domain feature coefficients obtained by frequency domain extraction to obtain physical features; the feature fusion module fuses the final shared features, the modal-specific features and the physical features to obtain fusion features; the five classifiers respectively predict the fault categories of the samples based on the vibration-specific features, the shared features, the pressure-specific features, the physical features and the fusion features;

[0009] The basic model is trained in combination with the classification loss of the five classifiers, after the training is completed, the modal learning module, the physical inspiration module, the feature fusion module and the classifier based on the fusion features for predicting the fault category are extracted to form a fault diagnosis model, which is used for diagnosing the fault categories of the multi-modal samples and the single-modal samples.

[0010] Preferably, for the multi-modal samples, the modal learning module encodes and cuts the pressure signals and the vibration signals respectively, then fuses the shared features obtained by cutting the encoded features of the two, obtains the final shared features and outputs, and the corresponding classifier and the feature fusion module process based on the final shared features; for the single-modal samples, the final shared features are the shared features corresponding to the pressure signals;

[0011] The final shared features of the multi-modal samples are obtained in the following manner: firstly, the shared features corresponding to the two signals are respectively processed by using an activation function and then averaged, and then the inverse operation of the activation function is performed on the average result to obtain the final shared features.

[0012] Preferably, the feature fusion module adopts a scaling dot product attention mechanism to process the final shared features , the physical features and the modal-specific features , and obtains the fusion features , the modal-specific features include the pressure-specific features and the vibration-specific features .

[0013] Preferably:

[0014] ;

[0015] ;

[0016] wherein, is an intermediate term, denotes a projection matrix in the scaled dot-product attention mechanism; is the attention weight vector corresponding to the fault type s , S is the total number of fault categories.

[0017] Preferably:

[0018] ;

[0019] ;

[0020] wherein, is a query vector, is a target vector of the sample under the fault type s, is a modal related mask vector, d is the dimension of the vector; is a vibration signal, m is the sample number;

[0021] ;

[0022] wherein, is a linear projection matrix for calculating the query vector .

[0023] Preferably, the loss function L in the model training process is denoted as:

[0024] ;

[0025] wherein, is a classification loss of the basic model based on the fusion features for diagnosis, is a modal alignment loss of the shared features from the vibration signal and the shared features from the pressure signal; is a difference loss between the final shared features and the pressure-specific features, is a difference loss between the vibration-specific features and the final shared features; is a signal reconstruction loss of the physical heuristic module based on the physical features to reconstruct the pressure signal; is a fault-aware loss, which is used to measure the ability of the final shared features, the pressure-specific features, the physical features and the vibration-specific features to predict the fault label; is a ranking loss, which makes the model pay more attention to the features with lower predicted loss;

[0026] λ1, λ2, λ3, λ4, λ5 are all set weight coefficients.

[0027] Preferably:

[0028] ;

[0029] is a binary number, when , , on the contrary, =0;1≤i≤4,1≤j≤4, , , and respectively represent the attention weight of the vibration-specific feature, the final shared feature, the pressure-specific feature and the physical feature; , , and respectively represent the classification loss corresponding to the vibration-specific feature, the final shared feature, the pressure-specific feature and the physical feature; is a set bias term.

[0030] Preferably, The sum of the classification losses based on the final shared feature, the pressure-specific feature, the physical feature and the vibration-specific feature under each fault category is adopted.

[0031] Preferably:

[0032] ;

[0033] wherein, is a shared feature from the vibration signal, is a shared feature from the pressure signal, and Z is the mean of the shared features and the shared features ; and KL is a divergence function.

[0034] The application provides a modal missing external meshing gear pump fault diagnosis system based on multi-modal fusion, which comprises a memory and a processor, the memory stores a computer program, the processor is connected with the memory, and the processor is used for executing the computer program to realize the modal missing external meshing gear pump fault diagnosis method based on multi-modal fusion.

[0035] The application has the following advantages:

[0036] (1) The modal missing external meshing gear pump fault diagnosis method based on multi-modal fusion provided by the application mainly processes the modal missing problem through two modules: one is a modal learning module, which can distinguish the shared fault features and exclusive features of each modal, and ensure that valuable information can be retained even if a modal is missing; the other is a physical inspiration module, which introduces physical principles to guide fault feature learning and improve the diagnosis accuracy.

[0037] (2) The present application shares features of modalities and modal-specific features through joint learning, and fully utilizes fault information of each modality. The method can effectively deal with the problem of missing modalities, because the modal shared features retain the fault characteristics of the missing modality data. At the same time, the physical inspiration module is a novel physical knowledge guided network that extracts interpretable features from available modality data, integrates the physical knowledge guided feature mask, and uses the pressure reconstruction model to guide and constrain the feature learning process. The present application proposes the first model structure that combines data-driven and physics-driven methods to solve the problem of missing modalities of external gear pumps, and fully captures the fault features embedded in vibration signals and pressure signals, so that it can achieve performance comparable to that of the full modality scenario even in the case of only pressure measurement data.

[0038] (3) Through the combination of the modal learning module and the physical inspiration module, the present application has a diagnosis effect close to the full modality mode in the case of missing modalities. It fills the gap between laboratory multi-modality methods and industrial single-modality methods, and provides a powerful tool for rotating machinery maintenance.

[0039] (4) The present application also proposes a fault perception weighting mechanism through the loss function, which gives priority to features that contribute most to each type of fault. BRIEF DESCRIPTION OF DRAWINGS

[0040] Figure 1 Figure 1 is a schematic diagram of the training process of the base model;

[0041] Figure 2 Figure 2 is a schematic diagram of the diagnosis process of the fault diagnosis model in the full modality;

[0042] Figure 3 Figure 3 is a schematic diagram of the diagnosis process of the fault diagnosis model in the missing modality;

[0043] Figure 4(a) is the feature visualization result of the model PIMFNet of the present application in the full modality in the embodiment;

[0044] Figure 4(b) is the feature visualization result of the ablation model no phy in the full modality in the embodiment;

[0045] Figure 4(c) is the feature visualization result of the comparative model MedFuse in the full modality in the embodiment;

[0046] Figure 4(d) is the feature visualization result of the comparative model MMTM in the full modality in the embodiment;

[0047] Figure 4(e) is the feature visualization result of the comparative model DAFT in the full modality in the embodiment;

[0048] Figure 5(a) is the feature visualization result of the model PIMFNet of the present application in the missing modality in the embodiment;

[0049] Figure 5(b) shows the feature visualization results of the ablation model no phy in the missing modality in the embodiment;

[0050] Figure 5(c) shows the feature visualization results of the MedFuse comparative model in the example on the missing modality;

[0051] Figure 5(d) shows the feature visualization results of the comparison model MMTM on the missing modalities in the embodiment;

[0052] Figure 5(e) shows the feature visualization results of the comparative model DAFT on the missing modalities in the embodiment;

[0053] Figure 6 This example compares the original and reconstructed signals of the physical revelation module. Detailed Implementation

[0054] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0055] like Figure 1 As shown, the fault diagnosis method for external gear pumps based on modal sharing and physical inspiration proposed in this invention first involves a multimodal dataset. and single-modal datasets The base model is trained on the mixed dataset. , and These represent the vibration signal, pressure signal, and fault label of the m-th multimodal sample, respectively; M is the number of multimodal samples. This indicates that the vibration signal is missing from the single-mode sample; and These represent the pressure signal and fault label of the m'th single-mode sample, respectively.

[0056] The basic model includes: a modality learning module, a physics-inspired module, a feature fusion module, and five classifiers, which are referred to as the first classifier, the second classifier, the third classifier, the fourth classifier, and the terminal classifier, respectively.

[0057] The modality learning module includes a first encoder, a first segmentation unit, a second encoder, a second segmentation unit, and an output layer;

[0058] Vibration signal The encoded features from the first encoder are then divided into vibration-specific features by the first segmentation unit. and the first shared feature wherein, is the time series length; is the number of features of the vibration signal, i.e., the number of vibration sensors; the pressure signal The encoded features passing through the second encoder are divided by the second segmentation unit into pressure-specific features and the second shared feature ; d is the feature dimension;

[0059] the first shared feature is the shared feature from the vibration signal, the second shared feature is the shared feature from the pressure signal, the shared feature contains the modal feature shared by the vibration signal and the pressure signal; the modal learning module assumes and are emission variables subject to probability distributions and respectively, i.e. subject to probability distribution , subject to probability distribution , represents the sigmoid function.

[0060] The output layer outputs the vibration-specific feature , the pressure-specific feature , and the final shared feature .

[0061] (1);

[0062] wherein, represents the case where the sample contains the vibration modal, i.e., a multi-modal sample; represents the inverse function of the sigmoid function.

[0063] The modal learning module aligns the two distributions by minimizing the alignment loss of the two probability distributions, thereby ensuring that when the vibration signal is missing, the modal can also be supplemented by the second shared feature to ensure the stability and robustness of the model.

[0064] Specifically, the alignment loss L JS is:

[0065] (2);

[0066]

[0067] wherein, Z is the first shared feature and the second shared feature the mean of ; KL is the special symbol of Kullback-Leibler (KL) divergence, is and the KL divergence of, is and the KL divergence of.

[0068] The physical inspiration module includes an encoder, a physical guiding unit and a signal reconstruction unit.

[0069] The physical guiding unit is used to map the input pressure signal P(t) to the frequency domain feature coefficients a i ,b j |1≤ i ≤I,1≤j≤J}; I and J are empirical values, respectively representing the number of low frequency components and the number of high frequency components after the frequency domain transformation of P(t); for the pressure signal there is P(t)= .

[0070] The mapping model is:

[0071] (3);

[0072] (4);

[0073] (5);

[0074] (6);

[0075] (7);

[0076] f 1(t) represents the low frequency component of the frequency domain transformation of P(t), f 2(t) represents the high frequency component of the frequency domain transformation of P(t), a i and are the amplitude and phase angle of the i-th low frequency component, respectively, b j , c j , d j and are the amplitude, amplitude modulation coefficient, frequency modulation coefficient and phase angle of the j-th high frequency component, respectively; in the model training, a i and b j , c j , d j are the coefficients to be learned; ω is the meshing frequency, l is the number of gears, and t is the time length; To concentrate on High-frequency components, To focus on ( The high-frequency components.

[0077] The physical guidance unit guides physical information based on the frequency domain variation of the pressure signal P(t), extracting the amplitude of the low-frequency component. a i and the amplitude b of the high-frequency component j Constitutes frequency domain characteristic coefficients { a i ,b j |1≤ i ≤I,1≤j≤J};

[0078] The encoder encodes the pressure signal P(t) to obtain the encoded feature coefficients. Specifically, the encoder can use a ResNet50 network.

[0079] The frequency domain feature coefficients and coding feature coefficients are superimposed by dimension to obtain the fault feature coefficients, and the signal reconstruction unit reconstructs the pressure signal based on the fault feature coefficients.

[0080] Let the reconstructed signal of P(t) be denoted as , These are the fault characteristic coefficients;

[0081] (8);

[0082] in, The prediction coefficients for the nth frequency component of the frequency domain transform of P(t); for a i The predicted value, For b j The predicted value;

[0083] (9);

[0084] (10);

[0085] in, This represents frequency domain transformation, specifically using Fourier transform; Indicates coefficient extraction; This refers to the encoder, which can specifically use a ResNet50 network; and They represent The corresponding frequency components are located in the frequency spectrum and their phase angles, where ω is the meshing frequency. This indicates the offset item.

[0086] Signal reconstruction loss of a physics-inspired module The two-norm square of the original input signal P(t) and the reconstructed signal .

[0087] The physical inspiration module linearly converts the fault feature coefficients into physical features and outputs them. The fourth classifier predicts the fault classification based on the physical features .

[0088] The feature fusion module processes the final shared features , the stress-specific features , the physical features , and the vibration-specific features using a scaled dot-product attention mechanism to obtain fusion features .

[0089] (11);

[0090] (12);

[0091] (13);

[0092] (14);

[0093] where H is an intermediate term; For single-modal samples, the vibration-specific features are empty sets and are processed by zeroing;

[0094] is the attention weight vector corresponding to the fault type s, and , , , and represent the attention weights of the vibration-specific features , the final shared features , the stress-specific features , and the physical features , respectively; S is the total number of fault categories, and the superscript T represents matrix transposition; represents the projection matrix in the scaled dot-product attention mechanism; is the query vector of the overall state, is the target vector of the sample under the fault type s, is the linear projection matrix for calculating the query vector , is the linear projection matrix for calculating the target vector ; , and are the learning hyperparameters of the scaled dot-product attention mechanism in the feature fusion module; d is the vector dimension, is the modal related mask vector ; wherein the values "1" and "-∞" are the modal attention scores, and "-∞" means that by setting the attention of the missing modal to a very small value (usually -∞), it is ensured that the missing modal gets a probability close to zero after the softmax operation.

[0095] The five classifiers respectively predict the fault classification based on the vibration-specific features , the final shared features , the pressure-specific features , the physical features and the fusion features .

[0096] The cross-entropy loss of the prediction result of the first classifier based on the vibration-specific features and the true label is denoted as the prediction loss L s1 , the cross-entropy loss of the prediction result of the second classifier based on the final shared features and the true label is denoted as the prediction loss L s2 , the cross-entropy loss of the prediction result of the third classifier based on the pressure-specific features and the true label is denoted as the prediction loss L s3 , the cross-entropy loss of the prediction result of the fourth classifier based on the physical features and the true label is denoted as L s4 ; and the cross-entropy loss of the prediction result of the end classifier based on the fusion features and the true label is denoted as the prediction loss .

[0097] During the training of the base model, the modal learning module, the physical heuristic module, the feature fusion module and the five classifiers are updated based on the loss function L.

[0098] The loss function L is:

[0099] (15);

[0100] (16);

[0101] (17);

[0102] wherein, is the classification loss of the base model based on the fusion features for diagnosis, which specifically adopts the cross-entropy loss of the predicted label and the true label; is the set bias term, which is 0 in the embodiment;

[0103] L JS the alignment loss calculated for equation (2);

[0104] the difference loss between the final shared feature and the pressure-specific feature ;

[0105] the difference loss between the vibration-specific feature and the final shared feature ;

[0106] Specifically, the difference loss between the features can be represented by a cosine distance.

[0107] the signal reconstruction loss of the physics-inspired module;

[0108] the fault-aware loss;

[0109] the ranking loss, a binary number, when , , otherwise, = 0; 1≤i≤4, 1≤j≤4, ; 1≤i≤4, 1≤j≤4; , , and respectively represent the attention weights of the vibration-specific feature , the final shared feature , the pressure-specific feature and the physical feature ; , , and respectively represent the classification loss of the vibration-specific feature , the final shared feature , the pressure-specific feature and the physical feature ;

[0110] λ1, λ2, λ3, λ4, λ5 are all set weight coefficients, all set to 1 in subsequent examples;

[0111] When the basic model is trained, the first classifier, the second classifier, the third classifier and the fourth classifier are deleted from the basic model, thereby obtaining a fault prediction model for fault diagnosis of the external gear gear pump based on modal sharing and physics-inspired.

[0112] Referring to Figure 2 , Figure 3 , when the sample is diagnosed by the fault prediction model, the steps are as follows:

[0113] St1, obtaining a signal sample to be diagnosed {vibration signal, pressure signal}; when the signal to be diagnosed only contains a pressure signal, constructing a signal sample to be diagnosed {0, pressure signal}, i.e. processing the vibration signal to zero;

[0114] St2, inputting the sample to be diagnosed into the fault prediction model; the pressure signal is processed by the physical inspiration module to obtain physical features, and the pressure signal is further processed by the second encoder and the second segmentation unit in the modal learning module to obtain pressure-specific features and shared features;

[0115] When the vibration signal is missing, the shared features output by the second segmentation unit are used as the final shared features output by the modal learning module, and the vibration-specific features output by the modal learning module are set to zero; when the vibration signal is present, the vibration signal is processed by the first encoder and the first segmentation unit in the modal learning module to obtain vibration-specific features and shared features, and the shared features output by the two segmentation units are fused to obtain the final shared features;

[0116] The feature fusion module fuses the physical features, the pressure-specific features, the final shared features and the vibration-specific features to obtain fusion features, and the end classifier predicts the fault category based on the fusion features.

[0117] That is, when the data sample is diagnosed by the fault prediction model, the second encoder extracts the encoding features of , the second segmentation unit divides the encoding features into pressure-specific features and second shared features , and the outputs of the vibration signal, the first encoder and the first segmentation unit are all set to zero; the fusion features are calculated according to formulas (1), (11), (12), (13) and (14); then the end classifier classifies the faults based on the fusion features .

[0118] Specifically, the data sample corresponds to the fusion features obtained in the following formula:

[0119] ;

[0120] ,

[0121] and ;

[0122] ;

[0123] ;

[0124] ;

[0125] The above multi-modal fusion-based external gear pump fault diagnosis method is verified in combination with specific embodiments.

[0126] In this embodiment, an external gear pump test bench is built, mainly including an oil tank, a motor, a gear pump, a vibration sensor, a pressure sensor, a control system, a simulated voltage collector and a simulated current collector. The experimental data consists of pressure signals and vibration signals. In the experiment, a total of 14 fault conditions are set under three motor speeds, as shown in Table 1. S1, S2 and S3 represent motor speed values of 600 rpm, 1050 rpm and 1500 rpm, respectively. Gear pump faults mainly include gear faults, seal ring faults, pump body faults and composite faults.

[0127] Table 1. Fault cases of different speeds and fault types

[0128] ;

[0129] The pressure signal used in this embodiment is a one-dimensional time series signal, and the vibration signal is a three-dimensional signal. All signals are divided into sample segments with a length of 5120. This embodiment uses all collected data to construct a mixed data set containing multi-modal samples and single-modal samples. The data set is divided into a training set, a validation set and a test set in a ratio of 6:2:2. The training set, the validation set and the test set respectively contain 2016, 672 and 672 samples. In the missing modal data set, the vibration signal is set to zero.

[0130] To reduce the influence of interference and unstable operating conditions, data preprocessing is performed on the pressure signal. For the case of unstable speed, a resampling method based on Fourier transform is used. For the problem of unstable pressure, the average value of the initial sample is first calculated as a reference value, and then the difference between the average value of each subsequent sample and the reference value is removed to eliminate data drift.

[0131] To verify the performance of the fault diagnosis model provided by the present application, comparative experiments and ablation experiments are performed in this embodiment.

[0132] In the comparative experiment, the fault diagnosis model provided by the present application (referred to as the present model PIMFNet) is compared with comparative models 1-3, respectively; in the ablation experiment, the fault diagnosis model provided by the present application is compared with the ablation model after eliminating the physical inspiration module.

[0133] The comparative models 1, 2 and 3 respectively adopt the existing multi-modal learning frameworks MMTM, MedFuse and DAFT; the loss function in the training process of the comparative model adopts the cross-entropy loss of the predicted label and the real label.

[0134] The ablation model (referred to as no phy) eliminates the physical inspiration module on the basis of the model of the application, the feature fusion module fuses the final shared features and the obtained modal specific features to obtain fusion features, and then inputs the end classifier for classification; the loss function of the ablation model is:

[0135] ;

[0136] That is, the loss function of the ablation model deletes the signal reconstruction loss of the physical inspiration module relative to the model of the application.

[0137] The model of the application, the comparative models 1-3 and the ablation model are all trained on the training set and the validation set provided in the embodiment, and then the performance is tested on the test set.

[0138] The model training process is: first, extract training samples on the training set and input them into the model for parameter updating, then extract validation samples from the validation set and input them into the model to calculate the loss function, and update the model according to the loss function; the above process is repeated until the model converges.

[0139] In the embodiment, the hyperparameters , , , and are set to {1, 1, 1, 1, 1}; the learning rate is set to 0.0001; the maximum number of training rounds is set to 100. To prevent overfitting of the model, an early stopping mechanism is implemented. The number of low-frequency components I and the number of high-frequency components J in the physical inspiration module are set to 12 and 18 respectively, because the rotational and gear meshing frequency components decay rapidly after passing through 12 and 18 frequency lines respectively. The Adam optimizer is used in the training process.

[0140] In the experiment, S1, S2 and S3 are run under three operating conditions, the training set and the validation set both contain multi-modal samples and single-modal samples, and two data sets are used in the test: one contains multi-modal samples and single-modal samples, and is recorded as the full-modal test set; the other contains only single-modal samples (i.e. pressure signals), and is recorded as the missing modal data set. The fault diagnosis results are shown in Table 2.

[0141] Table 2 Performance comparison under full-modal and missing modal conditions

[0142] ;

[0143] It can be seen that under two conditions, the model PIMFNet of the application all realizes the highest diagnostic accuracy: the average accuracy reaches 0.998 in the full modal mode, and the average accuracy is 0.976 under three rotating speeds in the missing modal mode. The model of the application surpasses other methods in the accuracy index, and shows strong generalization ability under different working conditions.

[0144] It is worth noting that when only the pressure signal exists, PIMFNet achieves a diagnostic performance comparable to the full modal scenario, which proves that it can effectively retain key information from missing modal data.

[0145] The ablation model no phy without introducing physical information ranks second with an average accuracy of 0.962 in the full modal mode and 0.899 in the missing modal mode. The addition of the physical inspiration module enables PIMFNet to extract features with clear physical meaning, understand the internal physical mechanism, and significantly improve the accuracy in the missing modal scenario compared to the ablation no phy.

[0146] MedFuse and DAFT are currently representative multi-modal learning methods. From the experimental results, the precision is high in the full modal mode, but the performance sharply decreases in the missing modal mode, and the effect differs greatly under different rotating speeds; further proving the performance stability of the model of the application.

[0147] As shown in FIGS. 4(a), 4(b), 4(c), 4(d), 4(e), 5(a), 5(b), 5(c), 5(d) and 5(e), to visually display the fault diagnosis effect, the embodiment further visualizes the end features (such as the fusion authentication in the model of the application) extracted by each model under the S3 rotating speed by t-SNE. It can be seen that in the full modal mode, the features extracted by MMTM, MedFuse and DAFT are obviously confused, and this problem is further worsened in the missing modal scenario. In contrast, the model PIMFNet of the application performs best, and its fault features in the full modal mode can clearly distinguish all 14 fault types. PIMFNet realizes a performance comparable to the complete modal mode in the missing modal scenario, and clearly identifies 12 kinds of faults and only causes slight confusion. The ablation model diagnoses nearly all 14 faults in the complete modal mode, but can only clearly identify 7 faults in the missing modal mode.

[0148] In the embodiment, the original pressure signal collected under the speed S3 working condition is also plotted against the reconstructed model of the physical inspiration module. As shown in FIG. 6, the original signal and the reconstructed signal are almost completely coincident, which indicates that the physical inspiration module can effectively understand the internal mechanism of the periodic motion of the gear pump components. Figure 6

[0149] In the embodiment, the reconstructed loss is also removed in the missing modal mode, and the results are shown in FIGS. 7(a) and 7(b). It can be seen that the accuracy of PIMFNet is significantly reduced, which proves that the reconstructed loss is essential for the model to understand the internal mechanism of the gear pump components. ​, and the case of further removing the encoder in the physical hinting module on the basis of eliminating The comparison results are shown in Table 3.

[0150] Table 3 Comparison of test accuracy of three cases

[0151]

[0152] Obviously, compared with the model of the application, the diagnostic accuracy in the missing modal test is overall decreased in the case without the reconstruction loss , which shows that the physical information loss forces the module to learn the fault coefficient that follows the physical rules, thereby improving the classification performance. In addition, after further removing the encoder in the physical hinting module on the basis of eliminating , the accuracy in the missing modal mode is decreased more significantly, which shows that the encoder is essential to the feature learning module, and the output of the encoder supplements the features extracted by the physical information masking module.

[0153] Of course, for those skilled in the art, the application is not limited to the details of the above exemplary embodiments, but also includes the same or similar structures that can be implemented in other specific forms without departing from the spirit or essential characteristics of the application. Therefore, the embodiments should be regarded as exemplary and non-limiting in any respect, and the scope of the application is defined by the appended claims rather than the above description, and therefore all changes falling within the meaning and scope of the equivalent elements of the claims are intended to be included in the application. Any reference signs in the claims should not be regarded as limiting the claims involved.

[0154] In addition, it should be understood that although the present specification is described in terms of embodiments, not every embodiment contains only one independent technical solution, and the description manner of the specification is only for the sake of clarity, and those skilled in the art should regard the specification as a whole, and the technical solutions in each embodiment can also be properly combined to form other embodiments that those skilled in the art can understand.

[0155] The technologies, shapes, and structural parts not described in detail in the application are well-known technologies.​

Claims

1. A multi-modal fusion-based fault diagnosis method for a missing mode external gear pump, characterized in that, The base model is first trained on a labeled mixed data set containing multi-modal and single-modal, the multi-modal including pressure signals and vibration signals, and the single-modal being pressure signals; the base model includes a modal learning module, a physical inspiration module, a feature fusion module, and five classifiers; The modal learning module encodes the modal signals and divides the encoded features into modal-specific features and shared features of the two modalities, and the modal learning module constructs final shared features based on the obtained shared features; the physical inspiration module fuses the encoded feature coefficients obtained by encoding the pressure signals and the frequency domain feature coefficients obtained by frequency domain extraction to obtain physical features; the feature fusion module fuses the final shared features, the modal-specific features, and the physical features to obtain fused features; the five classifiers respectively predict the fault categories of the samples based on the vibration-specific features, the shared features, the pressure-specific features, the physical features, and the fused features; The base model is trained in combination with the classification losses of the five classifiers, and after training, the modal learning module, the physical inspiration module, the feature fusion module, and the classifier based on the fused features to predict the fault category are extracted to form a fault diagnosis model, which is used to diagnose the fault categories of multi-modal samples and single-modal samples; For multi-modal samples, the modal learning module encodes and cuts the pressure signals and vibration signals respectively, then fuses the shared features obtained by cutting the two encoded features to obtain final shared features and outputs, and the corresponding classifier and feature fusion module process based on the final shared features; for single-modal samples, the final shared features are the shared features corresponding to the pressure signals; The way to obtain the final shared features of the multi-modal samples is: first, use an activation function to process the shared features corresponding to the two signals respectively and take the average, then perform the reverse operation of the activation function on the average result to obtain the final shared features.

2. The multi-modal fusion based missing modality external gear pump fault diagnosis method according to claim 1, wherein, The feature fusion module employs a scaled dot product attention mechanism to process the final shared features. Physical characteristics By processing modality-specific features, fused features are obtained. Modal-specific characteristics include pressure-specific characteristics. and vibration characteristics .

3. The multi-modal fusion-based fault diagnosis method for the externally meshing gear pump with missing modalities according to claim 2, wherein: wherein, is an intermediate term, denotes a projection matrix in the scaled dot-product attention mechanism; is the attention weight vector corresponding to the failure type s S is the total number of failure categories.

4. The multi-modal fusion-based fault diagnosis method for the externally meshing gear pump with missing modalities according to claim 3, wherein: wherein, is a query vector, is a target vector of a sample under fault type s, is a modality-dependent mask vector, d is the dimension of the vector; is a vibration signal, m is the sample serial number; wherein, is a linear projection matrix for computing the query vector ​ 5. The multi-modal fusion based missing modality external gear pump fault diagnosis method of claim 1, wherein, The loss function L in the model training process is denoted as: wherein, a classification loss for the base model to diagnose based on the fused features, a modality alignment loss for shared features from the vibration signals and shared features from the pressure signals; a discrepancy loss between the final shared features and the pressure-specific features, a discrepancy loss between the vibration-specific features and the final shared features; a signal reconstruction loss for the physics-inspired module to reconstruct the pressure signals based on the physics features; a failure-aware loss to measure the capability of the final shared features, the pressure-specific features, the physics features, and the vibration-specific features to predict the failure labels; a ranking loss to make the model pay more attention to the features with lower predicted loss; λ1, λ2, λ3, λ4, and λ5 are all set weight coefficients.

6. The multi-modal fusion-based fault diagnosis method for the externally meshing gear pump with missing modalities according to claim 5, wherein: is a binary number, when , , otherwise, = 0; 1≤i≤4, 1≤j≤4, , , and represent the attention weights of the vibration-specific feature, the final shared feature, the pressure-specific feature, and the physical feature, respectively; , , and represent the classification losses corresponding to the vibration-specific feature, the final shared feature, the pressure-specific feature, and the physical feature, respectively; is a set bias term.

7. The multi-modal fusion based missing modality external gear pump fault diagnosis method of claim 6, wherein, The sum of classification losses based on the final shared features, stress-specific features, physical features, and vibration-specific features under each failure category is adopted.

8. The multi-modal fusion-based fault diagnosis method for the externally meshing gear pump with missing modalities according to claim 6, wherein: wherein, is a shared feature from the vibration signal, is a shared feature from the pressure signal, Z is a shared feature and a shared feature of the mean; KL is a divergence function.

9. A multi-modal fusion-based fault diagnosis system for a missing mode external gear pump, the system comprising: The device comprises a memory and a processor, the memory stores a computer program, the processor is connected to the memory, and the processor is used to execute the computer program to realize the multi-modal fusion-based fault diagnosis method for the externally meshing gear pump with missing modalities according to any one of claims 1-8.

Images