A rotating machinery fault diagnosis method and system based on cross-modal time-frequency perception

Abstract

The present application relates to the technical field of mechanical fault diagnosis, in particular to a rotating machinery fault diagnosis method and system based on cross-modal time-frequency perception, which designs time domain, frequency domain and time-frequency domain perception embedding branches, can adaptively process cross-modal time domain, frequency domain and time-frequency domain signals from different devices, realize effective fusion of multi-modal signals, and improve the precision and robustness of fault diagnosis. For the private characteristic attributes of time domain and frequency domain and the public characteristic attributes of time-frequency domain, the present application designs corresponding Transformer branches for the embedding features of time domain, frequency domain and time-frequency domain, and uses a time-frequency convergence multi-head cross attention module to represent local features of time domain and frequency domain, aggregate global time-frequency domain feature representation, fully excavate the commonness and differences between time domain and frequency domain, and further improve the accuracy and robustness of fault diagnosis.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of mechanical fault diagnosis, in particular to a rotating machinery fault diagnosis method and system based on cross-modal time-frequency perception. BACKGROUND

[0002] Rotating machinery is widely used in industrial production, such as electric motors, wind turbines, robots, gearboxes and other equipment. These devices are prone to failure during operation, resulting in production stagnation, equipment damage, increased maintenance costs, and even more serious safety accidents. Therefore, the research and application of early fault diagnosis technology is of great significance to ensure the normal operation of equipment, improve production efficiency, and reduce maintenance costs.

[0003] At present, the basic model based on sequence signal data has achieved good results in the fault diagnosis task of rotating machinery. However, the existing cross-modal fault diagnosis method still has the following problems:

[0004] (1) Unable to adaptively perceive heterogeneous multi-modal signals of rotating machinery: signal embedding is an important link in the fault diagnosis task, which has an important influence on the diagnosis results of the model. Existing embedding methods include time domain embedding methods and frequency domain embedding methods. These methods only perceive signals from time domain or frequency domain, ignoring the importance of joint optimization of time domain and frequency domain signals, and cannot effectively adapt to heterogeneous multi-modal signals from different acquisition devices, resulting in that the diversified signals in the rotating machinery system cannot be fully processed and utilized, thereby affecting the diagnosis accuracy.

[0005] (2) Poor universality of time-frequency perception embedding method: existing time-frequency perception embedding methods are usually only for a single diagnosis model or a specific application scenario, lacking universality, and each time the model is replaced, a separate embedding module needs to be configured, reducing the application value and promotion of the embedding method.

[0006] (3) Insufficient feature fusion of time-frequency domain information: some existing fault diagnosis methods based on cross-modal learning, such as using sensor fusion technology, joint learning of image and time series data, etc., need to set up an encoder for each modality to extract features, which not only increases the calculation amount, but also reduces its flexibility and universality. In addition, some methods design a time-frequency joint Transformer fusion method, but this method inputs the joint time domain and frequency domain data to a Transformer branch, without considering the private and public feature attributes of time-frequency domain signals, which limits the accuracy and robustness of the model when processing unknown domain fault signals, reducing the generalization performance of the model.

[0007] In summary, in the existing rotating machinery fault diagnosis method, the embedding method cannot effectively solve the fusion problem between time domain and frequency domain information, and cannot complete cross-modal adaptive perception and has poor flexibility. Moreover, the existing time-frequency joint Transformer model fails to deeply consider the private feature attributes and public feature attributes of time-frequency domain signals, resulting in poor generalization performance of the model. SUMMARY

[0008] To this end, the technical problem to be solved by the present application is to overcome the problem that the prior art is difficult to adaptively perceive multi-modal time-frequency domain information and difficult to fuse time-frequency domain features, resulting in low fault diagnosis precision and poor generalization performance.

[0009] To solve the above technical problems, the present application provides a rotating machinery fault diagnosis method based on cross-modal time-frequency perception, comprising:

[0010] inputting the multi-modal time domain signal into a time domain perception embedding branch, sequentially passing through a time domain signal perception module and a first nonlinear activation mapping module to obtain a time domain activation mapping feature; and passing the time domain activation mapping feature through a linear mapping layer to obtain a time domain embedding feature;

[0011] inputting the multi-modal frequency domain signal into a frequency domain perception embedding branch, sequentially passing through a frequency domain signal perception module and a second nonlinear activation mapping module to obtain a frequency domain activation mapping feature; and passing the frequency domain activation mapping feature through a linear mapping layer to obtain a frequency domain embedding feature;

[0012] inputting the time domain activation mapping feature and the frequency domain activation mapping feature into a time-frequency domain perception embedding branch to obtain a time-frequency domain embedding feature;

[0013] inputting the time domain embedding feature into a time domain Transformer private branch containing a time domain multi-head self-attention module to obtain a time domain prediction classification label;

[0014] inputting the frequency domain embedding feature into a frequency domain Transformer private branch containing a frequency domain multi-head self-attention module to obtain a frequency domain prediction classification label;

[0015] inputting the time-frequency domain embedding feature into a time-frequency domain Transformer public branch containing a time-frequency convergence multi-head cross-attention module to obtain a time-frequency domain prediction classification label.

[0016] Preferably, the time domain signal perception module and the frequency domain signal perception module each include a plurality of perception kernels; the length and the number of channels of the perception kernels are respectively less than the length and the number of channels of the input signal, and each perception kernel has a different channel starting index value and a sequence starting index value.

[0017] Preferably, the data processing process of the time domain signal perception module and the frequency domain signal perception module each includes:

[0018] The plurality of perception cores of the current signal perception module act on different sub-regions of the input signal of the current signal perception module to obtain a perception feature corresponding to each perception core;

[0019] Based on the perception features corresponding to all the perception cores, a total perception feature is obtained;

[0020] The total perception feature is subjected to a flattening operation to obtain an output feature of the current signal perception module.

[0021] Preferably, the first nonlinear activation mapping module and the second nonlinear activation mapping module each comprise a linear mapping layer and a SiLU activation function connected in sequence.

[0022] Preferably, the time domain activation mapping feature and the frequency domain activation mapping feature are input into a time-frequency domain perception embedding branch to obtain a time-frequency domain embedding feature, and the process comprises:

[0023] The time domain activation mapping feature and the frequency domain activation mapping feature are spliced to obtain a time-frequency fusion feature;

[0024] The time-frequency fusion feature is sequentially subjected to a linear mapping layer, a SiLU activation function and a linear mapping layer to obtain the time-frequency domain embedding feature.

[0025] Preferably, the time domain multi-head self-attention module and the frequency domain multi-head self-attention module each comprise a self-attention module, a residual connection and a layer normalization operation, a feedforward network and a residual connection and a layer normalization operation connected in sequence.

[0026] Preferably, the time-frequency convergence multi-head cross-attention module comprises a cross-attention module, a convergence fusion layer, a linear mapping layer, a residual connection and a layer normalization operation, a feedforward network layer and a residual connection and a layer normalization operation connected in sequence; the cross-attention module comprises a time domain-time frequency domain cross-attention module, a time-frequency domain self-attention module and a frequency domain-time frequency domain cross-attention module connected side by side.

[0027] Preferably, the data processing process of the time-frequency convergence multi-head cross-attention module comprises:

[0028] The time-frequency domain embedding feature is respectively subjected to the time domain-time frequency domain cross-attention module, the time-frequency domain self-attention module and the frequency domain-time frequency domain cross-attention module of the cross-attention module to obtain a time domain-time frequency domain cross-attention feature, a time-frequency domain self-attention feature and a frequency domain-time frequency domain cross-attention feature;

[0029] The time domain-time frequency domain cross-attention feature, the time-frequency domain self-attention feature and the frequency domain-time frequency domain cross-attention feature are input into the convergence fusion layer for splicing to obtain a fusion attention feature;

[0030] The fusion attention feature is subjected to a linear mapping layer, and the output feature thereof is subjected to a residual connection and a layer normalization operation with the time-frequency domain embedding feature to obtain an input feature of a feedforward network layer;

[0031] The input feature of the feedforward network layer is subjected to the feedforward network layer to obtain an output feature of the feedforward network layer; and the input feature of the feedforward network layer and the output feature of the feedforward network layer are subjected to a residual connection and a layer normalization operation to obtain an output feature of the time-frequency convergence multi-head cross attention module.

[0032] Preferably, the data processing process of the cross attention module comprises:

[0033] The time-frequency domain embedding feature is subjected to linear mapping to obtain a time-frequency query, a time-frequency key and a time-frequency value;

[0034] A time domain query and a time domain key are obtained; a frequency domain query and a frequency domain key are obtained; the time domain query and the time domain key are obtained by subjecting the time domain embedding feature to linear mapping; and the frequency domain query and the frequency domain key are obtained by subjecting the frequency domain embedding feature to linear mapping;

[0035] The time domain query, the frequency domain key and the time-frequency value are subjected to a cross attention module of the time domain-time frequency domain to obtain a cross attention feature of the time domain-time frequency domain;

[0036] The time-frequency query, the time-frequency key and the time-frequency value are subjected to a self-attention module of the time-frequency domain to obtain a self-attention feature of the time-frequency domain;

[0037] The frequency domain query, the time domain key and the time-frequency value are subjected to a cross attention module of the frequency domain-time frequency domain to obtain a cross attention feature of the frequency domain-time frequency domain.

[0038] The application further provides a rotating machinery fault diagnosis system based on cross-modal time-frequency perception, comprising:

[0039] A time domain embedding module is configured to input a multi-modal time domain signal into a time domain perception embedding branch, sequentially pass the time domain signal through a time domain signal perception module and a first nonlinear activation mapping module to obtain a time domain activation mapping feature; and subject the time domain activation mapping feature to a linear mapping layer to obtain a time domain embedding feature;

[0040] A frequency domain embedding module is configured to input a multi-modal frequency domain signal into a frequency domain perception embedding branch, sequentially pass the frequency domain signal through a frequency domain signal perception module and a second nonlinear activation mapping module to obtain a frequency domain activation mapping feature; and subject the frequency domain activation mapping feature to a linear mapping layer to obtain a frequency domain embedding feature;

[0041] A time-frequency domain embedding module is configured to input the time domain activation mapping feature and the frequency domain activation mapping feature into a time-frequency domain perception embedding branch to obtain a time-frequency domain embedding feature;

[0042] The time domain feature extraction module is configured to input time domain embedding features into a time domain Transformer private branch containing a time domain multi-head self-attention module to obtain time domain prediction classification labels.

[0043] The frequency domain feature extraction module is configured to input frequency domain embedding features into a frequency domain Transformer private branch containing a frequency domain multi-head self-attention module to obtain frequency domain prediction classification labels.

[0044] The time-frequency domain feature extraction module is configured to input time-frequency domain embedding features into a time-frequency domain Transformer public branch containing a time-frequency convergence multi-head cross-attention module to obtain time-frequency domain prediction classification labels.

[0045] The above technical solutions of the present application have the following beneficial effects compared with the prior art:

[0046] The cross-modal time-frequency perception-based rotating machinery fault diagnosis method provided by the present application designs time domain, frequency domain and time-frequency domain perception embedding branches, can adaptively process cross-modal time domain, frequency domain and time-frequency domain signals from different devices, realize effective fusion of multi-modal signals, and generate unified representation time domain, frequency domain and time-frequency domain embedding features. This cross-modal perception embedding branch breaks through the limitation of existing embedding methods that can only process specific signal modes, has high universality, can be widely applied to different fault diagnosis models, and improves the accuracy and robustness of fault diagnosis. Further, the present application designs corresponding Transformer branches for the embedding features of the time domain, frequency domain and time-frequency domain according to the private feature attributes of the time domain and frequency domain and the public feature attributes of the time-frequency domain, and uses a time-frequency convergence multi-head cross-attention module to aggregate global time-frequency domain feature representations based on local feature representations of the time domain and the frequency domain. The multiple Transformer branches designed by the present application take into account the private feature attributes and public feature attributes of time-frequency information, can fully exploit the commonalities and differences between the time domain and the frequency domain, and improve the accuracy and robustness of fault diagnosis.

[0047] The present application can cope with signal changes of different devices and different fault types by deeply fusing time domain, frequency domain and time-frequency domain signals, improve the adaptability and universality of the diagnosis model. Compared with traditional methods, the model of the present application not only can be effectively applied to existing devices, but also can be quickly adapted to new devices or new signal modes, and has strong expansion ability. BRIEF DESCRIPTION OF DRAWINGS

[0048] In order to make the content of the present application more easily understood, the present application will be further described in detail below according to specific embodiments of the present application and in conjunction with the drawings, in which:

[0049] Figure 1is a flow chart of a rotating machinery fault diagnosis method based on cross-modal time-frequency perception of the present application;

[0050] Figure 2 is a structural diagram of a cross-modal time-frequency perception embedding module;

[0051] Figure 3 is a structural diagram of a time-frequency convergence fusion Transformer module;

[0052] Figure 4 is a schematic diagram of a SCARA experimental platform used in an embodiment of the present application, wherein Figure 4 (a) in is a robot experimental platform, Figure 4 (b) in is a ball screw jamming test experiment, Figure 4 (c) in is a robot ball shedding fault sample;

[0053] Figure 5 is a schematic diagram of a PU dataset experimental platform used in an embodiment of the present application, wherein Figure 5 (a) in is a bearing vibration signal, Figure 5 (b) in is a motor A-phase current signal, Figure 5 (c) in is a motor B-phase current signal;

[0054] Figure 6 is a comparison diagram of fault prediction results of different models;

[0055] Figure 7 is a schematic diagram of ablation model experiment results;

[0056] The description of the figures in the specification is as follows: 1, J3 axis driving motor; 2, control panel; 3, computer; 4, J3 axis; 5, screw; 6, load; 7, screw nut; 8, spline nut; 9, push-pull force gauge; 10, ball shedding; 11, motor; 12, torque strategy shaft; 13, bearing test module; 14, flywheel; 15, load motor. DETAILED DESCRIPTION

[0057] The present application will be further described below in conjunction with the drawings and specific embodiments, so that those skilled in the art can better understand the present application and implement it, but the embodiments are not limiting to the present application.

[0058] Referring to Figure 1 , the present application provides a rotating machinery fault diagnosis method based on cross-modal time-frequency perception, comprising:

[0059] S1: input the multi-modal time domain signal into the time domain perception embedding branch, sequentially pass through the time domain signal perception module and the first nonlinear activation mapping module to obtain the time domain activation mapping feature; pass the time domain activation mapping feature through the linear mapping layer to obtain the time domain embedding feature;

[0060] S2: input the multi-modal frequency domain signal into the frequency domain perception embedding branch, sequentially pass through the frequency domain signal perception module and the second nonlinear activation mapping module to obtain the frequency domain activation mapping feature; pass the frequency domain activation mapping feature through the linear mapping layer to obtain the frequency domain embedding feature;

[0061] S3: input the time domain activation mapping feature and the frequency domain activation mapping feature into the time-frequency domain perception embedding branch to obtain the time-frequency domain embedding feature;

[0062] S4: input the time domain embedding feature into the time domain Transformer private branch containing the time domain multi-head self-attention module to obtain the time domain predicted classification label;

[0063] S5: input the frequency domain embedding feature into the frequency domain Transformer private branch containing the frequency domain multi-head self-attention module to obtain the frequency domain predicted classification label;

[0064] S6: input the time-frequency domain embedding feature into the time-frequency domain Transformer public branch containing the time-frequency convergence multi-head cross-attention module to obtain the time-frequency domain predicted classification label.

[0065] Wherein, the time domain perception embedding branch, the frequency domain perception embedding branch and the time-frequency domain perception embedding branch constitute the cross-modal time-frequency perception embedding module. The time domain Transformer private branch, the frequency domain Transformer private branch and the time-frequency domain Transformer public branch constitute the time-frequency convergence fusion Transformer module. The cross-modal time-frequency perception embedding module and the time-frequency convergence fusion Transformer module constitute the rotating machinery fault detection model. The structure and processing process of each module are described in detail below.

[0066] Referring to Figure 2 As shown in the figure, the cross-modal time-frequency perception embedding module first performs fast Fourier transform on the multi-modal time domain signal to obtain the multi-modal frequency domain signal. The time domain perception embedding branch and the frequency domain perception embedding branch perceive the multi-modal time domain and frequency domain signals into time domain and frequency domain perception features respectively, then pass these features through nonlinear activation mapping to obtain activation mapping features, and then pass them through the linear mapping layer to obtain the embedding features represented uniformly. Finally, the time domain activation mapping feature and the frequency domain activation mapping feature are spliced into the time-frequency fusion feature, and the time-frequency fusion feature is nonlinearly activated and mapped into the time-frequency domain embedding feature represented uniformly.

[0067] The multi-modal signal includes a speed feedback signal, a current signal, an encoder signal, a vibration signal and the like of the rotating machinery equipment.

[0068] In the time domain perception embedding branch and the frequency domain perception embedding branch, the time domain signal perception module and the frequency domain signal perception module each include a plurality of perception kernels, the number, length and channel number of the perception kernels are respectively set to 、 and , wherein is also the number of modalities for cross-modality perception of the perception kernel. Assuming that the number of modalities (channels) and the length of the rotating machinery equipment signal are and , the length and the channel number of the perception kernel are respectively less than the length and the channel number of the input signal, and each perception kernel has a different channel starting index value and a sequence starting index value. Specifically, in the transverse sequence dimension, different perception kernels have different sequence starting index values. In the longitudinal channel dimension, different perception kernels also have different channel starting index values. Therefore, the perception kernel can perceive multi-modal information from the input signal in all directions and at multiple angles.

[0069] For example, assuming that the rotating machinery equipment has a signal of 10 modalities and the signal length is 1024. The channel number c of the perception kernel is set to be less than 10, and can generally be set to 6-8. Assuming that the channel number of the perception kernel is 6 and the length of the perception kernel is 128, the channel starting index value of one of the perception kernels is 2 (which is less than or equal to [10-6+1=5]), so the perception kernel can perceive the signal of the 2nd to 7th channel of the equipment. Similarly, in the transverse dimension, assuming that the sequence starting index value of one of the perception kernels is 10 (which is less than [1024-128+1=897]), so the perception kernel can perceive the features of the 10th to 137th data points of the equipment signal. Therefore, this perception kernel can perceive the sub-regions of the 2nd to 7th channels (longitudinally) and the 10th to 137th data points (transversely) of the input signal to obtain the corresponding perception features.

[0070] It should be noted that, in order to enable the perception kernel to extract signals of different channels and different positions, the present application parallelly sets a plurality of perception kernels, different perception kernels have different channel starting index values and sequence starting index values. Moreover, setting the channel number of the perception kernel to be less than the channel number of the input signal can reduce the amount of calculation, for example, the channel number of the input signal is 10 and the channel number of the perception kernel is 6, since different perception kernels have different channel starting index values, 10 modalities of information can also be perceived by setting a plurality of perception kernels, which can reduce the amount of calculation and avoid loss of signal features.

[0071] Specifically, the data processing process of the time domain signal perception module and the frequency domain signal perception module each includes:

[0072] The plurality of perception cores of the current signal perception module act on different sub-regions of the input signal of the current signal perception module to obtain a perception feature corresponding to each perception core;

[0073] Based on the perception features corresponding to all the perception cores, a total perception feature is obtained;

[0074] The total perception feature is subjected to a flattening operation to obtain an output feature of the current signal perception module.

[0075] Preferably, the first nonlinear activation mapping module and the second nonlinear activation mapping module each include a linear mapping layer and a SiLU activation function connected in sequence.

[0076] Suppose the multi-modal time-domain signal is The processing process of the time-domain perception embedding branch in S1 is as follows:

[0077] S11: A plurality of time-domain perception cores of the time-domain signal perception module act on different sub-regions of the multi-modal time-domain signal to obtain a perception feature corresponding to each time-domain perception core. Suppose the sequence start index value of the i-th time-domain perception core is , the channel start index value is , then the time-domain perception feature perceived by the i-th time-domain perception core can be represented as:

[0078] ;

[0079] Wherein, is the i-th time-domain perception feature, is a perception operation, is the multi-modal time-domain signal. It should be noted that the intervals [j: j+c) and [k: k+l) are both closed at the front and open at the back.

[0080] S12: Based on all the time-domain perception features, a time-domain total perception feature is obtained.

[0081] The time-domain total perception feature is represented as:

[0082] .

[0083] S13: The time-domain total perception feature is subjected to a flattening operation to obtain an output feature of the time-domain signal perception module:

[0084] ;

[0085] Wherein, is a flattening operation. ​​

[0086] S14: output features of the time domain signal perception module are mapped through a first nonlinear activation mapping module to obtain time domain activation mapping features , which is expressed by the formula:

[0087] ;

[0088] wherein, is the time domain activation mapping feature, is the SiLU activation function, is a first linear mapping layer in the time domain.

[0089] S15: the time domain activation mapping features are mapped through a linear mapping layer to obtain time domain embedding features in a unified representation , which is expressed by the formula:

[0090] ;

[0091] wherein, is the time domain embedding feature, is a second linear mapping layer in the time domain, is an output dimension of the time domain embedding feature.

[0092] The number of perception kernels and the output dimension can be flexibly set according to the data dimension of the model that can be connected, so as to ensure that the module can be applied to different models.

[0093] In S2, the processing process of the frequency domain perception embedding branch is the same as the processing principle of the time domain perception embedding branch, and the specific process is as follows:

[0094] S21: fast Fourier transform is performed on the multi-modal time domain signal to obtain a multi-modal frequency domain signal.

[0095] The multi-modal frequency domain signal can be calculated by the following formula:

[0096] ;

[0097] ;

[0098] wherein, is the output signal of the fast Fourier transform, is the multi-modal frequency domain signal, is a complex set, is a real number set.

[0099] S22: a plurality of frequency domain perception kernels of the frequency domain signal perception module act on different sub-regions of the multi-modal frequency domain signal to obtain perception features corresponding to each frequency domain perception kernel; assuming that the first frequency domain perception kernel is​​ The sequence starting index value of the frequency domain perception kernel is The channel starting index value is The frequency domain perception feature obtained by the i-th frequency domain perception kernel can be represented as:

[0100] ;

[0101] Wherein, is the i-th frequency domain perception feature.

[0102] S23: Based on all frequency domain perception features, a total frequency domain perception feature is obtained.

[0103] The total frequency domain perception feature is represented as:

[0104] .

[0105] S24: The total frequency domain perception feature is flattened to obtain the output feature of the frequency domain signal perception module :

[0106] .

[0107] S25: The output feature of the frequency domain signal perception module is subjected to a second nonlinear activation mapping module to obtain a time domain activation mapping feature , which is represented by the formula:

[0108] ;

[0109] Wherein, is the frequency domain activation mapping feature, is the SiLU activation function, is the frequency domain first linear mapping layer.

[0110] S26: The frequency domain activation mapping feature is subjected to a linear mapping layer to obtain a unified frequency domain embedding feature , which is represented by the formula:

[0111] ;

[0112] Wherein, is the frequency domain embedding feature, is the frequency domain second linear mapping layer.

[0113] The time-frequency perception embedding branch provided by the application can utilize the complementarity of time domain and frequency domain information to provide more comprehensive and accurate fault diagnosis basis. First, the time-frequency perception embedding branch will​ and frequency domain activation mapping features The time-frequency fusion features are spliced into time-frequency domain fusion features, and then the time-frequency fusion features are subjected to nonlinear activation mapping to obtain time-frequency domain embedding features in a unified representation.

[0114] Specifically, in S3, the time domain activation mapping features and the frequency domain activation mapping features are input into a time-frequency domain perception embedding branch to obtain time-frequency domain embedding features, and the process includes:

[0115] S31: The time domain activation mapping features and the frequency domain activation mapping features are spliced to obtain time-frequency fusion features, and the formula is represented as:

[0116] ;

[0117] wherein, is a feature splicing function, is the time-frequency fusion features.

[0118] S32: The time-frequency fusion features are sequentially subjected to a linear mapping layer, a SiLU activation function and a linear mapping layer to obtain time-frequency domain embedding features, and the formula is represented as:

[0119] ;

[0120] wherein, is the time-frequency domain embedding features, and are a first time-frequency domain linear mapping layer and a second time-frequency domain linear mapping layer respectively.

[0121] The time-frequency domain perception embedding branch generates time-frequency embedding features in a unified representation by processing the time domain and frequency domain features, and further enhances the comprehensive perception ability of the model to the time domain, frequency domain and time-frequency domain information.

[0122] The advantages of the cross-modal time-frequency perception embedding module designed in the application include that the number, length and cross-modal channel number of the perception kernel can be freely set, and after linear and activation conversion, it can be seamlessly connected to different fault diagnosis sequence models, and has the function of plug and play; the perception kernel has no weight parameter, which can reduce the parameter amount of the model.

[0123] Traditional fault diagnosis methods may be limited to time domain or frequency domain features, and cannot comprehensively utilize time domain, frequency domain and fusion features of the two, and the application designs a time-frequency convergence fusion Transformer module. Figure 3As shown, the time-frequency convergence fusion Transformer module aims to process and fuse the time domain, frequency domain and time-frequency domain embedding features through three branches: the time domain Transformer private branch, the frequency domain Transformer private branch and the time-frequency domain Transformer public branch, which are independent in function but can be optimized cooperatively, thereby improving the performance of the overall model. Among them, the time domain Transformer private branch is used to extract the time sequence dynamic information; the frequency domain Transformer private branch is used to model the spectral pattern and amplitude period information; and the time-frequency domain Transformer public branch adopts the cross attention mechanism to fuse the time / frequency key features.

[0124] The time domain Transformer private branch includes a time domain multi-head self-attention module and a time domain classification head. The frequency domain Transformer private branch includes a frequency domain multi-head self-attention module and a frequency domain classification head. The time-frequency domain Transformer public branch includes a time-frequency convergence multi-head cross-attention module and a time-frequency domain classification head. Among them, the time domain and frequency domain multi-head self-attention modules are respectively used to mine the long-term dependencies in the time domain and the frequency domain, and the classification head is used to perform the fault classification task.

[0125] The time domain multi-head self-attention module and the frequency domain multi-head self-attention module each include a self-attention module, a residual connection and layer normalization operation, a feedforward network and a residual connection and layer normalization operation connected in sequence.

[0126] In S4, the processing process of the time domain Transformer private branch is as follows:

[0127] S41: input the unified representation of the time domain embedding features to the time domain multi-head self-attention module, calculate the time domain query , the time domain key and the time domain value , and the calculation formula is:

[0128] ;

[0129] ;

[0130] ;

[0131] Among them, , and are weight parameters.

[0132] S42: input the time domain query , the time domain key and the time domain value Input the temporal self-attention module and calculate the temporal self-attention features using the following formula:

[0133] ;

[0134] in, For temporal self-attention features, The Softmax activation function is used. This is a scaling factor used to prevent gradient explosion caused by excessively large inner products.

[0135] S43: Incorporating temporal self-attention features With temporal embedding features The residual features are added together and then normalized at each layer before being input into the feedforward network. The input and output features of the feedforward network are added together and normalized at each layer to obtain the output features of the temporal multi-head self-attention module.

[0136] S44: Input the output features of the temporal multi-head self-attention module into the temporal classification head to perform fault classification and obtain temporal predicted classification labels.

[0137] Temporal domain predicted classification labels are calculated using the following formula:

[0138] ;

[0139] in, Predict classification labels for the time domain. For time-domain multi-head self-attention modules, For time-domain classification headers.

[0140] In S5, the processing principle of the frequency domain Transformer private branch is the same as that of the time domain Transformer private branch, and the specific process is as follows:

[0141] S51: Embedding features in the frequency domain with a unified representation Input a time-domain multi-head self-attention module and calculate a frequency-domain query. Frequency domain key and frequency domain value The calculation formula is:

[0142] ;

[0143] ;

[0144] .

[0145] S52: Frequency domain query Frequency domain key and frequency domain value The input frequency domain self-attention module calculates a frequency domain self-attention feature , and the calculation formula is

[0146] .

[0147] S53: The frequency domain self-attention feature is added through a residual connection and then subjected to a layer normalization operation, and the input feature of the feedforward network is added through a residual connection and then subjected to a layer normalization operation, to obtain an output feature of the frequency domain multi-head self-attention module.

[0148] S54: The output feature of the frequency domain multi-head self-attention module is input into a frequency domain classification head to perform fault classification, to obtain a frequency domain prediction classification label.

[0149] The frequency domain prediction classification label is calculated by the following formula:

[0150] ;

[0151] wherein, is the frequency domain prediction classification label, is the frequency domain multi-head self-attention module, is the frequency domain classification head.

[0152] The public branch of the time-frequency domain Transformer utilizes the time domain and frequency domain local feature representations to aggregate the global time-frequency domain feature representation. A joint scoring function composed of the time domain and frequency domain features calculates the weight of the time-frequency domain fusion feature, so that the model can fully mine the public features of the time domain, frequency domain and time-frequency domain.

[0153] Preferably, the time-frequency convergence multi-head cross-attention module comprises, in sequence, a cross-attention module, a convergence fusion layer, a linear mapping layer, a residual connection and a layer normalization operation, a feedforward network layer and a residual connection and a layer normalization operation; the cross-attention module comprises, in parallel, a time domain-time frequency domain cross-attention module, a time-frequency domain self-attention module and a frequency domain-time frequency domain cross-attention module.

[0154] In S6, the processing procedure of the public branch of the time-frequency domain Transformer is as follows:

[0155] S61: The time-frequency domain embedding feature is respectively input into the time domain-time frequency domain cross-attention module, the time-frequency domain self-attention module and the frequency domain-time frequency domain cross-attention module of the cross-attention module, to obtain a time domain-time frequency domain cross-attention feature , a time-frequency domain self-attention feature and a frequency domain-time frequency domain cross-attention feature ​.

[0156] Preferably, the cross-attention module of the time domain-time frequency domain, the self-attention module of the time frequency domain and the cross-attention module of the frequency domain-time frequency domain are parallel structures.

[0157] S611: Linearly mapping the time-frequency domain embedding feature to obtain a time-frequency query , a time-frequency key and a time-frequency value , the calculation formula is:

[0158] ;

[0159] ;

[0160] .

[0161] S612: Obtain a time domain query and a time domain key; obtain a frequency domain query and a frequency domain key; the time domain query and the time domain key are obtained by linearly mapping the time domain embedding feature; the frequency domain query and the frequency domain key are obtained by linearly mapping the frequency domain embedding feature.

[0162] S613: Linearly mapping the time domain query, the frequency domain key and the time-frequency value through the cross-attention module of the time domain-time frequency domain to obtain the cross-attention feature of the time domain-time frequency domain , the formula is:

[0163] .

[0164] S614: Linearly mapping the time-frequency query, the time-frequency key and the time-frequency value through the self-attention module of the time frequency domain to obtain the self-attention feature of the time frequency domain , the formula is:

[0165] .

[0166] S615: Linearly mapping the frequency domain query, the time domain key and the time-frequency value through the cross-attention module of the frequency domain-time frequency domain to obtain the cross-attention feature of the frequency domain-time frequency domain , the formula is:

[0167] .

[0168] S62: Input the cross-attention feature of the time domain-time frequency domain, the self-attention feature of the time frequency domain and the cross-attention feature of the frequency domain-time frequency domain into the convergence fusion layer to splice and obtain the fusion attention feature, the formula is:

[0169] ;

[0170] wherein, is the fusion attention feature.

[0171] S63: The fused attention feature is input into a linear mapping layer, and the output feature is input into a residual connection and a layer normalization operation together with the time-frequency domain embedding feature to obtain an input feature of a feedforward network layer.

[0172] S64: The input feature of the feedforward network layer is input into the feedforward network layer to obtain an output feature of the feedforward network layer; and the input feature of the feedforward network layer and the output feature of the feedforward network layer are input into a residual connection and a layer normalization operation to obtain an output feature of the time-frequency aggregation multi-head cross-attention module.

[0173] S65: The output feature of the time-frequency aggregation multi-head cross-attention module is input into a time-frequency domain classification head to perform fault classification and obtain a time-frequency domain prediction classification label.

[0174] The time-frequency domain prediction classification label is calculated by the following formula:

[0175] ;

[0176] wherein, is the time-frequency domain prediction classification label, is the time-frequency domain multi-head self-attention module, is the time-frequency domain classification head.

[0177] The core innovation of the time-frequency domain Transformer public branch lies in that it models the correlation characteristics of time-frequency, frequency-time and time-frequency-time-frequency through three cross-attention paths. Unlike the self-attention mechanism in the traditional Transformer, it is a multi-source cross-attention structure, which can more deeply excavate the synergistic relationship and potential coupling between different time domain features, thereby obtaining stronger discriminant ability and robustness in fault diagnosis.

[0178] In actual application, the time domain prediction classification label, the frequency domain prediction classification label and the time-frequency domain prediction classification label can be averaged to obtain a final output prediction classification label.

[0179] In order to make the time domain Transformer private branch, the frequency domain Transformer private branch and the time-frequency domain Transformer public branch not only learn their own advantages, but also cooperate and promote each other in the optimization target, thereby enhancing the feature sharing effect and semantic consistency, the invention adopts a multi-branch classification loss joint optimization strategy: by weighted summing the classification losses of the three branches, a global consistent optimization target is achieved.

[0180] Specifically, the time-domain classification loss, the frequency-domain classification loss and the time-frequency domain classification loss corresponding to the time-domain private branch of the Transformer, the frequency-domain private branch of the Transformer and the time-frequency domain public branch of the Transformer are calculated respectively, and the calculation formula is:

[0181] ;

[0182] ;

[0183] ;

[0184] wherein, , and are the time-domain classification loss, the frequency-domain classification loss and the time-frequency domain classification loss respectively, is the number of training samples, is the true classification label of the i-th sample, , and are the time-domain predicted classification label, the frequency-domain predicted classification label and the time-frequency domain predicted classification label of the i-th sample respectively.

[0185] Total loss function of multi-branch classification loss joint optimization strategy can be expressed as:

[0186] ;

[0187] wherein, , and are the trade-off parameters.

[0188] The total loss is used to train the rotating machinery fault detection model until the model is optimal. Then, the time-domain signal of the multi-modal rotating machinery equipment to be detected is input into the trained rotating machinery fault detection model to perform the fault prediction task.

[0189] The joint optimization of the classification losses of the three branches can make the model learn cooperatively and complement each other in multiple feature spaces, avoid overfitting or underfitting of a single branch, and significantly enhance the generalization ability of the model in complex and real industrial environments.

[0190] In general, the time-frequency convergence fusion Transformer module not only considers the private feature attributes of the time-domain features and the frequency-domain features, but also considers the public feature attributes of the time-frequency domain features. The model can improve the performance in signal processing and classification tasks by effectively fusing the multi-modal information of the time domain, the frequency domain and the time-frequency domain. In addition, the model uses the predicted classification results of the three branches to jointly optimize the model, realizing efficient information aggregation and decision fusion.

[0191] The present embodiment uses the SCARA (Selective Compliance Assembly Robot Arm) robot fault diagnosis multi-modal dataset and the PU (Paderborn University, Germany) bearing fault diagnosis multi-modal dataset to carry out experiments to verify the effectiveness and superiority of the present application.

[0192] The SCARA test platform is as shown in Figure 4 , wherein Figure 4 (a) in (a) is a robot experiment platform, Figure 4 (b) in (b) is a ball screw jamming test experiment, Figure 4 (c) in (c) is a robot ball shedding fault sample. Five kinds of electrical signals and three kinds of vibration signals are collected from the robot drive motor: J3 current feedback, J3U phase feedback current, J3V phase feedback current, J3W phase feedback current, J3d axis feedback current, and x, y, z axis vibration signals. The experiment is carried out under four different load conditions: 0 kg, 3 kg, 6 kg and 9 kg. The ball screw health status includes normal, screw nut jamming, spline nut jamming and ball shedding, and the sampling frequency is 1600 Hz. The PU test platform is as shown in Figure 5 , wherein Figure 5 (a) in (a) is a bearing vibration signal, Figure 5 (b) in (b) is a motor A-phase current signal, Figure 5 (c) in (c) is a motor B-phase current signal. The bearing vibration signal and the motor phase current signal are collected simultaneously, and the sampling frequency is 25600 Hz. The bearing health status includes normal, outer ring fault and inner ring fault.

[0193] In order to comprehensively evaluate the fault diagnosis performance of the method proposed in the present application, 13 current mainstream and representative time series modeling models and fault diagnosis models are selected as comparison objects. These models cover various variants of the Transformer structure, frequency domain enhancement methods, linear modeling methods and recurrent neural network structures, with good breadth and representativeness, as follows:

[0194] Autoforme: a time series prediction model based on trend decomposition mechanism, capable of modeling long-term trend components;

[0195] N-Transformer: a Transformer variant that introduces a game optimization mechanism to improve modeling convergence efficiency;

[0196] FEDformer: performs attention calculation in frequency space, with frequency domain modeling capability;

[0197] DLinear: Efficient time series modeling with linear prediction, emphasizing structural lightweight;

[0198] Informer: Address long sequence dependency issues through sparse attention mechanisms;

[0199] Reformer: Attention optimization using Local Sensitivity Hashing (LSH) to improve efficiency;

[0200] Transformer: Classic Transformer structure as a baseline model;

[0201] ETSformer: Fusion of exponential smoothing and Transformer to improve modeling smoothness;

[0202] MICN: Multi-input convolutional network suitable for multi-modal signal modeling;

[0203] Pyraformer: Pyramid attention structure, emphasizing modeling capabilities at different time scales;

[0204] Crossformer: Cross-time scale information modeling, focusing on changes in time series structure;

[0205] iTransformer: Simple and lightweight improved Transformer model;

[0206] SegRNN: Sequence segmentation-based recurrent structure, emphasizing segmented feature extraction.

[0207] In addition, the 14th model is the proposed method. As shown in the comparison experiment results Figure 6 , each dataset has four working conditions, three of which are used as the training set, and the remaining one is used as the test set. Taking the T0 migration task of the SCARA dataset as an example, 3, 6, and 9 kg working conditions are used as the training set, and 0 kg is used as the test set. As can be seen from Figure 6 , the proposed method performs stably and outstandingly in all working conditions of the two datasets, achieving the highest average accuracy (95.618%) among all models, significantly outperforming the 13 compared mainstream models. The proposed method comprehensively introduces a cross-modal time-frequency perception embedding module and a time-frequency convergence fusion Transformer module, through a multi-modal information adaptive extraction and joint optimization mechanism, achieving high-precision, multi-scenario adaptive rotating machinery fault diagnosis.

[0208] To further verify the contribution of each module in the invention to the final fault diagnosis performance, the ablation experiment as shown in Figure 7 is also designed to evaluate:

[0209] Ablation 1: "Temporal branch only";

[0210] Ablation 2: "Frequency branch only";

[0211] Ablation 3: "Temporal + frequency branch (without convergence)";

[0212] Ablation 4: "Temporal + frequency branch (with convergence)";

[0213] Ablation 5: "Temporal + frequency + temporal-frequency branch (without convergence)";

[0214] Complete model of the application: "Temporal + frequency + temporal-frequency branch (with convergence)" six structure configurations;

[0215] Among them, whether there is convergence specifically refers to whether to introduce a time-frequency convergence multi-head cross attention module.

[0216] The experimental results on the two multi-modal fault diagnosis data sets of SCARA robots and PU bearings show that the single branch model performs unstably and has a relatively low average accuracy (such as the frequency domain branch only 0.777914). Although there is a significant improvement after fusing the time domain and the frequency domain, there is still a performance bottleneck when the time domain and the frequency domain branch are jointly optimized (the average accuracy is 0.908375). After further introducing the "temporal and frequency domain branch", whether or not to converge, the model performance is significantly improved. The average accuracy of the final complete structure (with convergence) reaches the highest 0.95618. The above results fully show that: (1) the time-frequency domain fusion branch can effectively supplement the common characteristics that cannot be captured by the time / frequency private information; (2) the three-branch convergence mechanism helps to improve the model discrimination ability and robustness; (3) the overall structure proposed in the application has significant advantages in complex multi-modal scenarios.

[0217] Based on the above-mentioned one kind based on cross-modal time-frequency perception's rotating machinery fault diagnosis method, the application also provides a kind of rotating machinery fault diagnosis system based on cross-modal time-frequency perception, comprising:

[0218] The time domain embedding module is used to input the multi-modal time domain signal into the time domain perception embedding branch, sequentially pass through the time domain signal perception module and the first nonlinear activation mapping module, and obtain the time domain activation mapping feature layer;The time domain activation mapping feature is linearly mapped to obtain the time domain embedding feature;

[0219] The frequency domain embedding module is used to input the multi-modal frequency domain signal into the frequency domain perception embedding branch, sequentially pass through the frequency domain signal perception module and the second nonlinear activation mapping module, and obtain the frequency domain activation mapping feature layer;The frequency domain activation mapping feature is linearly mapped to obtain the frequency domain embedding feature;

[0220] The time-frequency domain embedding module is configured to input the time domain activation mapping feature and the frequency domain activation mapping feature into a time-frequency domain perception embedding branch to obtain a time-frequency domain embedding feature.

[0221] The time domain feature extraction module is configured to input the time domain embedding feature into a time domain Transformer private branch containing a time domain multi-head self-attention module to obtain a time domain prediction classification label.

[0222] The frequency domain feature extraction module is configured to input the frequency domain embedding feature into a frequency domain Transformer private branch containing a frequency domain multi-head self-attention module to obtain a frequency domain prediction classification label.

[0223] The time-frequency domain feature extraction module is configured to input the time-frequency domain embedding feature into a time-frequency domain Transformer public branch containing a time-frequency convergence multi-head cross-attention module to obtain a time-frequency domain prediction classification label.

[0224] In summary, the cross-modal time-frequency perception-based rotating machinery fault diagnosis method of the present application designs time domain, frequency domain and time-frequency domain perception embedding branches, which can adaptively process cross-modal time domain, frequency domain and time-frequency domain signals from different devices, effectively fuse multi-modal signals, comprehensively and multi-angulary perceive multi-modal signals, and generate unified representation of time domain, frequency domain and time-frequency domain embedding features, providing high-quality feature representation for subsequent fault diagnosis. This cross-modal perception embedding branch breaks through the limitation of existing embedding methods that can only process specific signal modes, has high universality, and can be widely applied to different fault diagnosis models to improve the accuracy and robustness of fault diagnosis. Further, the present application designs corresponding Transformer branches for the embedding features of time domain, frequency domain and time-frequency domain according to the private feature attributes of time domain and frequency domain and the public feature attributes of time-frequency domain, and uses a time-frequency convergence multi-head cross-attention module to aggregate global time-frequency domain feature representation based on local feature representation of time domain and frequency domain. The multiple Transformer branches designed in the present application consider the private feature attributes and public feature attributes of time-frequency information, can fully exploit the commonalities and differences between time domain and frequency domain, and improve the accuracy and robustness of fault diagnosis.

[0225] The present application can bring significant benefits in multiple aspects by solving the key problems in traditional fault diagnosis methods, including social, economic and technical benefits:

[0226] 1. Social benefits:

[0227] Improve equipment safety: The present application can effectively reduce production accidents and safety accidents caused by equipment failure by accurately and timely diagnosing faults in rotating machinery equipment, ensuring the stability of the production line and the safety of employees.

[0228] Promote the development of intelligent manufacturing and intelligent industry: The innovative technology of the invention can support the fault diagnosis module in the intelligent manufacturing system, improve the overall production efficiency and automation level, and make positive contributions to the progress of intelligent factories and intelligent industry.

[0229] 2. Economic benefits:

[0230] Reduce equipment maintenance cost: Through early fault detection and prediction, the invention can reduce the incidence of equipment failure, reduce the production stagnation caused by equipment downtime and maintenance, and significantly reduce maintenance cost and production loss;

[0231] Improve production efficiency: Precise fault diagnosis can shorten the maintenance time and reduce the downtime of the production line, thereby improving the overall production efficiency and output value, and bringing obvious economic benefits;

[0232] Promote the wide application of equipment and technology: The invention can be applied to various rotating mechanical equipment, not only improving the diagnosis level of traditional equipment, but also promoting the wide application and commercialization of new intelligent equipment, bringing greater economic returns to the manufacturing industry.

[0233] 3. Technical benefits:

[0234] Improve the accuracy and robustness of diagnosis: Through the cross-modal time-frequency perception Embedding module, the invention can adaptively process signals from different devices and different modalities, effectively fuse multi-modal signals, and improve the precision and robustness of fault diagnosis. The time-frequency convergence fusion Transformer can further optimize feature learning through multi-head self-attention mechanism, and improve the diagnosis ability of the model in complex fault signals;

[0235] Improve the universality and adaptability of the model: The invention can deal with signal changes of different devices and different fault types through deep fusion of time domain, frequency domain and time-frequency domain signals, improve the adaptability and universality of the diagnosis model. Compared with traditional methods, the model of the invention not only can be effectively applied to existing equipment, but also can be quickly adapted to new equipment or new signal patterns, with strong expansion ability;

[0236] Innovative time-frequency information fusion method: The cross-modal time-frequency perception Embedding module and the time-frequency convergence fusion Transformer module proposed by the invention provide a new idea for the fusion and optimization of time-frequency domain features, significantly improve the comprehensive processing ability of multi-modal signals in fault diagnosis, and promote the application and development of multi-modal deep learning in industrial diagnosis.

[0237] Those skilled in the art will appreciate that embodiments of the application can be devised for a method, a system, or a computer program product. Accordingly, the present application can be embodied in the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present application can take the form of a computer program product on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage devices, etc.) embodying computer readable program code.

[0238] The present application is described in reference to the flowchart and / or block diagrams of the method, apparatus (system) and computer program product according to embodiments of the application. It will be understood that each block of the flowchart and / or block diagrams, and combinations of blocks in the flowchart and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart and / or block diagram block or blocks. Figure 1 one or more functions specified in the flowchart and / or block diagram block or blocks. Figure 1 one or more functions specified in the flowchart and / or block diagram block or blocks.

[0239] These computer program instructions can also be stored in a computer- readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instructions which implement the function specified in the flowchart and / or block diagram block or blocks. Figure 1 one or more functions specified in the flowchart and / or block diagram block or blocks. Figure 1 one or more functions specified in the flowchart and / or block diagram block or blocks.

[0240] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart and / or block diagram block or blocks. Figure 1 one or more functions specified in the flowchart and / or block diagram block or blocks. Figure 1 one or more functions specified in the flowchart and / or block diagram block or blocks.

[0241] Obviously, the above-described embodiments are only examples and are not intended to limit the present application. Based on the above description, those skilled in the art can make other variations and changes of the present application. Here, it is not necessary or possible to enumerate all the embodiments. The obvious variations and changes derived from the present application are still within the scope of the present application.

Claims

1. A method for rotating machinery fault diagnosis based on cross-modal time-frequency perception, characterized in that, Comprise: inputting the multi-modal time domain signal into a time domain perception embedding branch, sequentially passing through a time domain signal perception module and a first nonlinear activation mapping module to obtain time domain activation mapping features; passing the time domain activation mapping features through a linear mapping layer to obtain time domain embedding features; inputting the multi-modal frequency domain signal into a frequency domain perception embedding branch, sequentially passing through a frequency domain signal perception module and a second nonlinear activation mapping module to obtain frequency domain activation mapping features; passing the frequency domain activation mapping features through a linear mapping layer to obtain frequency domain embedding features; inputting the time domain activation mapping features and the frequency domain activation mapping features into a time-frequency domain perception embedding branch to obtain time-frequency domain embedding features; inputting the time domain embedding features into a time domain Transformer private branch containing a time domain multi-head self-attention module to obtain time domain prediction classification labels; inputting the frequency domain embedding features into a frequency domain Transformer private branch containing a frequency domain multi-head self-attention module to obtain frequency domain prediction classification labels; inputting the time-frequency domain embedding features into a time-frequency domain Transformer public branch containing a time-frequency convergence multi-head cross-attention module to obtain time-frequency domain prediction classification labels; The time-frequency convergence multi-head cross-attention module comprises a cross-attention module, a convergence fusion layer, a linear mapping layer, a residual connection and layer normalization operation, a feedforward network layer and a residual connection and layer normalization operation connected in sequence; the cross-attention module comprises a time domain-time frequency domain cross-attention module, a time-frequency domain self-attention module and a frequency domain-time frequency domain cross-attention module connected side by side; The data processing process of the time-frequency convergence multi-head cross-attention module comprises: the time-frequency domain embedding features are respectively inputted into the time domain-time frequency domain cross-attention module, the time-frequency domain self-attention module and the frequency domain-time frequency domain cross-attention module of the cross-attention module to obtain time domain-time frequency domain cross-attention features, time-frequency domain self-attention features and frequency domain-time frequency domain cross-attention features; the time domain-time frequency domain cross-attention features, the time-frequency domain self-attention features and the frequency domain-time frequency domain cross-attention features are inputted into the convergence fusion layer for splicing to obtain fusion attention features; the fusion attention features are inputted into the linear mapping layer, and the output features thereof are inputted into the time-frequency domain embedding features through the residual connection and the layer normalization operation to obtain the input features of the feedforward network layer; the input features of the feedforward network layer are inputted into the feedforward network layer to obtain the output features of the feedforward network layer; the input features of the feedforward network layer and the output features of the feedforward network layer are inputted into the residual connection and the layer normalization operation to obtain the output features of the time-frequency convergence multi-head cross-attention module.

2. The method of claim 1, wherein, The time domain signal perception module and the frequency domain signal perception module each comprise a plurality of perception kernels; the length and the number of channels of each perception kernel are less than the length and the number of channels of the input signal, and each perception kernel has different channel starting index values and sequence starting index values.

3. The method of claim 2, wherein, The data processing process of the time domain signal perception module and the frequency domain signal perception module each comprises: the plurality of perception kernels of the current signal perception module act on different sub-regions of the input signal of the current signal perception module to obtain perception features corresponding to each perception kernel; Based on the perception features corresponding to all the perception kernels, total perception features are obtained; The total perception features are subjected to a flattening operation to obtain output features of the current signal perception module.

4. The method of claim 1, wherein, The first nonlinear activation mapping module and the second nonlinear activation mapping module each include a linear mapping layer and a SiLU activation function connected in sequence.

5. The method of claim 1, wherein, The time domain activation mapping features and the frequency domain activation mapping features are input into a time-frequency domain perception embedding branch to obtain time-frequency domain embedding features, and the process includes: The time domain activation mapping features and the frequency domain activation mapping features are spliced to obtain time-frequency fusion features; The time-frequency fusion features are sequentially subjected to a linear mapping layer, a SiLU activation function and a linear mapping layer to obtain time-frequency domain embedding features.

6. The method of claim 1, wherein, The time domain multi-head self-attention module and the frequency domain multi-head self-attention module each include a self-attention module, a residual connection and layer normalization operation, a feedforward network and a residual connection and layer normalization operation connected in sequence.

7. The method of claim 1, wherein, The data processing process of the cross-attention module includes: The time-frequency domain embedding features are subjected to linear mapping to obtain time-frequency queries, time-frequency keys and time-frequency values; The time domain queries and the time domain keys are obtained by linear mapping of the time domain embedding features; the frequency domain queries and the frequency domain keys are obtained by linear mapping of the frequency domain embedding features; The time domain queries, the frequency domain keys and the time-frequency values are input into the time domain-time frequency domain cross-attention module to obtain time domain-time frequency domain cross-attention features; The time-frequency queries, the time-frequency keys and the time-frequency values are input into the time-frequency domain self-attention module to obtain time-frequency domain self-attention features; The frequency domain queries, the time domain keys and the time-frequency values are input into the frequency domain-time frequency domain cross-attention module to obtain frequency domain-time frequency domain cross-attention features.

8. A cross-modality time-frequency perception based rotating machinery fault diagnosis system, configured to implement the cross-modality time-frequency perception based rotating machinery fault diagnosis method according to any one of claims 1 to 7. It includes: The time domain embedding module is used for inputting the multi-modal time domain signal into the time domain perception embedding branch, sequentially passing through the time domain signal perception module and the first nonlinear activation mapping module to obtain the time domain activation mapping features; The time domain activation mapping features are subjected to a linear mapping layer to obtain time domain embedding features; The frequency domain embedding module is used for inputting the multi-modal frequency domain signal into the frequency domain perception embedding branch, sequentially passing through the frequency domain signal perception module and the second nonlinear activation mapping module to obtain the frequency domain activation mapping features; The frequency domain activation mapping features are subjected to a linear mapping layer to obtain frequency domain embedding features; The time-frequency domain embedding module is used for inputting the time domain activation mapping features and the frequency domain activation mapping features into the time-frequency domain perception embedding branch to obtain time-frequency domain embedding features; The time domain feature extraction module is used for inputting the time domain embedding features into the time domain Transformer private branch containing the time domain multi-head self-attention module to obtain the time domain predicted classification label; The frequency domain feature extraction module is used for inputting the frequency domain embedding features into the frequency domain Transformer private branch containing the frequency domain multi-head self-attention module to obtain the frequency domain predicted classification label; The time-frequency domain feature extraction module is used for inputting the time-frequency domain embedding features into the time-frequency domain Transformer public branch containing the time-frequency convergent multi-head cross-attention module to obtain the time-frequency domain predicted classification label.

Images