Rotating machinery fault diagnosis method and system based on cross-modal time frequency perception

Abstract

The invention relates to the technical field of mechanical fault diagnosis, in particular to a rotating machine fault diagnosis method and system based on cross-modal time-frequency perception, which designs time domain, frequency domain and time-frequency domain perception embedded branches, can adaptively process cross-modal time domain, frequency domain and time-frequency domain signals from different devices, and improves the accuracy of fault diagnosis. Effective fusion of multi-modal signals is realized, and the precision and robustness of fault diagnosis are improved. For private feature attributes of a time domain and a frequency domain and public feature attributes of the time domain and the frequency domain, corresponding Transformer branches are designed for embedded features of the time domain, the frequency domain and the time-frequency domain respectively, a time-frequency convergence multi-head cross attention module is utilized, local features of the time domain and the frequency domain are used for representing, global time-frequency domain features are aggregated for representing, and the time-frequency domain and the frequency domain are combined for representing. And the generality and the difference between the time domain and the frequency domain are fully mined, and the accuracy and the robustness of fault diagnosis are further improved.

Description

Technical Field

[0001] This invention relates to the field of mechanical fault diagnosis technology, and in particular to a method and system for diagnosing rotating machinery faults based on cross-modal time-frequency sensing. Background Technology

[0002] Rotating machinery is widely used in industrial production, such as electric motors, wind turbines, robots, and gearboxes. These devices are prone to malfunctions during operation, leading to production stoppages, equipment damage, increased maintenance costs, and even more serious safety accidents. Therefore, the research and application of early fault diagnosis technologies are of great significance for ensuring the normal operation of equipment, improving production efficiency, and reducing maintenance costs.

[0003] Currently, basic models based on sequence signal data have achieved good results in fault diagnosis of rotating machinery. However, existing cross-modal fault diagnosis methods still have the following problems:

[0004] (1) Inability to adaptively perceive heterogeneous multimodal signals from rotating machinery: Signal embedding is a crucial step in fault diagnosis and significantly impacts the diagnostic results of the model. Existing embedding methods include time-domain embedding and frequency-domain embedding. These methods only perceive signals from the time or frequency domain, neglecting the importance of joint optimization of time and frequency domain signals. Furthermore, they cannot effectively adapt to heterogeneous multimodal signals from different acquisition devices, resulting in the inadequate processing and utilization of diverse signals in rotating machinery systems, thus affecting diagnostic accuracy.

[0005] (2) The time-frequency sensing embedding method has poor versatility: Existing time-frequency sensing embedding methods are usually only for a single diagnostic model or a specific application scenario, lacking versatility. Each time the model is changed, a separate embedding module needs to be configured, which reduces the application value and promotion of the embedding method.

[0006] (3) Insufficient feature fusion of time-frequency domain information: Some existing fault diagnosis methods based on cross-modal learning, such as those using sensor fusion technology and joint learning of image and time series data, require setting up a separate encoder for each modality to extract features. This operation not only increases the amount of computation but also reduces its flexibility and versatility. In addition, some methods have designed time-frequency joint Transformer fusion methods. However, this method inputs the combined time-domain and frequency-domain data into a Transformer branch without considering the private and common feature attributes of the time-frequency domain signals. As a result, when processing fault signals in unknown domains, the accuracy and robustness of the model are limited, reducing the model's generalization performance.

[0007] In summary, existing methods for fault diagnosis of rotating machinery struggle to effectively address the fusion of time-domain and frequency-domain information through embedding techniques, and they also lack cross-modal adaptive sensing and flexibility. Furthermore, existing time-frequency joint Transformer models fail to adequately consider the private and public feature attributes of time-frequency domain signals, resulting in poor generalization performance. Summary of the Invention

[0008] Therefore, the technical problem to be solved by the present invention is to overcome the problem that the existing technology is difficult to adaptively perceive multimodal time-frequency domain information and difficult to fuse time-frequency domain features, resulting in reduced fault diagnosis accuracy and poor generalization performance.

[0009] To address the aforementioned technical problems, this invention provides a method for diagnosing rotating machinery faults based on cross-modal time-frequency sensing, comprising: The multimodal time-domain signal is input into the time-domain sensing embedding branch, and then passes through the time-domain signal sensing module and the first nonlinear activation mapping module in sequence to obtain the time-domain activation mapping feature; the time-domain activation mapping feature is then passed through the linear mapping layer to obtain the time-domain embedding feature; The multimodal frequency domain signal is input into the frequency domain sensing embedding branch, and then passes through the frequency domain signal sensing module and the second nonlinear activation mapping module to obtain the frequency domain activation mapping features. The frequency domain activation mapping features are then passed through the linear mapping layer to obtain the frequency domain embedding features. Input the time-domain activation mapping features and the frequency-domain activation mapping features into the time-frequency domain perceptual embedding branch to obtain the time-frequency domain embedding features; Input the temporal embedded features into the temporal Transformer private branch containing the temporal multi-head self-attention module to obtain the temporal predicted classification label; Input the frequency domain embedded features into the frequency domain Transformer private branch containing the frequency domain multi-head self-attention module to obtain the frequency domain predicted classification label; By inputting the time-frequency domain embedded features into the common branch of the time-frequency domain Transformer, which includes the time-frequency convergence multi-head cross-attention module, the predicted classification label in the time-frequency domain is obtained.

[0010] Preferably, both the time-domain signal sensing module and the frequency-domain signal sensing module include multiple sensing cores; the length and number of channels of the sensing core are smaller than the length and number of channels of the input signal, respectively, and each sensing core has a different channel start index value and sequence start index value.

[0011] Preferably, the data processing procedures of both the time-domain signal sensing module and the frequency-domain signal sensing module include: The multiple sensing kernels of the current signal sensing module act on different sub-regions of the input signal of the current signal sensing module to obtain the sensing features corresponding to each sensing kernel; Based on the perceptual features corresponding to all perceptual kernels, the total perceptual features are obtained; The total sensed features are flattened to obtain the output features of the current signal sensing module.

[0012] Preferably, both the first nonlinear activation mapping module and the second nonlinear activation mapping module include a linear mapping layer and a SiLU activation function connected in sequence.

[0013] Preferably, the time-domain activation mapping features and frequency-domain activation mapping features are input into the time-frequency domain perceptual embedding branch to obtain time-frequency domain embedding features. The process includes: The time-domain activation mapping features and the frequency-domain activation mapping features are concatenated to obtain the time-frequency fusion features; The time-frequency fusion features are sequentially passed through a linear mapping layer, a SiLU activation function, and another linear mapping layer to obtain the time-frequency domain embedded features.

[0014] Preferably, both the time-domain multi-head self-attention module and the frequency-domain multi-head self-attention module include a self-attention module, a residual connection and layer normalization operation, a feedforward network, and a residual connection and layer normalization operation connected in sequence.

[0015] Preferably, the time-frequency convergence multi-head cross-attention module includes a cross-attention module, a convergence fusion layer, a linear mapping layer, a residual connection and layer normalization operation, a feedforward network layer, and a residual connection and layer normalization operation connected in sequence; the cross-attention module includes a time-domain-time-frequency domain cross-attention module, a time-frequency domain self-attention module, and a frequency domain-time-frequency domain cross-attention module connected in parallel.

[0016] Preferably, the data processing procedure of the time-frequency convergence multi-head cross-attention module includes: The time-frequency domain embedded features are passed through the time-frequency domain cross-attention module, the time-frequency domain self-attention module, and the frequency domain-time-frequency domain cross-attention module, respectively, to obtain the time-frequency domain cross-attention features, the time-frequency domain self-attention features, and the frequency domain-time-frequency domain cross-attention features. The cross-attention features between the time domain and the time-frequency domain, the self-attention features in the time-frequency domain, and the cross-attention features between the frequency domain and the time-frequency domain are input into the convergence and fusion layer and spliced ​​together to obtain the fused attention features; The fused attention features are passed through a linear mapping layer, and their output features and time-frequency domain embedded features are passed through residual connections and layer normalization operations to obtain the input features of the feedforward network layer. The input features of the feedforward network layer are passed through the feedforward network layer to obtain the output features of the feedforward network layer. The input features and output features of the feedforward network layer are then passed through residual connections and layer normalization operations to obtain the output features of the time-frequency convergence multi-head cross-attention module.

[0017] Preferably, the data processing procedure of the cross-attention module includes: The time-frequency domain embedded features are linearly mapped to obtain time-frequency query, time-frequency key, and time-frequency value; Obtain the time-domain query and time-domain key; obtain the frequency-domain query and frequency-domain key; the time-domain query and time-domain key are obtained by linear mapping from the time-domain embedding features; the frequency-domain query and frequency-domain key are obtained by linear mapping from the frequency-domain embedding features. The time-domain query, frequency-domain key, and time-frequency value are passed through a time-domain-time-frequency-domain cross-attention module to obtain the time-domain-time-frequency-domain cross-attention feature; The time-frequency query, time-frequency key, and time-frequency value are passed through a time-frequency domain self-attention module to obtain the time-frequency domain self-attention features; The frequency domain query, time domain key, and time-frequency value are passed through a frequency domain-time-frequency domain cross-attention module to obtain the frequency domain-time-frequency domain cross-attention feature.

[0018] The present invention also provides a rotating machinery fault diagnosis system based on cross-modal time-frequency sensing, comprising: The time-domain embedding module is used to input multimodal time-domain signals into the time-domain sensing embedding branch, which then passes through the time-domain signal sensing module and the first nonlinear activation mapping module to obtain time-domain activation mapping features. The time-domain activation mapping features are then passed through a linear mapping layer to obtain time-domain embedding features. The frequency domain embedding module is used to input multimodal frequency domain signals into the frequency domain sensing embedding branch, which then passes through the frequency domain signal sensing module and the second nonlinear activation mapping module to obtain frequency domain activation mapping features. The frequency domain activation mapping features are then passed through a linear mapping layer to obtain frequency domain embedding features. The time-frequency domain embedding module is used to input the time-domain activation mapping features and the frequency-domain activation mapping features into the time-frequency domain perceptual embedding branch to obtain the time-frequency domain embedding features. The temporal feature extraction module is used to input temporal embedded features into the temporal Transformer private branch containing the temporal multi-head self-attention module to obtain temporal predicted classification labels. The frequency domain feature extraction module is used to input frequency domain embedded features into the frequency domain Transformer private branch containing the frequency domain multi-head self-attention module to obtain frequency domain predicted classification labels. The time-frequency domain feature extraction module is used to input the time-frequency domain embedded features into the common branch of the time-frequency domain Transformer containing the time-frequency convergence multi-head cross attention module, and obtain the time-frequency domain predicted classification label.

[0019] Compared with the prior art, the above-described technical solution of the present invention has the following advantages: This invention discloses a rotating machinery fault diagnosis method based on cross-modal time-frequency sensing. It designs time-domain, frequency-domain, and time-frequency-domain sensing embedding branches, enabling adaptive processing of cross-modal time-domain, frequency-domain, and time-frequency-domain signals from different devices. This achieves effective fusion of multi-modal signals and generates unified representations of time-domain, frequency-domain, and time-frequency-domain embedded features. This cross-modal sensing embedding branch overcomes the limitation of existing embedding methods that can only process specific signal modes, exhibiting high versatility and wide applicability to different fault diagnosis models, thus improving the accuracy and robustness of fault diagnosis. Furthermore, this invention designs corresponding Transformer branches for the embedded features in the time-domain, frequency-domain, and time-frequency-domain, respectively, considering both private and public feature attributes in the time and frequency domains. It also utilizes a time-frequency convergence multi-head cross-attention module to aggregate global time-frequency-domain feature representations from local time-domain and frequency-domain feature representations. The multiple Transformer branches designed in this invention consider both private and public feature attributes of time-frequency information, fully exploring the commonalities and differences between the time and frequency domains, thereby improving the accuracy and robustness of fault diagnosis.

[0020] This invention, through deep fusion of time-domain, frequency-domain, and time-frequency-domain signals, can address signal variations across different devices and fault types, improving the adaptability and universality of the diagnostic model. Compared to traditional methods, the model of this invention can not only be effectively applied to existing devices but also quickly adapt to new devices or new signal patterns, demonstrating strong scalability. Attached Figure Description

[0021] To make the content of this invention easier to understand, the invention will be further described in detail below with reference to specific embodiments and accompanying drawings, wherein: Figure 1 This is a flowchart of a rotating machinery fault diagnosis method based on cross-modal time-frequency sensing according to the present invention; Figure 2 This is a structural diagram of the cross-modal time-frequency sensing embedded module; Figure 3 This is a structural diagram of the time-frequency convergence and fusion Transformer module; Figure 4 This is a schematic diagram of the SCARA experimental platform used in this embodiment of the invention, wherein... Figure 4 (a) in the image represents the robot experimental platform. Figure 4 (b) in the figure represents the ball screw jamming test experiment. Figure 4 (c) in the image represents a robot ball bearing detachment failure sample; Figure 5 This is a schematic diagram of the PU dataset experimental platform used in this embodiment of the invention, wherein... Figure 5 (a) in the diagram represents the vibration signal of the bearing. Figure 5 (b) in the diagram represents the A-phase current signal of the motor. Figure 5 (c) in the figure represents the B-phase current signal of the motor; Figure 6 This is a comparison chart of fault prediction results from different models; Figure 7 This is a schematic diagram of the ablation model experimental results; Instruction manual diagram markings: 1. J3 axis drive motor; 2. Control panel; 3. Computer; 4. J3 axis; 5. Lead screw; 6. Load; 7. Helical nut; 8. Spline nut; 9. Push-pull force gauge; 10. Ball bearing detachment; 11. Motor; 12. Torque strategy axis; 13. Bearing test module; 14. Flywheel; 15. Load motor. Detailed Implementation

[0022] The present invention will be further described below with reference to the accompanying drawings and specific embodiments, so that those skilled in the art can better understand and implement the present invention. However, the embodiments described are not intended to limit the present invention.

[0023] Reference Figure 1 As shown, this invention provides a method for diagnosing rotating machinery faults based on cross-modal time-frequency sensing, comprising: S1: Input the multimodal time-domain signal into the time-domain sensing embedding branch, and pass it sequentially through the time-domain signal sensing module and the first nonlinear activation mapping module to obtain the time-domain activation mapping feature; pass the time-domain activation mapping feature through the linear mapping layer to obtain the time-domain embedding feature; S2: Input the multimodal frequency domain signal into the frequency domain sensing embedding branch, and pass it through the frequency domain signal sensing module and the second nonlinear activation mapping module in sequence to obtain the frequency domain activation mapping features; pass the frequency domain activation mapping features through the linear mapping layer to obtain the frequency domain embedding features; S3: Input the time-domain activation mapping features and the frequency-domain activation mapping features into the time-frequency domain perceptual embedding branch to obtain the time-frequency domain embedding features; S4: Input the temporal embedded features into the temporal Transformer private branch containing the temporal multi-head self-attention module to obtain the temporal predicted classification label; S5: Input the frequency domain embedded features into the frequency domain Transformer private branch containing the frequency domain multi-head self-attention module to obtain the frequency domain predicted classification label; S6: Input the time-frequency domain embedded features into the common branch of the time-frequency domain Transformer containing the time-frequency convergence multi-head cross attention module to obtain the time-frequency domain predicted classification label.

[0024] The cross-modal time-frequency sensing embedding module comprises three branches: a time-domain sensing embedding branch, a frequency-domain sensing embedding branch, and a time-frequency domain sensing embedding branch. The time-domain Transformer private branch, the frequency-domain Transformer private branch, and the time-frequency domain Transformer public branch constitute the time-frequency convergence and fusion Transformer module. The cross-modal time-frequency sensing embedding module and the time-frequency convergence and fusion Transformer module together form the rotating machinery fault detection model. The structure and processing of each module are explained in detail below.

[0025] Reference Figure 2 As shown, the cross-modal time-frequency sensing embedding module first performs a Fast Fourier Transform on the multimodal time-domain signal to obtain the multimodal frequency-domain signal. The time-domain sensing embedding branch and the frequency-domain sensing embedding branch perceive the multimodal time-domain and frequency-domain signals as time-domain and frequency-domain sensing features, respectively. Then, these features are passed through a nonlinear activation mapping to obtain activation mapping features, and then through a linear mapping layer to obtain a unified representation of the embedding features. Finally, the time-domain activation mapping features and the frequency-domain activation mapping features are concatenated to form a time-frequency fusion feature, and the time-frequency fusion feature is passed through a nonlinear activation mapping to obtain a unified representation of the time-frequency domain embedding features.

[0026] Multimodal signals include speed feedback signals, current signals, encoder signals, and vibration signals from rotating machinery.

[0027] In both the time-domain sensing embedding branch and the frequency-domain sensing embedding branch, the time-domain signal sensing module and the frequency-domain signal sensing module include multiple sensing kernels. The number, length, and number of channels of the sensing kernels are set as follows: , and ,in This also refers to the number of modes for cross-modal sensing by the sensing kernel. Assume the number and length of the modes (channels) of the signal from the rotating machinery are respectively... and The length and number of channels of the sensing kernel are smaller than the length and number of channels of the input signal, respectively, and each sensing kernel has a different channel start index value and sequence start index value. Specifically, in the horizontal sequence dimension, different sensing kernels have different sequence start index values. In the vertical channel dimension, different sensing kernels also have different channel start index values. Therefore, the sensing kernel can perceive multimodal information from the input signal from all directions and multiple angles.

[0028] For example, suppose a rotating mechanical device has 10 modal signals with a signal length of 1024. The number of channels 'c' of the sensing core needs to be less than 10, typically set to 6-8. Assuming the sensing core has 6 channels and a length of 128, and one sensing core has a channel start index of 2 (less than or equal to [10-6+1=5]), then this sensing core can sense the signals from channels 2 to 7 of the device. Similarly, in the horizontal dimension, assuming one sensing core has a sequence start index of 10 (less than [1024-128+1=897]), then this sensing core can sense the features of data points 10 to 137 of the device signal. Therefore, this sensing core can sense sub-regions of channels 2 to 7 (vertical) and data points 10 to 137 (horizontal) of the input signal, obtaining the corresponding sensing features.

[0029] It is important to note that, in order to enable the sensing cores to extract signals from different channels and locations, this invention sets up multiple sensing cores in parallel. Each sensing core has a different channel start index value and sequence start index value. Furthermore, setting the number of channels in the sensing cores to be less than the number of channels in the input signal reduces computational load. For example, if the input signal has 10 channels and the sensing cores have 6 channels, since different sensing cores have different channel start index values, setting up multiple sensing cores can still sense information from all 10 modalities. This design reduces computational load and avoids signal feature loss.

[0030] Specifically, the data processing procedures of both the time-domain signal sensing module and the frequency-domain signal sensing module include: The multiple sensing kernels of the current signal sensing module act on different sub-regions of the input signal of the current signal sensing module to obtain the sensing features corresponding to each sensing kernel; Based on the perceptual features corresponding to all perceptual kernels, the total perceptual features are obtained; The total sensed features are flattened to obtain the output features of the current signal sensing module.

[0031] Preferably, both the first nonlinear activation mapping module and the second nonlinear activation mapping module include a linear mapping layer and a SiLU activation function connected in sequence.

[0032] Assume the multimodal time-domain signal is The processing procedure of the time-domain-aware embedding branch in S1 is as follows:

[0033] S11: Multiple time-domain sensing kernels of the time-domain signal sensing module act on different sub-regions of the multimodal time-domain signal to obtain the sensing features corresponding to each time-domain sensing kernel; assuming the first... The sequence start index value of each time-domain sensing kernel is The channel starting index value is Then the first The temporal sensing features obtained from a temporal sensing kernel can be represented as: ; in, For the i-th time-domain perceived feature, For sensing operations, This is a multimodal time-domain signal. It's important to note that the intervals [j: j+c) and [k: k+l) are both closed at the beginning and open at the end.

[0034] S12: Based on all time-domain sensing features, obtain the total time-domain sensing features.

[0035] Total temporal sensing features Represented as: .

[0036] S13: Incorporate temporal domain total sensing features After flattening, the output characteristics of the time-domain signal sensing module are obtained. : ; in, For flattening operation.

[0037] S14: The output characteristics of the time-domain signal sensing module After passing through the first nonlinear activation mapping module, the temporal activation mapping features are obtained. The formula is expressed as: ; in, For time-domain activation mapping features, The SiLU activation function is used. It is the first linear mapping layer in the time domain.

[0038] S15: Map features to the time-domain activation domain After a linear mapping layer, a unified representation of the temporal embedding features is obtained. The formula is expressed as: ; in, For temporal embedding features, It is the second linear mapping layer in the time domain. The output dimension of the temporal embedded features.

[0039] Number of sensing nuclei and output dimensions The data dimensions that the model can connect to can be flexibly set, thus ensuring that the module can be used with different models.

[0040] In S2, the processing principle of the frequency domain-aware embedding branch is the same as that of the time domain-aware embedding branch, and the specific process is as follows:

[0041] S21: Perform a fast Fourier transform on the multimodal time-domain signal to obtain the multimodal frequency-domain signal.

[0042] Multimodal frequency domain signals can be calculated using the following formula: ; ; in, The output signal of the Fast Fourier Transform. It is a multi-mode frequency domain signal. For the set of complex numbers, It is the set of real numbers.

[0043] S22: Multiple frequency domain sensing kernels of the frequency domain signal sensing module act on different sub-regions of the multimodal frequency domain signal to obtain the sensing features corresponding to each frequency domain sensing kernel; assuming the first... The sequence start index value of each frequency domain sensing kernel is The channel starting index value is Then the first The frequency domain sensing features obtained from a single frequency domain sensing kernel can be represented as: ; in, Let i be the i-th frequency domain sensing feature.

[0044] S23: Based on all frequency domain sensing features, obtain the total frequency domain sensing features.

[0045] Frequency domain total sensing features Represented as: .

[0046] S24: Total frequency domain sensing features After flattening, the output characteristics of the frequency domain signal sensing module are obtained. : .

[0047] S25: The output characteristics of the frequency domain signal sensing module After passing through the second nonlinear activation mapping module, the temporal activation mapping features are obtained. The formula is expressed as: ; in, For frequency domain activation mapping features, The SiLU activation function is used. It is the first linear mapping layer in the frequency domain.

[0048] S26: Map frequency domain activation features After a linear mapping layer, frequency domain embedding features with a unified representation are obtained. The formula is expressed as: ; in, For frequency domain embedding features, It is the second linear mapping layer in the frequency domain.

[0049] The time-frequency sensing embedding branch proposed in this invention can leverage the complementarity of time-domain and frequency-domain information to provide a more comprehensive and accurate basis for fault diagnosis. Firstly, the time-frequency sensing embedding branch maps time-domain activation features... and frequency domain activation mapping features The time-frequency domain fusion features are concatenated and then processed through nonlinear activation mapping to obtain a unified time-frequency domain embedded feature.

[0050] Specifically, in S3, the time-domain activation mapping features and the frequency-domain activation mapping features are input into the time-frequency domain perceptual embedding branch to obtain the time-frequency domain embedding features. The process includes:

[0051] S31: Concatenate the time-domain activation mapping features and the frequency-domain activation mapping features to obtain the time-frequency fusion features, expressed by the formula: ; in, For feature concatenation function, It is a time-frequency fusion feature.

[0052] S32: The time-frequency fusion features are sequentially passed through a linear mapping layer, a SiLU activation function, and another linear mapping layer to obtain the time-frequency domain embedding features, expressed by the formula: ; in, For time-frequency domain embedding features, and These are the first linear mapping layer in the time-frequency domain and the second linear mapping layer in the time-frequency domain, respectively.

[0053] The time-frequency domain-aware embedding branch enhances the model's ability to comprehensively perceive information in the time, frequency, and time-frequency domains by generating time-frequency embedding features with a unified representation of features in the time and frequency domains.

[0054] The advantages of the cross-modal time-frequency sensing embedding module designed in this invention include: the number, length, and number of cross-modal channels of sensing kernels can be freely set; after linear and activation transformations, it can be seamlessly connected to different fault diagnosis sequence models, and has a plug-and-play function; the sensing kernels have no weight parameters, which can reduce the number of parameters in the model.

[0055] Traditional fault diagnosis methods may be limited to time-domain or frequency-domain characteristics, failing to fully utilize the characteristics of the time domain, frequency domain, and their fusion. This invention designs a time-frequency convergence and fusion Transformer module. (Refer to...) Figure 3 As shown, the time-frequency convergence and fusion Transformer module aims to improve the overall model performance through three branches: a private branch in the time domain, a private branch in the frequency domain, and a common branch in the time-frequency domain. These three branches are functionally independent but can be optimized collaboratively. They can separately process and fuse embedded features from the time domain, frequency domain, and time-frequency domain, thereby improving the overall model performance. Specifically, the private branch in the time domain is used to extract temporal dynamic information; the private branch in the frequency domain is used to model spectral patterns and amplitude periodicity information; and the common branch in the time-frequency domain employs a cross-attention mechanism to fuse key time / frequency features.

[0056] The temporal Transformer private branch includes a temporal multi-head self-attention module and a temporal classification head. The frequency domain Transformer private branch includes a frequency domain multi-head self-attention module and a frequency domain classification head. The time-frequency domain Transformer public branch includes a time-frequency convergent multi-head cross-attention module and a time-frequency domain classification head. The temporal and frequency domain multi-head self-attention modules are used to mine long-term dependencies in the time and frequency domains, respectively, while the classification head is used to perform fault classification tasks.

[0057] Both the time-domain multi-head self-attention module and the frequency-domain multi-head self-attention module include a self-attention module, a residual connection and layer normalization operation, a feedforward network, and a residual connection and layer normalization operation connected in sequence.

[0058] In S4, the processing procedure for the private branch of the time-domain Transformer is as follows:

[0059] S41: Embedding features in the temporal domain with a unified representation Input a time-domain multi-head self-attention module to compute time-domain queries. Time domain key and time domain value The calculation formula is: ; ; ; in, , and These are weight parameters.

[0060] S42: Time domain query Time domain key and time domain value Input the temporal self-attention module and calculate the temporal self-attention features using the following formula: ; in, For temporal self-attention features, The Softmax activation function is used. This is a scaling factor used to prevent gradient explosion caused by excessively large inner products.

[0061] S43: Incorporating temporal self-attention features With temporal embedding features The residual features are added together and then normalized at each layer before being input into the feedforward network. The input and output features of the feedforward network are added together and normalized at each layer to obtain the output features of the temporal multi-head self-attention module.

[0062] S44: Input the output features of the temporal multi-head self-attention module into the temporal classification head to perform fault classification and obtain temporal predicted classification labels.

[0063] Temporal domain predicted classification labels are calculated using the following formula: ; in, Predict classification labels for the time domain. For time-domain multi-head self-attention modules, For time-domain classification headers.

[0064] In S5, the processing principle of the frequency domain Transformer private branch is the same as that of the time domain Transformer private branch, and the specific process is as follows:

[0065] S51: Embedding features in the frequency domain with a unified representation Input a time-domain multi-head self-attention module and calculate a frequency-domain query. Frequency domain key and frequency domain value The calculation formula is: ; ; .

[0066] S52: Frequency domain query Frequency domain key and frequency domain value Input the frequency domain self-attention module and calculate the frequency domain self-attention features. The calculation formula is: .

[0067] S53: Frequency domain self-attention features With frequency domain embedding features The residual features are added together and then normalized at each layer before being input into the feedforward network. The input and output features of the feedforward network are added together and normalized at each layer to obtain the output features of the frequency domain multi-head self-attention module.

[0068] S54: Input the output features of the frequency domain multi-head self-attention module into the frequency domain classification head to perform fault classification and obtain the frequency domain predicted classification label.

[0069] Frequency domain predicted classification labels are calculated using the following formula: ; in, Predict classification labels in the frequency domain. It is a frequency domain multi-head self-attention module. This is a frequency domain classification header.

[0070] The common branch of the time-frequency domain Transformer aggregates global time-frequency domain feature representations using local time-domain and frequency-domain feature representations. A joint scoring function composed of time-domain and frequency-domain features calculates the weights of the fused time-frequency domain features, enabling the model to fully exploit the common features of the time-domain, frequency-domain, and time-frequency-domain domains.

[0071] Preferably, the time-frequency convergence multi-head cross-attention module includes a cross-attention module, a convergence fusion layer, a linear mapping layer, a residual connection and layer normalization operation, a feedforward network layer, and a residual connection and layer normalization operation connected in sequence; the cross-attention module includes a time-domain-time-frequency domain cross-attention module, a time-frequency domain self-attention module, and a frequency domain-time-frequency domain cross-attention module connected in parallel.

[0072] In S6, the processing procedure for the common branch of the Transformer in the time-frequency domain is as follows:

[0073] S61: Embedding features in the time-frequency domain The time-frequency domain cross-attention features are obtained by passing the cross-attention module through the time-frequency domain cross-attention module, the time-frequency domain self-attention module, and the frequency domain-time-frequency domain cross-attention module, respectively. Self-attention features in the time and frequency domain and frequency domain-time frequency domain cross-attention features .

[0074] Preferably, the time-domain-time-frequency domain cross-attention module, the time-frequency domain self-attention module, and the frequency domain-time-frequency domain cross-attention module are parallel structures.

[0075] S611: Obtain the time-frequency query by linearly mapping the time-frequency domain embedded features. Time / Frequency Key and time-frequency values The calculation formula is: ; ; .

[0076] S612: Obtain time-domain query and time-domain key; obtain frequency-domain query and frequency-domain key; the time-domain query and time-domain key are obtained by linear mapping of time-domain embedding features; the frequency-domain query and frequency-domain key are obtained by linear mapping of frequency-domain embedding features.

[0077] S613: Pass the time-domain query, frequency-domain key, and time-frequency value through a time-domain-time-frequency-domain cross-attention module to obtain the time-domain-time-frequency-domain cross-attention feature. The formula is: .

[0078] S614: Pass the time-frequency query, time-frequency key, and time-frequency value through a time-frequency domain self-attention module to obtain the time-frequency domain self-attention features. The formula is: .

[0079] S615: Pass the frequency domain query, time domain key, and time-frequency value through a frequency domain-time-frequency domain cross-attention module to obtain the frequency domain-time-frequency domain cross-attention feature. The formula is: .

[0080] S62: The cross-attention features between the time domain and the time-frequency domain, the self-attention features in the time-frequency domain, and the cross-attention features between the frequency domain and the time-frequency domain are input into the convergence and fusion layer and concatenated to obtain the fused attention features, expressed by the formula: ; in, To integrate attention features.

[0081] S63: The fused attention features are passed through a linear mapping layer, and their output features and time-frequency domain embedded features are passed through residual connections and layer normalization operations to obtain the input features of the feedforward network layer.

[0082] S64: Pass the input features of the feedforward network layer through the feedforward network layer to obtain the output features of the feedforward network layer; pass the input features and output features of the feedforward network layer through residual connection and layer normalization operations to obtain the output features of the time-frequency convergence multi-head cross-attention module.

[0083] S65: Input the output features of the time-frequency convergence multi-head cross-attention module into the time-frequency domain classification head to perform fault classification and obtain the time-frequency domain predicted classification label.

[0084] The time-frequency domain predicted classification label is calculated by the following formula: ; in, Predict classification labels in the time-frequency domain. It is a time-frequency domain multi-head self-attention module. This is a time-frequency domain classification header.

[0085] The core innovation of the time-frequency domain Transformer public branch lies in its modeling of the correlation characteristics between time and frequency, frequency and time, and time-frequency and time-frequency through three cross-attention paths. Unlike the self-attention mechanism in the traditional Transformer, it is a multi-source cross-attention structure that can more deeply explore the synergistic relationships and potential couplings between different time-domain features, thereby achieving stronger discrimination ability and robustness in fault diagnosis.

[0086] In practical applications, the average of the time-domain predicted classification label, the frequency-domain predicted classification label, and the time-frequency-domain predicted classification label can be calculated and used as the final output predicted classification label.

[0087] To enable the private branches of the time-domain Transformer, the private branches of the frequency-domain Transformer, and the common branches of the time-frequency-domain Transformer to not only learn their respective advantageous features, but also to cooperate and promote each other in terms of optimization objectives, thereby enhancing feature sharing effects and semantic consistency, a multi-branch classification loss joint optimization strategy is proposed: by weighted summation of the classification losses of the three branches, a globally consistent optimization objective is achieved.

[0088] Specifically, the time-domain classification loss, frequency-domain classification loss, and time-frequency-domain classification loss are calculated for the time-domain Transformer private branch, the frequency-domain Transformer private branch, and the time-frequency-domain Transformer public branch, respectively, using the following formulas: ; ; ; in, , and These are time-domain classification loss, frequency-domain classification loss, and time-frequency-domain classification loss, respectively. The number of training samples, Let i be the true classification label of the i-th sample. , and These are the time-domain predicted classification label, frequency-domain predicted classification label, and time-frequency-domain predicted classification label for the i-th sample, respectively.

[0089] The total loss function of the multi-branch classification loss joint optimization strategy It can be represented as: ; in, , and To weigh the parameters.

[0090] Total loss Train the rotating machinery fault detection model until it reaches its optimal state. Then, input the time-domain signal of the multimodal rotating machinery to be detected into the trained rotating machinery fault detection model to perform the fault prediction task.

[0091] Joint optimization of the classification loss of the three branches enables the model to learn collaboratively and complement each other in multiple feature spaces, avoiding overfitting or underfitting of a single branch, and significantly enhancing the model's generalization ability in complex and real industrial environments.

[0092] In summary, the time-frequency convergence and fusion Transformer module considers not only the private feature attributes of time-domain and frequency-domain features, but also the common feature attributes of time-frequency-domain features. By effectively fusing multimodal information from the time, frequency, and time-frequency domains, the model can improve performance in signal processing and classification tasks. Furthermore, the model utilizes the prediction and classification results from the three branches to jointly optimize the model, achieving efficient information aggregation and decision fusion.

[0093] This implementation uses the SCARA (Selective Compliance Assembly Robot Arm) robot fault diagnosis multimodal dataset and the PU (Paderborn University) bearing fault diagnosis multimodal dataset to conduct experiments, in order to verify the effectiveness and superiority of the present invention.

[0094] SCARA testing platform such as Figure 4 As shown, where Figure 4 (a) in the image represents the robot experimental platform. Figure 4 (b) in the figure represents the ball screw jamming test experiment. Figure 4 (c) shows a sample of a robot ball screw failure. Five electrical signals and three vibration signals were collected from the robot's drive motor: J3 current feedback, J3U-phase feedback current, J3V-phase feedback current, J3W-phase feedback current, J3d-axis feedback current, and vibration signals along the x, y, and z axes. The experiment was conducted under four different load conditions: 0 kg, 3 kg, 6 kg, and 9 kg. Ball screw health status included normal, stuck helical nut, stuck spline nut, and ball screw failure. The sampling frequency was 1600 Hz. The PU testing platform was as follows: Figure 5 As shown, where Figure 5 (a) in the diagram represents the vibration signal of the bearing. Figure 5 (b) in the diagram represents the A-phase current signal of the motor. Figure 5 (c) represents the B-phase current signal of the motor. Bearing vibration signals and motor phase current signals are simultaneously acquired at a sampling frequency of 25600Hz. Bearing health status includes normal, outer ring fault, and inner ring fault.

[0095] To comprehensively evaluate the fault diagnosis performance of the proposed method, this embodiment selects 13 mainstream and representative time series modeling and fault diagnosis models as comparison objects. These models cover various variants of the Transformer structure, frequency domain enhancement methods, linear modeling methods, and recurrent neural network structures, demonstrating good breadth and representativeness, as detailed below: Autoforme: A time series forecasting model based on a trend decomposition mechanism, capable of modeling long-term trend components; N-Transformer: A Transformer variant that introduces a game-theoretic optimization mechanism to improve modeling convergence efficiency; FEDformer: Performs attention calculations in the frequency space and has frequency domain modeling capabilities; DLinear: Utilizes linear forecasting for efficient time series modeling, emphasizing lightweight structure; Informer: Handles long sequence dependencies through sparse attention mechanisms; Reformer: Employs Locality Sensitive Hash (LSH) for attention optimization to improve efficiency; Transformer: The classic Transformer architecture, used as the baseline model; ETSformer: Combines exponential smoothing and Transformer to improve model stability; MICN: Multi-input Convolutional Network, suitable for multimodal signal modeling; Pyraformer: A pyramid-shaped attention structure that emphasizes modeling capabilities at different time scales; Crossformer: Models information across time scales, focusing on temporal structural changes; iTransformer: A simplified Transformer model that strives for extreme lightweight design; SegRNN: A recurrent structure based on sequence segmentation, emphasizing segmented feature extraction.

[0096] Furthermore, the 14th model is the method proposed in this invention. Comparative experimental results are as follows: Figure 6 As shown, each dataset has four load cases, with three load cases used as the training set and the remaining load case as the test set. Taking the T0 transfer task of the SCARA dataset as an example, load cases of 3kg, 6kg, and 9kg are used as the training set, and 0kg is used as the test set. Figure 6 As can be seen, the method proposed in this invention exhibits stable and excellent performance under all operating conditions in both datasets, achieving the highest average accuracy (95.618%) among all models, significantly outperforming the 13 mainstream models compared. This invention comprehensively introduces a cross-modal time-frequency sensing embedding module and a time-frequency convergence fusion Transformer module, achieving high-precision, multi-scenario adaptable fault diagnosis of rotating machinery through multi-modal information adaptive extraction and joint optimization mechanisms.

[0097] To further verify the contribution of each module in this invention to the final fault diagnosis performance, this embodiment also includes the following: Figure 7 The ablation experiments shown are evaluated respectively: Ablation 1: "Temporal branch only"; Ablation 2: "Frequency Domain Branch Only"; Ablation 3: "Time-frequency domain branch only (no convergence)"; Ablation 4: "Time-frequency domain branch only (with convergence)"; Ablation 5: "Time + Frequency + Time-Frequency Branch (No Convergence)"; The complete model of this invention includes six structural configurations: "time + frequency + time-frequency branch (with convergence)". Specifically, whether or not there is convergence refers to whether a time-frequency convergence multi-head cross-attention module is introduced.

[0098] Experimental results on two multimodal fault diagnosis datasets, SCARA robot and PU bearing, show that the single-branch model is unstable and has a low average accuracy (e.g., only 0.777914 for the frequency domain branch alone). Although there is a significant improvement after fusing the time and frequency domains, there is still a performance bottleneck when there is no joint optimization of the time and frequency domain branches (average accuracy is 0.908375). After further introducing "time and frequency domain branches", the model performance is significantly improved regardless of whether they are converged. The average accuracy of the complete structure (with convergence) reaches the highest of 0.95618. The above results fully demonstrate that: (1) the time and frequency domain fusion branch can effectively supplement the common features that cannot be captured by time / frequency private information; (2) the three-branch convergence mechanism helps to improve the model's discrimination ability and robustness; (3) the overall structure proposed in this invention has significant advantages in complex multimodal scenarios.

[0099] Based on the above-mentioned rotating machinery fault diagnosis method based on cross-modal time-frequency sensing, the present invention also provides a rotating machinery fault diagnosis system based on cross-modal time-frequency sensing, comprising: The time-domain embedding module is used to input multimodal time-domain signals into the time-domain sensing embedding branch, which then passes through the time-domain signal sensing module and the first nonlinear activation mapping module to obtain the time-domain activation mapping feature layer; the time-domain activation mapping features are then linearly mapped to obtain the time-domain embedding features. The frequency domain embedding module is used to input multimodal frequency domain signals into the frequency domain sensing embedding branch, which then passes through the frequency domain signal sensing module and the second nonlinear activation mapping module to obtain the frequency domain activation mapping feature layer. The frequency domain activation mapping features are then linearly mapped to obtain the frequency domain embedding features. The time-frequency domain embedding module is used to input the time-domain activation mapping features and the frequency-domain activation mapping features into the time-frequency domain perceptual embedding branch to obtain the time-frequency domain embedding features. The temporal feature extraction module is used to input temporal embedded features into the temporal Transformer private branch containing the temporal multi-head self-attention module to obtain temporal predicted classification labels. The frequency domain feature extraction module is used to input frequency domain embedded features into the frequency domain Transformer private branch containing the frequency domain multi-head self-attention module to obtain frequency domain predicted classification labels. The time-frequency domain feature extraction module is used to input the time-frequency domain embedded features into the common branch of the time-frequency domain Transformer containing the time-frequency convergence multi-head cross attention module, and obtain the time-frequency domain predicted classification label.

[0100] In summary, the rotating machinery fault diagnosis method based on cross-modal time-frequency sensing described in this invention designs time-domain, frequency-domain, and time-frequency-domain sensing embedding branches. This enables adaptive processing of cross-modal time-domain, frequency-domain, and time-frequency-domain signals from different devices, achieving effective fusion of multi-modal signals. It provides comprehensive, multi-angle sensing of multi-modal signals, generating unified time-domain, frequency-domain, and time-frequency-domain embedded features, providing high-quality feature representations for subsequent fault diagnosis. This cross-modal sensing embedding branch overcomes the limitation of existing embedding methods that can only process specific signal modes, exhibiting high versatility and wide applicability to different fault diagnosis models, improving the accuracy and robustness of fault diagnosis. Furthermore, this invention designs corresponding Transformer branches for the embedded features in the time-domain, frequency-domain, and time-frequency-domain, respectively, for the private feature attributes in the time and frequency domains and the public feature attributes in the time-frequency domain. It also utilizes a time-frequency convergence multi-head cross-attention module to aggregate global time-frequency-domain feature representations from local feature representations in the time and frequency domains. The multiple Transformer branches designed in this invention consider both private and public feature attributes of time-frequency information, which can fully explore the commonalities and differences between the time domain and the frequency domain, thereby improving the accuracy and robustness of fault diagnosis.

[0101] This invention addresses key issues in traditional fault diagnosis methods, resulting in significant benefits at multiple levels, including social, economic, and technological advantages.

[0102] 1. Social benefits: Improving equipment safety: This invention can effectively reduce production and safety accidents caused by equipment failures by accurately and in real time diagnosing faults in rotating machinery, thus ensuring the stability of the production line and the safety of employees. Promoting the development of intelligent manufacturing and intelligent industry: The innovative technology of this invention can support the fault diagnosis module in intelligent manufacturing systems, improve overall production efficiency and automation level, and make a positive contribution to promoting the progress of intelligent factories and intelligent industry.

[0103] 2. Economic benefits: Reduce equipment maintenance costs: By detecting and predicting faults in the early stages, this invention can reduce the incidence of equipment failures, reduce production stoppages caused by equipment downtime and maintenance, and significantly reduce maintenance costs and production losses. Improve production efficiency: Accurate fault diagnosis can shorten maintenance time and reduce production line downtime, thereby improving overall production efficiency and output value, bringing significant economic benefits; Promoting the widespread application of equipment and technology: This invention is applicable to various rotating machinery and equipment, which can not only improve the diagnostic level of traditional equipment, but also promote the widespread application and commercialization of new intelligent equipment, bringing significant economic returns to the manufacturing industry.

[0104] 3. Technical benefits: Improving diagnostic accuracy and robustness: Through the cross-modal time-frequency sensing embedding module, this invention can adaptively process signals from different devices and different modalities, achieving effective fusion of multimodal signals and improving the accuracy and robustness of fault diagnosis. The time-frequency convergence fusion Transformer can further optimize feature learning through a multi-head self-attention mechanism, enhancing the model's diagnostic capabilities in complex fault signals; Improving the model's versatility and adaptability: This invention, through deep fusion of time-domain, frequency-domain, and time-frequency-domain signals, can cope with signal variations in different devices and fault types, improving the adaptability and universality of the diagnostic model. Compared with traditional methods, the model of this invention can not only be effectively applied to existing devices but also quickly adapt to new devices or new signal patterns, possessing strong scalability. Innovative Time-Frequency Information Fusion Method: The cross-modal time-frequency sensing embedding module and time-frequency convergence fusion Transformer module proposed in this invention provide a brand-new approach for the fusion and optimization of time-frequency domain features, significantly improving the comprehensive processing capability of multimodal signals in fault diagnosis, and promoting the application and development of multimodal deep learning in industrial diagnosis.

[0105] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0106] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0107] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0108] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0109] Obviously, the above embodiments are merely illustrative examples for clear explanation and are not intended to limit the implementation. Those skilled in the art will recognize that other variations or modifications can be made based on the above description. It is neither necessary nor possible to exhaustively list all possible implementations here. However, obvious variations or modifications derived therefrom are still within the scope of protection of this invention.

Claims

1. A method for fault diagnosis of rotating machinery based on cross-modal time-frequency sensing, characterized in that, include: The multimodal time-domain signal is input into the time-domain sensing embedding branch, and then passes through the time-domain signal sensing module and the first nonlinear activation mapping module in sequence to obtain the time-domain activation mapping features; The temporal activation mapping features are passed through a linear mapping layer to obtain temporal embedding features; The multimodal frequency domain signal is input into the frequency domain sensing embedding branch, and then passes through the frequency domain signal sensing module and the second nonlinear activation mapping module in sequence to obtain the frequency domain activation mapping features. The frequency domain activation mapping features are passed through a linear mapping layer to obtain the frequency domain embedding features; Input the time-domain activation mapping features and the frequency-domain activation mapping features into the time-frequency domain perceptual embedding branch to obtain the time-frequency domain embedding features; Input the temporal embedded features into the temporal Transformer private branch containing the temporal multi-head self-attention module to obtain the temporal predicted classification label; Input the frequency domain embedded features into the frequency domain Transformer private branch containing the frequency domain multi-head self-attention module to obtain the frequency domain predicted classification label; By inputting the time-frequency domain embedded features into the common branch of the time-frequency domain Transformer, which includes the time-frequency convergence multi-head cross-attention module, the predicted classification label in the time-frequency domain is obtained.

2. The rotating machinery fault diagnosis method based on cross-modal time-frequency sensing according to claim 1, characterized in that, Both the time-domain signal sensing module and the frequency-domain signal sensing module include multiple sensing cores; the length and number of channels of the sensing core are smaller than the length and number of channels of the input signal, respectively, and each sensing core has a different channel start index value and sequence start index value.

3. The rotating machinery fault diagnosis method based on cross-modal time-frequency sensing according to claim 2, characterized in that, The data processing procedures for both the time-domain signal sensing module and the frequency-domain signal sensing module include: The multiple sensing kernels of the current signal sensing module act on different sub-regions of the input signal of the current signal sensing module to obtain the sensing features corresponding to each sensing kernel; Based on the perceptual features corresponding to all perceptual kernels, the total perceptual features are obtained; The total sensed features are flattened to obtain the output features of the current signal sensing module.

4. The rotating machinery fault diagnosis method based on cross-modal time-frequency sensing according to claim 1, characterized in that, Both the first and second nonlinear activation mapping modules include a linear mapping layer and a SiLU activation function connected in sequence.

5. The rotating machinery fault diagnosis method based on cross-modal time-frequency sensing according to claim 1, characterized in that, The time-domain activation mapping features and frequency-domain activation mapping features are input into the time-frequency domain perceptual embedding branch to obtain the time-frequency domain embedding features. The process includes: The time-domain activation mapping features and the frequency-domain activation mapping features are concatenated to obtain the time-frequency fusion features; The time-frequency fusion features are sequentially passed through a linear mapping layer, a SiLU activation function, and another linear mapping layer to obtain the time-frequency domain embedded features.

6. The rotating machinery fault diagnosis method based on cross-modal time-frequency sensing according to claim 1, characterized in that, Both the time-domain multi-head self-attention module and the frequency-domain multi-head self-attention module include a self-attention module, a residual connection and layer normalization operation, a feedforward network, and a residual connection and layer normalization operation connected in sequence.

7. The rotating machinery fault diagnosis method based on cross-modal time-frequency sensing according to claim 1, characterized in that, The time-frequency convergence multi-head cross-attention module includes a cross-attention module, a convergence fusion layer, a linear mapping layer, a residual connection and layer normalization operation, a feedforward network layer, and a residual connection and layer normalization operation connected in sequence; the cross-attention module includes a time-domain-time-frequency domain cross-attention module, a time-frequency domain self-attention module, and a frequency domain-time-frequency domain cross-attention module connected in parallel.

8. The rotating machinery fault diagnosis method based on cross-modal time-frequency sensing according to claim 7, characterized in that, The data processing procedure of the time-frequency convergence multi-head cross-attention module includes: The time-frequency domain embedded features are passed through the time-frequency domain cross-attention module, the time-frequency domain self-attention module, and the frequency domain-time-frequency domain cross-attention module, respectively, to obtain the time-frequency domain cross-attention features, the time-frequency domain self-attention features, and the frequency domain-time-frequency domain cross-attention features. The cross-attention features between the time domain and the time-frequency domain, the self-attention features in the time-frequency domain, and the cross-attention features between the frequency domain and the time-frequency domain are input into the convergence and fusion layer and spliced ​​together to obtain the fused attention features; The fused attention features are passed through a linear mapping layer, and their output features and time-frequency domain embedded features are passed through residual connections and layer normalization operations to obtain the input features of the feedforward network layer. The input features of the feedforward network layer are passed through the feedforward network layer to obtain the output features of the feedforward network layer. The input features and output features of the feedforward network layer are then passed through residual connections and layer normalization operations to obtain the output features of the time-frequency convergence multi-head cross-attention module.

9. A method for diagnosing rotating machinery faults based on cross-modal time-frequency sensing according to claim 7, characterized in that, The data processing procedure of the cross-attention module includes: The time-frequency domain embedded features are linearly mapped to obtain time-frequency query, time-frequency key, and time-frequency value; Obtain the time-domain query and time-domain key; obtain the frequency-domain query and frequency-domain key; the time-domain query and time-domain key are obtained by linear mapping from the time-domain embedding features; the frequency-domain query and frequency-domain key are obtained by linear mapping from the frequency-domain embedding features. The time-domain query, frequency-domain key, and time-frequency value are passed through a time-domain-time-frequency-domain cross-attention module to obtain the time-domain-time-frequency-domain cross-attention feature; The time-frequency query, time-frequency key, and time-frequency value are passed through a time-frequency domain self-attention module to obtain the time-frequency domain self-attention features; The frequency domain query, time domain key, and time-frequency value are passed through a frequency domain-time-frequency domain cross-attention module to obtain the frequency domain-time-frequency domain cross-attention feature.

10. A rotating machinery fault diagnosis system based on cross-modal time-frequency sensing, characterized in that, include: The time-domain embedding module is used to input multimodal time-domain signals into the time-domain sensing embedding branch, which then passes through the time-domain signal sensing module and the first nonlinear activation mapping module to obtain time-domain activation mapping features. The temporal activation mapping features are passed through a linear mapping layer to obtain temporal embedding features; The frequency domain embedding module is used to input multimodal frequency domain signals into the frequency domain sensing embedding branch, which then passes through the frequency domain signal sensing module and the second nonlinear activation mapping module to obtain frequency domain activation mapping features. The frequency domain activation mapping features are passed through a linear mapping layer to obtain the frequency domain embedding features; The time-frequency domain embedding module is used to input the time-domain activation mapping features and the frequency-domain activation mapping features into the time-frequency domain perceptual embedding branch to obtain the time-frequency domain embedding features. The temporal feature extraction module is used to input temporal embedded features into the temporal Transformer private branch containing the temporal multi-head self-attention module to obtain temporal predicted classification labels. The frequency domain feature extraction module is used to input frequency domain embedded features into the frequency domain Transformer private branch containing the frequency domain multi-head self-attention module to obtain frequency domain predicted classification labels. The time-frequency domain feature extraction module is used to input the time-frequency domain embedded features into the common branch of the time-frequency domain Transformer containing the time-frequency convergence multi-head cross attention module, and obtain the time-frequency domain predicted classification label.

Images