Electric power communication equipment fault diagnosis method and system based on multi-modal fusion adversarial learning

Abstract

The invention discloses an electric power communication equipment fault diagnosis method and system based on multi-modal fusion adversarial learning, and the method comprises the steps: obtaining time sequence reference signals containing radio frequency front-end fault information in normal and abnormal environments, obtaining a time-frequency spectrogram through short-time Fourier transform, constructing a multi-modal data set, and carrying out the preprocessing; respectively extracting a signal time domain feature and a time spectrogram frequency domain feature through a double-branch convolutional network; performing dynamic weighted fusion on the time domain features and the frequency domain features by using an attention fusion module to generate multi-modal fusion features; constructing a dual-task branch comprising a fault classifier and an environment discriminator, identifying a fault type by using the fault classifier, and predicting an environment type by using the environment discriminator; through multi-modal fusion adversarial learning, non-intrusive accurate diagnosis of the radio frequency front-end fault of the electric power communication equipment is realized, the problems of physical inaccessibility, characteristic mode drift and characteristic confusion of a traditional method are effectively solved, and stable diagnosis performance can still be kept in a complex environment.

Description

Technical Field

[0001] This invention relates to the field of fault diagnosis technology for power communication equipment, and in particular to a fault diagnosis method and system for power communication equipment based on multimodal fusion adversarial learning. Background Technology

[0002] In power grid wireless transmission, massive MIMO is the core of 5G base stations, and its stability is crucial for the reliable transmission of power grid dispatch commands. The radio frequency (RF) front-end is a key component of communication equipment in MIMO communication, and any damage to it will severely affect communication quality. Therefore, monitoring damage to the RF front-end of communication equipment and accurately identifying faults through base station receivers is of great significance for ensuring the continuity of power grid wireless transmission.

[0003] Traditional fault diagnosis methods for power communication equipment have significant physical limitations. One type is physical model-based testing technology, which requires obtaining key parameters through preset internal test nodes. However, the highly integrated 5G base station equipment drastically reduces the accessibility of physical nodes. The other type is online monitoring solutions using external test equipment, which suffer from complex deployment and high costs, making it difficult to meet the continuous monitoring needs of the power grid. Furthermore, both types of methods require intrusive access to the equipment, which cannot meet the non-intrusive diagnostic requirements of the complex outdoor environment of the power system.

[0004] While non-intrusive methods based on receiver reference signals circumvent physical access limitations, they face a core challenge: because they are essentially a superposition of fault characteristics and channel interference, they can cause mode drift and feature confusion. Existing data-driven methods have failed to effectively address environmental interference issues, resulting in insufficient diagnostic robustness. Summary of the Invention

[0005] In view of this, the purpose of the present invention is to provide a fault diagnosis method and system for power communication equipment based on multimodal fusion adversarial learning. The method uses a multimodal feature extractor to predict multimodal fusion features.

[0006] To achieve the above objectives, the present invention provides the following technical solution: The fault diagnosis method for power communication equipment based on multimodal fusion adversarial learning provided by this invention includes the following steps: S1. Obtain the time-series reference signals containing radio frequency front-end fault information in power communication equipment under normal and abnormal environments respectively, and obtain the time spectrum diagram through short-time Fourier transform to construct a multi-modal dataset and perform preprocessing. S2, construct a dual-branch convolutional network to extract the time-domain features and frequency-domain features of the time-spectrum of the reference signal respectively; S3, which uses a cross-modal attention fusion module to fuse time-domain features and frequency-domain features into multimodal fusion features to construct a multimodal feature extractor; S4, construct a dual-task branch including a fault classifier and an environment discriminator, and input the multimodal fusion features into the fault classifier to predict the fault category and the environment discriminator to predict the environment category respectively; S5 sets up a gradient inversion layer between the multimodal feature extractor and the environment discriminator, and uses a dynamic adversarial learning mechanism to train the model, enabling the model to learn robust fault features that are independent of the environment.

[0007] Furthermore, the timing reference signal in step S1 is a detection reference signal sent from the power communication equipment to the base station, which includes in-phase components and quadrature components.

[0008] Furthermore, the construction and preprocessing of the multimodal dataset in step S1 includes the following steps: S11: Extract the effective segment of the timing reference signal and compress it through a preset window, then normalize the amplitude range through a linear transformation. S12 converts the time-spectrum image into a single-channel grayscale image and adjusts the resolution using bilinear interpolation. S13. The multimodal dataset is divided into training set, validation set and test set according to a preset ratio using a stratified sampling strategy.

[0009] Furthermore, the dual-branch convolutional network in step S2 includes: The first branch is a temporal feature extractor: the input is a temporal reference signal, including a dual-channel reference signal with in-phase and quadrature components. Its structure consists of two one-dimensional convolutional modules and an adaptive average pooling layer connected in sequence. The one-dimensional convolutional module includes a one-dimensional convolutional layer, batch normalization, ReLU activation, Dropout regularization, and max pooling layer. The second branch is the frequency domain feature extractor: the input is a time-domain spectrogram, including a dual-channel time-domain spectrogram of in-phase and quadrature components. Its structure consists of two two-dimensional convolutional modules and an adaptive average pooling layer connected in sequence. The two-dimensional convolutional module includes a two-dimensional convolutional layer, batch normalization, ReLU activation, Dropout regularization, and max pooling layer.

[0010] Furthermore, in step S3, the cross-modal attention fusion module is used to fuse temporal and frequency domain features to obtain multimodal fusion features, which is specifically performed according to the following steps: S31, time domain features and frequency domain features Projecting onto the common feature space, an attention weight matrix is ​​generated: in This represents the Sigmoid activation function. and For learnable parameter matrix, and These represent the time-domain attention weight matrix and the frequency-domain attention weight matrix, respectively. S32, the generated weighted attention matrix is ​​used to perform weighted fusion of the projected features to obtain multimodal fusion features. : in Indicates multimodal fusion features, and Let represent the learnable parameter matrices used for projection. This indicates element-wise multiplication.

[0011] Furthermore, in step S4, the fault classifier distinguishes the RF front-end fault types of 5G communication equipment, including nonlinear power amplifier distortion, I / Q imbalance, and phase noise; the environment discriminator distinguishes the environment types, mainly including the normal environment and abnormal environment of the area where the 5G communication equipment is located.

[0012] Furthermore, the total loss function of the dynamic adversarial learning mechanism described in step S5 is: in Indicates the total loss. Indicates the parameters of the multimodal feature extractor. Indicates the parameters of the fault classifier. Indicates the parameters of the environment classifier; The cross-entropy loss is calculated using only samples with fault classification labels under normal conditions. The environmental classification loss is represented by the cross-entropy loss calculated using samples from normal and abnormal environments in the entire multimodal dataset. The dynamic resistance coefficient is represented by the following formula: ; in Indicates training progress. Indicates use for control Hyperparameters of curve rate.

[0013] Furthermore, in step S5, the gradient reversal layer maintains multimodal fusion characteristics during forward propagation. The classification loss of the environment discriminator remains unchanged during backpropagation. Relative to multimodal feature extractor parameters gradient multiplied by The calculation formula is: Where GRL represents the gradient inversion layer. This represents the dynamic resistance coefficient.

[0014] The fault diagnosis system for power communication equipment based on multimodal fusion adversarial learning provided by the present invention includes a memory, a processor, and a computer program stored in the memory and capable of running on the processor. When the processor executes the program, it implements the above-mentioned method.

[0015] The beneficial effects of this invention are as follows: This invention provides a fault diagnosis method and system for power communication equipment based on multimodal fusion adversarial learning. The method first acquires time-series reference signals containing radio frequency front-end fault information from power communication equipment under normal and abnormal environments, respectively, and obtains time-spectrum maps through short-time Fourier transform, constructing a multimodal dataset and performing preprocessing. Then, a dual-branch convolutional network is constructed to extract the time-domain features of the reference signals and the frequency-domain features of the time-spectrum maps. The time-domain and frequency-domain features are fused into multimodal fusion features using a cross-modal attention fusion module to construct a multimodal feature extractor. Finally, a dual-task branch including a fault classifier and an environment discriminator is constructed, with the multimodal fusion features input to the fault classifier to predict the fault category and the environment discriminator to predict the environment category, respectively. A gradient inversion layer is set between the multimodal feature extractor and the environment discriminator, and a dynamic adversarial learning mechanism is used to train the model, enabling the model to learn robust fault features independent of the environment. This invention achieves non-invasive and accurate diagnosis of radio frequency front-end faults in power communication equipment through multimodal fusion adversarial learning, effectively solving the problems of physical inaccessibility, feature mode drift, and feature confusion in traditional methods, and maintaining stable diagnostic performance even in complex environments.

[0016] This method does not rely on pre-set test nodes inside the device or external test equipment. Fault diagnosis can be achieved through the reference signal at the receiving end. It adapts to the problem of reduced physical accessibility caused by the high integration of power communication equipment and meets the needs of non-intrusive, remote monitoring in complex outdoor environments of power systems.

[0017] This method extracts the time-domain and frequency-domain features of the reference signal through a dual-branch convolutional network. Combined with a cross-modal attention fusion module, it dynamically weighs the contribution of different modal features to fault diagnosis, achieving complementary fusion of transient features in the time domain and steady-state spectral features in the frequency domain, thereby enhancing the comprehensiveness and discriminativeness of fault features.

[0018] This method constructs an adversarial relationship between a multimodal fusion feature extractor and an environment discriminator through a gradient inversion layer, forcing the model to learn robust fault features that are independent of environmental changes. This effectively solves the problems of mode drift and feature confusion in traditional methods, ensuring stable diagnostic performance even in complex wireless transmission environments.

[0019] Other advantages, objectives, and features of the invention will be set forth in part in the description which follows, and in part will be apparent to those skilled in the art from the following examination, or may be learned from practice of the invention. The objectives and other advantages of the invention can be realized and obtained through the following description. Attached Figure Description

[0020] To make the objectives, technical solutions, and beneficial effects of this invention clearer, the following drawings are provided for illustration.

[0021] Figure 1 This is a simplified flowchart of the method.

[0022] Figure 2 This is a diagram of the model structure of this method.

[0023] Figure 3 The training and validation loss curves are shown for the multimodal fusion adversarial learning model of this method.

[0024] Figure 4 This is the overall confusion matrix of the proposed method model on the test set.

[0025] Figure 5 This is a comparison chart of the visualization of principal component analysis of multimodal fusion adversarial learning features and the visualization of principal component analysis of multimodal fusion features in this method.

[0026] Figure 6 This is a comparison chart of the accuracy of different models under normal and abnormal environments. Detailed Implementation

[0027] The present invention will be further described below with reference to the accompanying drawings and specific embodiments, so that those skilled in the art can better understand and implement the present invention. However, the embodiments described are not intended to limit the present invention.

[0028] refer to Figure 1 and Figure 2 , Figure 1 This is a simplified flowchart of the method of the present invention. Figure 2 This is a model structure diagram of the method. This invention discloses a fault diagnosis method for power communication equipment based on multimodal fusion adversarial learning, comprising the following steps: S1: Obtain the time-series reference signals containing radio frequency front-end fault information in power communication equipment under normal and abnormal environments, respectively, and obtain the time-spectrum diagram through short-time Fourier transform to construct a multi-modal dataset and perform preprocessing.

[0029] It should be noted that the dataset in this embodiment is built based on MIMO-OFDM system-level simulation, and all samples are generated strictly in accordance with the 5G NR specification. The system uses a 4×4 planar antenna array, with a base station height of 15 meters, a communication equipment terminal height of 1.5 meters, a transmission distance on the order of 50 meters, a center operating frequency of 2.3 GHz, a system bandwidth of 100 MHz, and uses a 3GPP channel model to characterize the wireless transmission characteristics of the power grid. Each sample generation process is based on a complete communication link simulation, specifically including the following five steps: A probe reference signal is generated at the transmitting end of the communication equipment and mapped onto a resource grid as an analysis carrier. Three types of faults—nonlinear power amplification, I / Q imbalance, and phase noise—are introduced at the transmitting end of communication equipment, and four fault label types are generated by combining them with normal conditions. The fading characteristics of wireless propagation are simulated based on the 3GPP model, and the multipath channel coefficients under this simulated environment are generated. After the signal is transmitted and received by the multi-antenna system, it is superimposed with channel fading and different environmental noise. The sample with a signal-to-noise ratio between 22 and 28 dB is defined as the sample under normal environment, and the sample with a signal-to-noise ratio between 5 and 15 dB is defined as the sample under abnormal environment. Receive timing reference signals including in-phase and quadrature components.

[0030] After the above stages, two types of datasets are generated. One type consists of 2,000 samples with fault labels under normal conditions; the other type is a mixed dataset, which contains 2,000 labeled samples under normal conditions and 2,000 unlabeled samples under abnormal conditions, used to simulate scenarios where fault labels are difficult to obtain under abnormal conditions in reality.

[0031] The power communication device described in this embodiment is a 5G power communication device.

[0032] It should be noted that in S1, the timing reference signal is the detection reference signal sent by the power communication equipment to the base station, which includes in-phase components and quadrature components; It should be noted that in S1, the short-time Fourier transform uses a 1024-point Hanning window to process the in-phase and quadrature components of the reference signal, and obtains the time spectrum diagrams of the in-phase and quadrature components respectively. It should be noted that in S1, constructing and preprocessing the multimodal dataset includes the following steps: In the multimodal dataset, samples under normal conditions contain fault labels, while samples under abnormal conditions do not contain fault labels. The effective segment of the time-series reference signal is extracted and compressed to 2000 points by taking the maximum value within a 4-point window. The amplitude range is normalized to [-1, 1] through linear transformation. The time-spectrum is converted into a single-channel grayscale image, and the resolution is adjusted to [-1, 1] through bilinear interpolation. ; The multimodal dataset was divided into training, validation, and test sets in a 6:2:2 ratio using a stratified sampling strategy.

[0033] S2, construct a dual-branch convolutional network to extract the time-domain features and frequency-domain features of the time-spectrum of the reference signal respectively; It should be noted that in S2, the dual-branch convolutional network specifically includes: The first branch is the temporal feature extractor: its input is a dual-channel reference signal including in-phase and quadrature components. Its structure consists of two one-dimensional convolutional modules and an adaptive average pooling layer. The one-dimensional convolutional module includes a one-dimensional convolutional layer, batch normalization, ReLU activation, Dropout regularization, and max pooling layer. The second branch is the frequency domain feature extractor: its input is a two-channel time-spectrum map including in-phase and quadrature components. Its structure consists of two two-dimensional convolutional modules and an adaptive average pooling layer. The two-dimensional convolutional modules include two-dimensional convolutional layers, batch normalization, ReLU activation, Dropout regularization, and max pooling layers.

[0034] S3, which uses a cross-modal attention fusion module to fuse time-domain features and frequency-domain features into multimodal fusion features to construct a multimodal feature extractor; It should be noted that in S3, the process of fusing temporal and frequency domain features using the cross-modal attention fusion module to obtain multimodal fused features includes the following steps: Time domain features and frequency domain features Projecting onto the common feature space, an attention weight matrix is ​​generated: in This represents the Sigmoid activation function. and For learnable parameter matrix, and These represent the time-domain attention weight matrix and the frequency-domain attention weight matrix, respectively. The generated weighted attention matrix is ​​used to perform weighted fusion of the projected features to obtain multimodal fusion features. : in Indicates multimodal fusion features, and Let represent the learnable parameter matrices used for projection. This indicates element-wise multiplication.

[0035] S4, construct a dual-task branch including a fault classifier and an environment discriminator, and input the multimodal fusion features into the fault classifier to predict the fault category and the environment discriminator to predict the environment category respectively; It should be noted that in S4, the fault classifier distinguishes the types of RF front-end faults of 5G communication equipment, including nonlinear power amplifier distortion, I / Q imbalance, and phase noise; the environment discriminator distinguishes the environmental types, mainly including the normal environment and abnormal environment of the area where the 5G communication equipment is located.

[0036] S5 sets up a gradient inversion layer between the multimodal feature extractor and the environment discriminator, and uses a dynamic adversarial learning mechanism to train the model, so that the model learns robust fault features that are independent of the environment. It should be noted that in S5, the total loss function of the dynamic adversarial learning mechanism is: in Indicates the total loss. Indicates the parameters of the multimodal feature extractor. Indicates the parameters of the fault classifier. Indicates the parameters of the environment classifier; The cross-entropy loss is calculated using only samples with fault classification labels under normal conditions. The environmental classification loss is represented by the cross-entropy loss calculated using samples from normal and abnormal environments in the entire multimodal dataset. The dynamic resistance coefficient is represented by the following formula: ; in Indicates training progress. Indicates use for control Hyperparameters of curve rate; It should be noted that the hyperparameters used during model training include a total of 100 training epochs, an initial learning rate of 1e-4, a cosine annealing scheduling strategy, the Adam optimizer, and a Dropout probability of 0.2. Set to 10, for reference. Figure 3 The figure shows the phased changes in the training and validation loss curves of the model, reflecting the dynamic equilibrium of adversarial learning: initially, the total loss decreases rapidly, focusing on fault classification; in the middle stage, it fluctuates and adjusts, forming an adversarial balance; in the later stage, it tends to stabilize, with no overfitting and good generalization ability. The model was tested on a test set, and the confusion matrix after testing is shown in the figure. Figure 4 , Figure 3 The image shows the classification results of samples of each category on the test set.

[0037] refer to Figure 5 , Figure 5 This is a comparison chart of the principal component analysis (PCA) visualizations of the multimodal fusion adversarial learning features and the multimodal fusion features. Based on this chart, we can compare their distribution in the feature space. Compared to the multimodal features, the overlapping regions of the feature distributions of the multimodal fusion adversarial learning features are significantly shrunk in both environments. This indicates that the adaptive mechanism of adversarial learning forces the fused features to focus more on the essential patterns of the fault, rather than environmental interference, by maximizing the environmental classification loss.

[0038] refer to Figure 6 , Figure 6 The figure shows a comparison of the accuracy of different models under normal and abnormal environments. According to the figure, it can be seen that the multimodal fusion feature adopted in this method can achieve complementary fusion of transient features in the time domain and steady-state spectral features in the frequency domain, which enhances the comprehensiveness and discriminativeness of fault features. At the same time, by using the dynamic adversarial learning mechanism, it can effectively resist the mode drift and feature confusion caused by environmental anomalies, and ensure that stable diagnostic performance can still be maintained in complex wireless transmission environments.

[0039] It should be noted that in S5, the gradient reversal layer maintains the multimodal fusion characteristics during forward propagation. The classification loss of the environment discriminator remains unchanged during backpropagation. Relative to multimodal feature extractor parameters gradient multiplied by The calculation formula is: Where GRL represents the gradient inversion layer. This represents the dynamic resistance coefficient.

[0040] The power communication equipment fault diagnosis system based on multimodal fusion adversarial learning provided in this embodiment includes a memory, a processor, and a computer program stored in the memory and capable of running on the processor. When the processor executes the program, it implements the above-mentioned method.

[0041] The above-described embodiments are merely preferred embodiments provided to fully illustrate the present invention, and the scope of protection of the present invention is not limited thereto. Equivalent substitutions or modifications made by those skilled in the art based on the present invention are all within the scope of protection of the present invention. The scope of protection of the present invention is defined by the claims.

Claims

1. A fault diagnosis method for power communication equipment based on multimodal fusion adversarial learning, characterized in that: Includes the following steps: S1. Obtain the time-series reference signals containing radio frequency front-end fault information in power communication equipment under normal and abnormal environments respectively, and obtain the time spectrum diagram through short-time Fourier transform to construct a multi-modal dataset and perform preprocessing. S2, construct a dual-branch convolutional network to extract the time-domain features and frequency-domain features of the time-spectrum of the reference signal respectively; S3, which uses a cross-modal attention fusion module to fuse time-domain features and frequency-domain features into multimodal fusion features to construct a multimodal feature extractor; S4, construct a dual-task branch including a fault classifier and an environment discriminator, and input the multimodal fusion features into the fault classifier to predict the fault category and the environment discriminator to predict the environment category respectively; S5 sets up a gradient inversion layer between the multimodal feature extractor and the environment discriminator, and uses a dynamic adversarial learning mechanism to train the model, enabling the model to learn robust fault features that are independent of the environment.

2. The fault diagnosis method for power communication equipment based on multimodal fusion adversarial learning according to claim 1, characterized in that: In step S1, the timing reference signal is a detection reference signal sent from the power communication equipment to the base station, which includes in-phase components and quadrature components.

3. The fault diagnosis method for power communication equipment based on multimodal fusion adversarial learning according to claim 1, characterized in that: The construction and preprocessing of the multimodal dataset in step S1 includes the following steps: S11: Extract the effective segment of the timing reference signal and compress it through a preset window, then normalize the amplitude range through a linear transformation. S12 converts the time-spectrum image into a single-channel grayscale image and adjusts the resolution using bilinear interpolation. S13. The multimodal dataset is divided into training set, validation set and test set according to a preset ratio using a stratified sampling strategy.

4. The fault diagnosis method for power communication equipment based on multimodal fusion adversarial learning according to claim 1, characterized in that: The dual-branch convolutional network in step S2 includes: The first branch is a temporal feature extractor: the input is a temporal reference signal, including a dual-channel reference signal with in-phase and quadrature components. Its structure consists of two one-dimensional convolutional modules and an adaptive average pooling layer connected in sequence. The one-dimensional convolutional module includes a one-dimensional convolutional layer, batch normalization, ReLU activation, Dropout regularization, and max pooling layer. The second branch is the frequency domain feature extractor: the input is a time-domain spectrogram, including a dual-channel time-domain spectrogram of in-phase and quadrature components. Its structure consists of two two-dimensional convolutional modules and an adaptive average pooling layer connected in sequence. The two-dimensional convolutional module includes a two-dimensional convolutional layer, batch normalization, ReLU activation, Dropout regularization, and max pooling layer.

5. The fault diagnosis method for power communication equipment based on multimodal fusion adversarial learning according to claim 1, characterized in that: In step S3, the cross-modal attention fusion module is used to fuse temporal and frequency domain features to obtain multimodal fusion features, which is specifically carried out according to the following steps: S31, time domain features and frequency domain features Projecting onto the common feature space, an attention weight matrix is ​​generated: in This represents the Sigmoid activation function. and For learnable parameter matrix, and These represent the time-domain attention weight matrix and the frequency-domain attention weight matrix, respectively. S32, the generated weighted attention matrix is ​​used to perform weighted fusion of the projected features to obtain multimodal fusion features. : in Indicates multimodal fusion features, and Let represent the learnable parameter matrices used for projection. This indicates element-wise multiplication.

6. The fault diagnosis method for power communication equipment based on multimodal fusion adversarial learning according to claim 1, characterized in that: In step S4, the fault classifier distinguishes the types of RF front-end faults of 5G communication equipment, including nonlinear power amplifier distortion, I / Q imbalance, and phase noise; the environment discriminator distinguishes the environmental types, mainly including the normal environment and abnormal environment of the area where the 5G communication equipment is located.

7. The fault diagnosis method for power communication equipment based on multimodal fusion adversarial learning according to claim 1, characterized in that: The total loss function of the dynamic adversarial learning mechanism described in step S5 is: in Indicates the total loss. Indicates the parameters of the multimodal feature extractor. Indicates the parameters of the fault classifier. Indicates the parameters of the environment classifier; The cross-entropy loss is calculated using only samples with fault classification labels under normal conditions. The environmental classification loss is represented by the cross-entropy loss calculated using samples from normal and abnormal environments in the entire multimodal dataset. The dynamic resistance coefficient is represented by the following formula: ; in Indicates training progress. Indicates use for control Hyperparameters of curve rate.

8. The fault diagnosis method for power communication equipment based on multimodal fusion adversarial learning according to claim 1, characterized in that: In step S5, the gradient inversion layer maintains multimodal fusion characteristics during forward propagation. The classification loss of the environment discriminator remains unchanged during backpropagation. Relative to multimodal feature extractor parameters gradient multiplied by The calculation formula is: Where GRL represents the gradient inversion layer. This represents the dynamic resistance coefficient.

9. The fault diagnosis method for power communication equipment based on multimodal fusion adversarial learning according to claim 1, characterized in that: The power communication equipment is a 5G power communication equipment.

10. A fault diagnosis system for power communication equipment based on multimodal fusion adversarial learning, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the method described in any one of claims 1 to 9.

Images