Power communication equipment fault diagnosis method and system based on multi-modal fusion adversarial learning

Abstract

The application discloses a kind of power communication equipment fault diagnosis method and system based on multi-modal fusion confrontation learning, obtains time sequence reference signal containing radio frequency front-end fault information under normal and abnormal environment and obtains time-frequency spectrogram by short-time Fourier transform, constructs multi-modal dataset and carries out pre-processing;Through double-branch convolution network, signal time domain features and time-frequency spectrogram frequency domain features are extracted respectively;Dynamic weighted fusion time domain features and frequency domain features are generated by using attention fusion module multi-modal fusion features;A double-task branch including fault classifier and environment discriminator is constructed, and the fault type is identified by using the fault classifier, and the environment discriminator predicts the environment category;The application realizes the non-invasive accurate diagnosis of radio frequency front-end fault of power communication equipment through multi-modal fusion confrontation learning, effectively solves the physical inaccessibility, feature mode drift and feature confusion problem of traditional method, and still can maintain stable diagnostic performance in complex environment.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of power communication equipment fault diagnosis, in particular to a power communication equipment fault diagnosis method and system based on multi-modal fusion adversarial learning. BACKGROUND

[0002] In power grid wireless transmission, large-scale MIMO is the core of 5G base station, and its stability is crucial for the reliable transmission of power grid dispatching instructions. The radio frequency front end is a key component of communication equipment in MIMO communication, and any damage to it will seriously affect the communication quality. Therefore, monitoring the damage of the radio frequency front end of the communication equipment by the base station receiver and accurately identifying the fault are of great significance to ensure the continuity of power grid wireless transmission.

[0003] Traditional power communication equipment fault diagnosis methods have obvious physical limitations. One is the test technology based on physical model, which needs to obtain key parameters through pre-set internal test nodes, but the physical node accessibility of highly integrated 5G base station equipment has decreased sharply; the other is the online monitoring scheme of external test equipment, which has the problems of complex deployment and high cost, and cannot meet the continuous monitoring demand of power grid, and both methods need to access the equipment internally, which cannot meet the non-invasive diagnosis requirements in complex outdoor environment of power system.

[0004] Although the non-invasive method based on the reference signal of the receiving end avoids the physical access limitation, it still faces the core challenge, because its essence is the superposition of fault features and channel interference, which will cause mode drift and feature confusion, and the existing data-driven methods cannot effectively solve the environmental interference problem, resulting in insufficient diagnosis robustness. SUMMARY

[0005] Therefore, the purpose of the present application is to provide a power communication equipment fault diagnosis method and system based on multi-modal fusion adversarial learning, which uses a multi-modal feature extractor to predict multi-modal fusion features.

[0006] To achieve the above purpose, the present application provides the following technical scheme:

[0007] The power communication equipment fault diagnosis method based on multi-modal fusion adversarial learning provided by the present application comprises the following steps:

[0008] S1, respectively acquiring time sequence reference signals containing radio frequency front end fault information in power communication equipment under normal environment and abnormal environment and obtaining time-frequency spectrograms through short-time Fourier transform, constructing a multi-modal data set and pre-processing;

[0009] S2, constructing a double-branch convolutional network to extract time domain features of the reference signal and frequency domain features of the time-frequency spectrogram;

[0010] S3, the time domain features and the frequency domain features are fused into multi-modal fusion features by a cross-modal attention fusion module to construct a multi-modal feature extractor;

[0011] S4, a double-task branch including a fault classifier and an environment discriminator is constructed, and the multi-modal fusion features are input into the fault classifier and the environment discriminator respectively to predict the fault category and the environment category;

[0012] S5, a gradient inversion layer is set between the multi-modal feature extractor and the environment discriminator, and a dynamic adversarial learning mechanism is used to train the model, so that the model learns robust fault features irrelevant to the environment.

[0013] Further, the time sequence reference signal in step S1 is a sounding reference signal sent by a power communication device to a base station, including in-phase component and quadrature component.

[0014] Further, the step S1 of constructing a multi-modal data set and preprocessing includes the following steps:

[0015] S11, extracting the effective segment of the time sequence reference signal, and compressing through a preset window to normalize the amplitude range through linear transformation;

[0016] S12, converting the time-frequency spectrogram into a single-channel grayscale image, and adjusting the resolution through bilinear interpolation;

[0017] S13, using a hierarchical sampling strategy to divide the multi-modal data set into a training set, a validation set and a test set according to a preset proportion.

[0018] Further, the double-branch convolutional network in step S2 includes:

[0019] The first branch is a time domain feature extractor: the input is a time sequence reference signal, including a double-channel reference signal of in-phase component and quadrature component, and its structure is composed of two one-dimensional convolution modules and an adaptive average pooling layer connected in turn, wherein the one-dimensional convolution module includes a one-dimensional convolution layer, batch normalization, ReLU activation, Dropout regularization and a maximum pooling layer;

[0020] The second branch is a frequency domain feature extractor: the input is a time-frequency spectrogram, including a double-channel time-frequency spectrogram of in-phase component and quadrature component, and its structure is composed of two two-dimensional convolution modules and an adaptive average pooling layer connected in turn, wherein the two-dimensional convolution module includes a two-dimensional convolution layer, batch normalization, ReLU activation, Dropout regularization and a maximum pooling layer.

[0021] Further, the step S3 of fusing the time domain features and the frequency domain features into multi-modal fusion features by the cross-modal attention fusion module is performed according to the following steps:

[0022] S31, the time domain features and frequency domain features projecting to a common feature space, generating an attention weight matrix:

[0023]

[0024] wherein denotes a Sigmoid activation function, and is a learnable parameter matrix, and denote a time domain attention weight matrix and a frequency domain attention weight matrix, respectively;

[0025] S32, using the generated weight attention matrix to weight and fuse the projected features to obtain a multi-modal fusion feature :

[0026]

[0027] wherein denotes a multi-modal fusion feature, and denote learnable parameter matrices used for projection, denotes an element-wise multiplication.

[0028] Further, the 5G communication device radio frequency front-end fault types distinguished by the fault classifier in the step S4 include nonlinear power amplifier distortion, I / Q imbalance, and phase noise; and the environment types distinguished by the environment discriminator mainly include normal environment and abnormal environment of the area where the 5G communication device is located.

[0029] Further, the total loss function of the dynamic adversarial learning mechanism in the step S5 is:

[0030]

[0031] wherein denotes a total loss, denotes a multi-modal feature extractor parameter, denotes a fault classifier parameter, denotes an environment classifier parameter;

[0032] denotes a fault classification loss, and only the samples containing fault classification labels under normal environment are used to calculate the cross-entropy loss;

[0033] denotes an environment classification loss, and the samples under normal environment and the samples under abnormal environment in the entire multi-modal data set are used to calculate the cross-entropy loss;

[0034] denotes a dynamic adversarial coefficient, and the calculation formula is: ;

[0035] wherein denotes a training progress, denotes a hyperparameter for controlling a curve rate.

[0036] Further, the gradient reversal layer in the step S5 keeps the multi-modal fusion feature unchanged when forward propagation, and multiplies the classification loss of the environment discriminator with respect to the gradient of the multi-modal feature extractor parameter when backward propagation. The calculation formula is:

[0037]

[0038] wherein GRL denotes a gradient reversal layer, and denotes a dynamic adversarial coefficient.

[0039] The power communication equipment fault diagnosis system based on multi-modal fusion adversarial learning provided by the application comprises a memory, a processor, and a computer program stored in the memory and capable of running on the processor, and the processor implements the above method when executing the program.

[0040] The application has the following beneficial effects:

[0041] The power communication equipment fault diagnosis method and system based on multi-modal fusion adversarial learning provided by the application first acquire time sequence reference signals containing radio frequency front-end fault information in power communication equipment under normal and abnormal environments respectively, and obtain time-frequency spectrograms through short-time Fourier transform, construct a multi-modal data set and perform pretreatment; then construct a double-branch convolution network to extract time-domain features of the reference signals and frequency-domain features of the time-frequency spectrograms; the time-domain features and the frequency-domain features are fused into multi-modal fusion features by using a cross-modal attention fusion module to construct a multi-modal feature extractor; finally, a double-task branch including a fault classifier and an environment discriminator is constructed, and the multi-modal fusion features are input into the fault classifier and the environment discriminator to predict fault categories and environment categories respectively; a gradient reversal layer is set between the multi-modal feature extractor and the environment discriminator, and a dynamic adversarial learning mechanism is used to train the model, so that the model learns robust fault features irrelevant to the environment. Through multi-modal fusion adversarial learning, the application realizes non-invasive and accurate diagnosis of radio frequency front-end faults of power communication equipment, effectively solves the problems of physical inaccessibility, feature mode drift and feature confusion of traditional methods, and still maintains stable diagnosis performance in complex environments.

[0042] The method does not need to rely on the test node preset in the device or the external test equipment, and can realize fault diagnosis through the reference signal of the receiving end, adapt to the problem of physical accessibility decline caused by the high integration of power communication equipment, and meet the needs of non-intrusive and remote monitoring in complex outdoor environments of power systems.

[0043] The method extracts time domain and frequency domain features of the reference signal through a double-branch convolutional network, combines a cross-modal attention fusion module, dynamically weighs the contribution of different modal features to fault diagnosis, realizes complementary fusion of time domain transient features and frequency domain steady-state spectrum features, and enhances the comprehensiveness and distinguishability of fault features.

[0044] The method constructs an adversarial relationship between the multi-modal fusion feature extractor and the environment discriminator through a gradient reversal layer, forces the model to learn robust fault features irrelevant to environmental changes, effectively solves the mode drift and feature confusion problems existing in traditional methods, and ensures stable diagnostic performance in complex wireless transmission environments.

[0045] Other advantages, objects and features of the present application will be in part apparent and in part pointed out hereinafter in the specification, and it is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory thereof, and are intended to provide further explanation of the present application as claimed. BRIEF DESCRIPTION OF DRAWINGS

[0046] In order to make the purposes, technical solutions and beneficial effects of the present application clearer, the present application provides the following drawings for illustration.

[0047] Figure 1 It is a simplified process diagram of the method.

[0048] Figure 2 It is a model structure diagram of the method.

[0049] Figure 3 It is the training and verification loss curve of the multi-modal fusion adversarial learning model of the method.

[0050] Figure 4 It is the overall confusion matrix of the model of the method on the test set.

[0051] Figure 5 It is a comparison diagram of multi-modal fusion adversarial learning feature principal component analysis visualization and multi-modal fusion feature principal component analysis visualization of the method.

[0052] Figure 6 It is an accuracy comparison diagram of different models under normal and abnormal environments. DETAILED DESCRIPTION

[0053] The present application will be further described below in conjunction with the accompanying drawings and specific embodiments so that those skilled in the art can better understand the present application and implement it. The embodiments are not intended to limit the present application.

[0054] Reference Figure 1 and Figure 2 , Figure 1 is a simplified flowchart of the method of the present application, Figure 2 is a model structure diagram of the method. The embodiment of the present application discloses a power communication equipment fault diagnosis method based on multi-modal fusion adversarial learning, comprising the following steps:

[0055] S1, respectively acquire time sequence reference signals containing radio frequency front end fault information in power communication equipment under normal environment and abnormal environment, and obtain time-frequency spectrograms through short-time Fourier transform, construct a multi-modal data set and perform pretreatment.

[0056] It should be noted that the data set of the embodiment is constructed based on MIMO-OFDM system level simulation, and all samples are generated in strict accordance with 5GNR specification. The system uses a 4x4 planar antenna array, the base station height is 15 meters, the communication equipment terminal height is 1.5 meters, the transmission distance is 50 meters, the center working frequency is 2.3GHz, the system bandwidth is 100MHz, and the 3GPP channel model is used to characterize the power grid wireless transmission characteristics. Each sample generation process is based on complete communication link simulation, which specifically includes the following five steps:

[0057] Generate a probe reference signal at the transmitting end of the communication equipment, and map it to a resource grid as an analysis carrier;

[0058] Introduce three types of faults of nonlinear power amplification, I / Q imbalance and phase noise at the transmitting end of the communication equipment, and generate four types of fault labels in combination with normal conditions;

[0059] Simulate the fading characteristics of wireless propagation based on the 3GPP model to generate multipath channel coefficients in this simulated environment;

[0060] After the signal is transmitted and received by the multi-antenna system, the channel fading and different environmental noise are superimposed, wherein the samples with a signal-to-noise ratio range of 22-28dB are defined as samples under normal environment, and the samples with a signal-to-noise ratio range of 5-15dB are defined as samples under abnormal environment;

[0061] Receive the time sequence reference signal including the in-phase component and the quadrature component.

[0062] After the above stage, two types of data sets are generated in total, one type is 2000 samples with fault labels under normal environment; the other type is a mixed data set containing 2000 samples with labels under normal environment and 2000 samples without labels under abnormal environment, which is used to simulate the scene that fault labels are difficult to obtain under abnormal environment in practice.

[0063] The power communication device in the embodiment is a 5G power communication device.

[0064] It should be noted that in S1, the timing reference signal is a sounding reference signal sent by the power communication device to the base station, including an in-phase component and a quadrature component;

[0065] It should be noted that in S1, the short-time Fourier transform adopts a 1024-point Hanning window to process the in-phase component and the quadrature component of the reference signal, and respectively obtains the time-frequency spectrum of the in-phase component and the quadrature component;

[0066] It should be noted that in S1, constructing a multi-modal data set and preprocessing includes the following steps:

[0067] The samples in the multi-modal data set under normal environment contain fault labels, and the samples under abnormal environment do not contain fault labels;

[0068] The effective segment of the timing reference signal is extracted and compressed to 2000 points by taking the maximum value in a 4-point window, and the amplitude range is normalized to [-1, 1] by linear transformation; the time-frequency spectrum is converted into a single-channel grayscale image, and the resolution is adjusted to 128x128 by bilinear interpolation. ;

[0069] The multi-modal data set is divided into a training set, a validation set and a test set according to a stratified sampling strategy in a ratio of 6:2:2.

[0070] S2, a double-branch convolutional network is constructed to extract time-domain features of the reference signal and frequency-domain features of the time-frequency spectrum;

[0071] It should be noted that in S2, the double-branch convolutional network specifically includes:

[0072] The first branch is a time-domain feature extractor: its input is a double-channel reference signal including an in-phase component and a quadrature component, and its structure is composed of two one-dimensional convolution modules and an adaptive average pooling layer, wherein the one-dimensional convolution module includes a one-dimensional convolution layer, a batch normalization, a ReLU activation, a Dropout regularization and a maximum pooling layer;

[0073] The second branch is a frequency domain feature extractor, an input of which is a dual-channel time-frequency spectrogram including in-phase components and quadrature components, and a structure of which is composed of two two-dimensional convolution modules and an adaptive average pooling layer, wherein each two-dimensional convolution module includes a two-dimensional convolution layer, a batch normalization, a ReLU activation, a Dropout regularization and a max pooling layer.

[0074] S3, fusing the time domain features and the frequency domain features into multi-modal fusion features by using a cross-modal attention fusion module to construct a multi-modal feature extractor;

[0075] It should be noted that in S3, the step of fusing the time domain features and the frequency domain features into multi-modal fusion features by using the cross-modal attention fusion module includes the following steps:

[0076] projecting the time domain features and the frequency domain features into a common feature space to generate an attention weight matrix:

[0077]

[0078] wherein represents a Sigmoid activation function, and are learnable parameter matrices, and represent a time domain attention weight matrix and a frequency domain attention weight matrix, respectively;

[0079] using the generated weight attention matrix to perform weighted fusion on the projected features to obtain multi-modal fusion features :

[0080]

[0081] wherein represents the multi-modal fusion features, and are learnable parameter matrices used for projection, represents an element-wise multiplication.

[0082] S4, constructing a dual-task branch including a fault classifier and an environment discriminator, and inputting the multi-modal fusion features into the fault classifier and the environment discriminator respectively to predict fault categories and environment categories;

[0083] It should be noted that in S4, the fault types of the 5G communication device radio frequency front-end distinguished by the fault classifier include non-linear power amplifier distortion, I / Q imbalance and phase noise; and the environment types distinguished by the environment discriminator mainly include normal environment and abnormal environment of the area where the 5G communication device is located.

[0084] S5, a gradient reversal layer is set between the multi-modal feature extractor and the environment discriminator, a dynamic adversarial learning mechanism is used to train the model, so that the model learns robust fault features irrelevant to the environment;

[0085] It should be noted that in S5, the total loss function of the dynamic adversarial learning mechanism is:

[0086]

[0087] Wherein represents the total loss, represents the multi-modal feature extractor parameter, represents the fault classifier parameter, represents the environment classifier parameter;

[0088] represents the fault classification loss, and the cross-entropy loss is calculated only by using the samples with fault classification labels under normal environment;

[0089] represents the environment classification loss, and the cross-entropy loss is calculated by using the samples under normal environment and the samples under abnormal environment in the whole multi-modal data set;

[0090] represents the dynamic adversarial coefficient, and the calculation formula is: ;

[0091] Wherein represents the training progress, represents the hyperparameter for controlling the curve rate;

[0092] It should be noted that the hyperparameters during model training include total training rounds 100, initial learning rate 1e-4, cosine annealing scheduling strategy, Adam optimizer, Dropout probability 0.2, 10, reference Figure 3 , the training and verification loss curve of the model is shown in the figure, which reflects the dynamic balance of adversarial learning: the total loss decreases rapidly in the early stage, focusing on fault classification; In the middle stage, the fluctuation is adjusted to form an adversarial balance; In the later stage, the model is not over-fitted, and the generalization ability is good. Using this model, test on the test set, the confusion matrix after testing is shown in reference Figure 4 , Figure 3 , which shows the sample classification results of each class on the test set.

[0093] Reference Figure 5 , Figure 5For the contrast chart of the multi-modal fusion adversarial learning feature principal component analysis visualization and the multi-modal fusion feature principal component analysis visualization, according to the chart, the distribution of them in the feature space is compared. Compared with the multi-modal feature, the feature distribution overlap area of the multi-modal fusion adversarial learning feature under the two environments is significantly shrunk, which shows that the adaptive mechanism of adversarial learning forces the fusion feature to focus more on the essential mode of the fault rather than the environmental interference by maximizing the environmental classification loss.

[0094] Reference Figure 6 , Figure 6 For the accuracy contrast chart of different models under normal environment and abnormal environment, according to the chart, it is explained that the multi-modal fusion feature adopted in the method can realize the complementary fusion of the time domain transient feature and the frequency domain steady state spectrum feature, enhance the comprehensiveness and distinguishability of the fault feature, and at the same time, the dynamic adversarial learning mechanism can effectively resist the mode drift and feature confusion problem caused by the environmental anomaly, and ensure the stable diagnostic performance in the complex wireless transmission environment.

[0095] It should be noted that in S5, the gradient reversal layer keeps the multi-modal fusion feature unchanged when forward propagating , and multiplies the gradient of the classification loss of the environment discriminator with respect to the parameters of the multi-modal feature extractor when backward propagating , and the calculation formula is:

[0096]

[0097] Wherein GRL represents the gradient reversal layer, and represents the dynamic adversarial coefficient.

[0098] The power communication equipment fault diagnosis system based on multi-modal fusion adversarial learning provided in the embodiment includes a memory, a processor, and a computer program stored on the memory and executable on the processor, and the processor implements the above method when executing the program.

[0099] The above-described embodiments are only preferred embodiments for fully illustrating the present application, and the protection scope of the present application is not limited thereto. Any equivalent substitution or transformation made by those skilled in the art based on the present application is within the protection scope of the present application. The protection scope of the present application is subject to the claims.​

Claims

1. A power communication equipment fault diagnosis method based on multi-modal fusion adversarial learning, characterized in that: The method comprises the following steps: S1, respectively acquiring time sequence reference signals containing radio frequency front-end fault information in power communication equipment under normal environment and abnormal environment, and obtaining time-frequency spectrograms through short-time Fourier transform, constructing a multi-modal data set and pre-processing; S2, constructing a double-branch convolutional network to extract time domain features of the reference signal and frequency domain features of the time-frequency spectrogram; S3, fusing the time domain features and the frequency domain features into multi-modal fusion features by using a cross-modal attention fusion module to construct a multi-modal feature extractor; S4, constructing a double-task branch including a fault classifier and an environment discriminator, and inputting the multi-modal fusion features into the fault classifier and the environment discriminator to predict the fault category and the environment category respectively; S5, setting a gradient reversal layer between the multi-modal feature extractor and the environment discriminator, and training the model by using a dynamic adversarial learning mechanism, so that the model learns robust fault features independent of the environment; The step S1 of constructing a multi-modal data set and pre-processing comprises the following steps: S11, extracting the effective segment of the time sequence reference signal, and compressing the amplitude range through a pre-set window to normalize it through linear transformation; S12, converting the time-frequency spectrogram into a single-channel grayscale image and adjusting the resolution through bilinear interpolation; S13, dividing the multi-modal data set into a training set, a validation set and a test set according to a pre-set proportion by using a hierarchical sampling strategy; In the step S3, the cross-modal attention fusion module is used to fuse the time domain features and the frequency domain features to obtain the multi-modal fusion features, which are specifically performed according to the following steps: S31, projecting the time-domain features and the frequency-domain features to a common feature space to generate an attention weight matrix: wherein denotes a Sigmoid activation function, and is a learnable parameter matrix, and denote a time-domain attention weight matrix and a frequency-domain attention weight matrix, respectively; S32, the generated weight attention matrix is used to weight and fuse the projected features to obtain a multi-modal fusion feature : wherein denotes a multi-modal fusion feature, and denote learnable parameter matrices for projection, denotes an element-wise multiplication.

2. The power communication device fault diagnosis method based on multi-modal fusion adversarial learning according to claim 1, characterized in that: The time sequence reference signal in the step S1 is a sounding reference signal sent by the power communication equipment to the base station, including in-phase component and quadrature component.

3. The power communication device fault diagnosis method based on multi-modal fusion adversarial learning according to claim 1, characterized in that: The double-branch convolutional network in the step S2 comprises: The first branch is a time domain feature extractor: the input is a time sequence reference signal, including a double-channel reference signal of in-phase component and quadrature component, the structure is composed of two one-dimensional convolution modules and an adaptive average pooling layer connected in sequence, and the one-dimensional convolution module comprises a one-dimensional convolution layer, batch normalization, ReLU activation, Dropout regularization and a maximum pooling layer; The second branch is a frequency domain feature extractor: the input is a time-frequency spectrogram, including a double-channel time-frequency spectrogram of in-phase component and quadrature component, the structure is composed of two two-dimensional convolution modules and an adaptive average pooling layer connected in sequence, and the two-dimensional convolution module comprises a two-dimensional convolution layer, batch normalization, ReLU activation, Dropout regularization and a maximum pooling layer.

4. The power communication device fault diagnosis method based on multi-modal fusion adversarial learning according to claim 1, characterized in that: The fault types of the 5G communication equipment distinguished by the fault classifier in the step S4 include nonlinear power amplifier distortion, I / Q imbalance and phase noise; and the environment types distinguished by the environment discriminator mainly include normal environment and abnormal environment of the area where the 5G communication equipment is located.

5. The power communication device fault diagnosis method based on multi-modal fusion adversarial learning according to claim 1, characterized in that: The total loss function of the dynamic adversarial learning mechanism in step S5 is: wherein denotes the total loss, denotes the multi-modal feature extractor parameters, denotes the fault classifier parameters, denotes the environment classifier parameters; represents the fault classification loss, and the cross-entropy loss is calculated only by using the samples with fault classification labels under normal environment; represents an environment classification loss, and a cross-entropy loss is calculated using samples in a normal environment and samples in an abnormal environment in the entire multi-modal data set; denotes the dynamic adversarial coefficient, and the calculation formula is: ; wherein represents the training progress, represents the control curve rate of the hyperparameters.

6. The power communication device fault diagnosis method based on multi-modal fusion adversarial learning according to claim 1, characterized in that: The gradient inversion layer in the step S5 keeps the multi-modal fusion feature when propagating forward Invariant, the classification loss of the environment discriminator when propagating backward The gradient of the multi-modal feature extractor parameter is multiplied by The calculation formula is: where GRL represents a gradient reversal layer, denotes a dynamic adversarial coefficient.

7. The power communication device fault diagnosis method based on multi-modal fusion adversarial learning according to claim 1, characterized in that: The power communication equipment is a 5G power communication equipment. 8.A power communication equipment fault diagnosis system based on multi-modal fusion adversarial learning, comprising a memory, a processor and a computer program stored in the memory and capable of running on the processor, characterized in that, The processor executes the program to implement the method of any one of claims 1-7. The processor executes the program to implement the method of any one of claims 1-7.

Images