Telecommunication signal clustering analysis method based on deep clustering variational autoencoder with attention mechanism

Abstract

The application relates to a kind of deep variational autoencoder algorithm of aircraft electric signal clustering analysis. Including: variational autoencoder module, multiscale residual convolution network module and signal clustering module. When aircraft electric signal is clustered (101), first, signal acquisition and transmission (102) are carried out, are sent into multiscale residual convolution module (103) and attention mechanism layer (104), weighted multiscale features are input into maximum pooling layer (104) to complete feature extraction, feature mapping is respectively through two parallel fully connected layers (105) and (106) to obtain feature mean (107) and feature variance (108), then through decoder network, using the structure completely symmetrical with encoder, finally output reconstruction signal (113), aims at completely matching with input signal. Through the construction of loss function and two-stage training mode, the network is trained, and finally the clustering analysis (115) is completed through the reconstruction feature vector (109).

Description

Technical Field

[0001] This invention relates to an electrical signal clustering analysis method based on a deep clustering variational autoencoder with an attention mechanism. Background Technology

[0002] During the operational testing of aircraft, continuous time-series signals collected by various sensors constitute the primary source of the dataset. However, the datasets acquired by sensors are mostly unlabeled samples, posing challenges for further analysis and signal classification. To achieve signal identification and monitoring, we need to extract and analyze the data, and construct labeled datasets for supervised training of classifier models and the construction of signal recognition systems. Traditional methods involve manual labeling by domain experts, but for large-scale and complex datasets, the time and manpower costs of manual classification are enormous.

[0003] In the field of machine learning, clustering algorithms are widely used to process unlabeled data. For databases and datasets containing large amounts of data, quickly and effectively labeling the data is a challenging and time-consuming task. Clustering algorithms provide a feasible solution for labeling large datasets by automatically classifying and organizing data. Since signal data of the same category have a certain degree of similarity, clustering algorithms can classify and organize signals of multiple categories, and, with the assistance of manual identification, build an expert knowledge base.

[0004] Therefore, clustering analysis algorithms have significant application value in this context. They can quickly perform preliminary classification and organization of large amounts of complex data, reducing the workload of manual annotation and improving efficiency. By combining clustering analysis algorithms with manual identification, an expert database can be constructed, laying the foundation for subsequent classifier model training and signal recognition system development.

[0005] Traditional clustering algorithms include K-means, density clustering, and fuzzy clustering. However, these algorithms are highly dependent on the representativeness of the data's features. While they perform well on low-dimensional datasets, their clustering performance on high-dimensional data is poor. For high-dimensional data, unsupervised feature extraction algorithms are typically used to map the high-dimensional data to a low-dimensional feature space, and then clustering is performed based on the feature vectors in the low-dimensional space. However, electrical signal data is noisy and extremely large in volume, making the features extracted using this traditional method unrepresentative. Furthermore, this type of method is fragmented, lacking data feedback between the clustering and feature extraction algorithms, resulting in poor clustering results. Summary of the Invention

[0006] To address the aforementioned problems in existing technologies, the inventors have improved the integrated deep clustering algorithm based on variational autoencoders by utilizing the feature extraction capabilities of deep neural networks with attention mechanisms. This has led to the development of an electrical signal clustering analysis method based on a deep clustering variational autoencoder with an attention mechanism, enabling autonomous clustering analysis of complex electrical signal data.

[0007] Currently, in the construction of classification models for aircraft electrical signal testing, labeled electrical signal data is needed to perform supervised training of the classification model, thereby optimizing classifier performance. However, most of the raw data is unlabeled and requires preprocessing methods such as data labeling before it can be applied to actual classification tasks. Traditional data labeling methods often rely on manual work by experts, which is time-consuming and labor-intensive, significantly impacting the efficiency of model construction.

[0008] Therefore, this invention makes significant improvements to the clustering method based on deep convolutional variational autoencoder networks, enabling the electrical signal clustering analysis method based on deep clustering variational autoencoders with attention mechanisms to achieve unsupervised preliminary clustering of raw electrical signal data without using prior data labels. Furthermore, the clustering results can be used for auxiliary annotation of electrical signals, effectively improving the annotation efficiency of expert data.

[0009] In a more specific aspect, this invention improves the feature extraction model of variational autoencoders by integrating convolutional neural network structures and self-designed clustering structures. It proposes an electrical signal clustering analysis method based on a deep clustering variational autoencoder with an attention mechanism, which has stronger feature extraction and generalization capabilities, better applicability to both high-dimensional and low-dimensional signals, and demonstrates good algorithm performance.

[0010] Meanwhile, this invention incorporates an attention mechanism layer in the encoder and decoder of the autoencoder to fuse multi-scale feature signals. The addition of this attention mechanism layer fully utilizes the scale differences of different electrical signals from the aircraft, better assigning weights to signal features at different scales, helping the entire network distinguish between different types of electrical signals, and further improving the accuracy of electrical signal clustering analysis.

[0011] According to one aspect of the present invention, a deep variational autoencoder based on an attention mechanism for clustering aircraft electrical signals is provided, characterized by comprising:

[0012] Variational autoencoder module;

[0013] Multi-scale residual convolutional network module;

[0014] Attention mechanism module;

[0015] Signal clustering module.

[0016] The deep variational autoencoder based on the attention mechanism in this invention for clustering aircraft electrical signals overcomes the problem in existing aircraft signal clustering analysis that high-dimensional signals are heavily affected by noise and it is difficult to effectively extract features. It effectively solves the problems of large data volume and strong noise in high-dimensional signals, and significantly improves the accuracy of aircraft electrical signal clustering analysis. Attached Figure Description

[0017] Figure 1 A flowchart of an electrical signal clustering analysis method based on a deep clustering variational autoencoder with an attention mechanism according to an embodiment of the present invention is shown.

[0018] Figure 2 The diagram shows a training flowchart for an electrical signal clustering analysis method based on a deep clustering variational autoencoder with an attention mechanism, according to an embodiment of the present invention.

[0019] Figure 3 A flowchart illustrating the attention mechanism of an electrical signal clustering analysis method based on a deep clustering variational autoencoder with an attention mechanism, according to an embodiment of the present invention, is shown. Detailed Implementation

[0020] like Figure 1 As shown, an electrical signal clustering analysis method based on a deep clustering variational autoencoder with an attention mechanism according to an embodiment of the present invention includes:

[0021] When cluster analysis is performed on the aircraft's electrical signals (101), signal acquisition and transmission are performed first (102);

[0022] The acquired electrical signals are sent to the first multi-scale residual dilated convolution module (103) and the attention mechanism module (104);

[0023] Then, a max pooling layer (105) is stacked. In the encoder network, the input of the network is the original signal, which is then processed by the feature extraction module of the stack to obtain the feature map of the input signal;

[0024] Then, the feature mapping is passed through two parallel first fully connected layers (106) and second fully connected layers (107) to obtain the feature mean (108) and feature variance (109), respectively. The encoder structure is similar to that of the variational autoencoder.

[0025] In the decoder network, the reconstructed feature vector (110) is used as the input of the decoder network and passes through the third fully connected layer (111) to enter the feature reconstruction module. The feature reconstruction module includes an upsampling layer (112) and a second multi-scale residual dilated convolution module (113). The structure of the decoder is completely symmetrical with that of the encoder, including the stacking of layers and output channels. The upsampling layer (112) is the inverse operation of the max pooling layer (105). After passing through the feature reconstruction module, the output of the decoder is the reconstructed signal (114), which aims to completely match the input signal of the encoder network. After the feature reconstruction module, the decoder network also includes a clustering module. In this clustering module, the reconstructed feature vector is transformed through the fourth fully connected layer (115) and combined with the Softmax function in the final layer to output the clustering result (116) in the mode of outputting the classification result.

[0026] In the attention mechanism module introduced in this invention, the multi-scale features (301) extracted by the first multi-scale residual dilated convolution module (103) are used as the input of the attention mechanism module (104). This invention introduces the attention mechanism module using two feature scales as an example. The multi-scale feature (301) vector is split into feature (302 and 303) vectors of scale one and scale two according to different scales. The feature vectors are extracted by the operation of the max pooling layer (304 and 306) and the average pooling layer (305 and 307), respectively. The feature vectors are concatenated (308) and input into the first fully connected layer (309). The obtained features are processed by the ReLU activation function (310) and enter the second fully connected layer (311). The obtained features are processed by the Sigmoid activation function (312) to obtain the weight values ​​of the two scales. In the scale transformation module (313), the weight values ​​are multiplied by the corresponding multi-scale features extracted by the first multi-scale residual dilation convolution module (103) to perform scale transformation and obtain the transformed multi-scale features. The transformed multi-scale features are concatenated with the corresponding multi-scale feature matrix extracted by the first multi-scale residual dilation convolution module (103) and input into the max pooling layer (105).

[0027] The following sections will describe the first and second multi-scale residual dilated convolution modules (103, 114), the attention mechanism module (104), and the variational autoencoder.

[0028] Variational autoencoders (VAEs) build upon autoencoders by constraining the acquired feature vectors to give each value a universal statistical meaning, thus forming the algorithmic part of the generative model. In the autoencoder process, each value in the feature vector can be considered to represent a latent attribute of the original data. In the variational autoencoder algorithm, each latent attribute follows a probability distribution. To ensure successful network training in practice, each feature value is assumed to follow a standard normal distribution. Statistically, the original data X is considered to satisfy a certain probability distribution p(X) in its dimensional space, but its distribution law is difficult to calculate; the current original data sample is merely a sample x obtained by sampling from this distribution law. In the variational autoencoder, the distribution law of X is represented by a latent variable Z, i.e.:

[0029]

[0030] Therefore, p(X|Z) here represents a model that generates sample X from Z. By adjusting the values ​​in the Z vector within the distribution, the generated sample X will also satisfy the distribution of the dataset X in its dimensional space. Thus, the variational autoencoder is a generative algorithm, with its encoder part serving as the recognition model and its decoder part serving as the generative model.

[0031] Variational autoencoders (VAEs) incorporate inherent statistical functions in their network and loss function design. During encoding, the network fits two vectors of equal length, representing the mean and variance of feature Z, respectively. The encoder network structure here is a joint distribution p(X,Z) of X and Z. However, this probability distribution is difficult to solve. Therefore, the network's approach is to make p(X,Z) approximate a distribution q(X,Z), where q is set as a standard normal distribution. To ensure that the feature mean and variance satisfy the standard normal distribution, the loss function, in addition to constraints on reconstruction error, also considers the KL divergence between the distribution of Z and the standard normal distribution, i.e.:

[0032]

[0033] Where μ is the characteristic mean, σ 2 Let N(0,I) be the characteristic variance, and let N(0,I) be the standard normal distribution.

[0034] From a probabilistic perspective, the latent variable Z of the original variational autoencoder is extended to (Z, Y), where Y is a discrete variable representing the output category of the clustering layer. Therefore, its KL loss can be written as:

[0035]

[0036] x represents "sample x obtained from sampling this distribution law"; z represents the one-dimensional feature vector in the latent variable Z. In order to make the loss function easier to calculate, this invention makes a series of settings in the model:

[0037] Let p(Z|X) be a normal distribution, and its output is the characteristic mean and characteristic variance;

[0038] Let q(X|Z) represent a normal distribution with constant variance, which is equivalent to using the mean squared error as the reconstruction error;

[0039] We assume that q(Z|Y) is a normal distribution with a mean similar to the mean of the output Y and a variance of 1.

[0040] We assume that q(Y) is uniformly distributed so that the distribution of categories is balanced;

[0041] Then, the clustering module is used to fit p(Y|Z).

[0042] Finally, the network's loss function is characterized as:

[0043]

[0044] The three losses are defined as follows: (1) Reconstruction loss Used to constrain the similarity between the reconstructed signal and the input signal, the error between the input signal and the reconstructed signal is calculated using the mean square error and minimized; (2) KL loss The distribution of the constrained features is specifically determined by calculating the KL divergence of the p(Y|Z) and q(Y) distributions to constrain the distribution of the latent vectors; (3) Class loss. This is used to constrain the correspondence between the output cluster categories and feature vectors. Cross-entropy error is used to limit the output of clustering results, thereby aligning cluster categories with feature vectors.

[0045] The clustering variational autoencoder of this invention, by employing a deep neural network, can extract deep features from high-dimensional data. Multi-scale residual dilated convolutional modules have the ability to extract features from complex electrical signals, and their feature extraction performance outperforms fully connected neural networks (DNNs) and convolutional neural networks (CNNs). Therefore, this invention introduces multi-scale residual dilated convolutional modules into the clustering variational autoencoder as a major component of the encoder and decoder in the network, enabling the autoencoder network to obtain improved signal feature extraction and signal reconstruction capabilities.

[0046] The first and second multi-scale residual dilated convolution modules include the following features:

[0047] Multi-scale feature extraction branch: For high-dimensional signal features, the network is expected to capture large-scale features with a small number of layers, while not neglecting small-scale features. To this end, drawing on the well-known GoogLEnet network, this invention introduces a multi-scale feature extraction branch in the module design. The feature extraction module (multi-scale residual dilated convolution module (103), attention mechanism module (104), and max pooling layer module (105)) includes two branches with convolution kernels of different scales:

[0048] - Small-scale feature extraction branch, with a convolution kernel size of 3, is used to perceive small-scale features;

[0049] - The large-scale feature extraction branch, with a convolution kernel size of 5, is used to perceive large-scale features.

[0050] Through the separate action of these two branches, the feature extraction module can capture signal features at different scales in parallel.

[0051] Dilated convolution: Dilated convolution (or dilated convolution) refers to adding holes of several elements (dilation rate) between each element of the convolution kernel, thereby expanding the receptive field of the convolutional layer without increasing the number of kernel parameters. In this invention, the second convolutional layer of the small-scale feature extraction branch uses a convolution with a dilation rate of 2, while the first and second layers of the large-scale feature extraction branch use convolutions with dilation rates of 2 and 5, respectively. Using this technique, the receptive field of this module can be effectively expanded. Without dilated convolution, kernel sizes of 3 and 5 in the two-layer convolutional neural network result in receptive fields of lengths of 5 and 9, respectively; with dilated convolution, taking the dilated convolution module proposed in this invention as an example, the receptive fields can be expanded to 9 and 25, respectively.

[0052] This feature extraction module effectively solves the problem of extracting multi-scale features from high-dimensional signals by integrating multiple advanced neural network technologies, while overcoming the gradient vanishing problem that occurs in deep networks. Simultaneously, the use of dilated convolution effectively controls the number of network parameters and complexity, further improving operational efficiency. In summary, the proposed first and second multi-scale residual dilated convolution modules can effectively meet the feature extraction requirements of high-dimensional aircraft electrical signals.

[0053] The attention mechanism module of this invention uses an additionally designed fully connected network to calculate the multi-scale feature weights of electrical signals, thus fully considering the different scale data features of different types of signals.

[0054] Scale feature extraction: Max pooling and average pooling layers are used to extract statistical information of features at different scales. The extracted feature information at different scales is then concatenated and fused as input data for the attention mechanism module.

[0055]

[0056]

[0057] Among them U i Let m be the number of data features extracted by the i-th scale convolutional module.

[0058] Scale weight calculation: The above-concatenated multi-scale feature vector U is input into two fully connected layers and a non-linear layer (W2 and W1). The two fully connected layers can learn and calculate the weight information S of the multi-scale features in an end-to-end manner.

[0059] S = F ex (U,W)=σ(g(U,W))=σ(W2δ(W1U)) (7)

[0060] The feature matrix U is multiplied element-wise with the corresponding weight matrix S to obtain the weighted multi-scale features of the electrical signal. Then U and Perform matrix addition to obtain the final multi-scale signal features, where The calculation is shown in the following formula:

[0061]

[0062] Compared to traditional variational autoencoders, the clustering variational autoencoder of this invention exhibits better adaptability to clustering tasks. Firstly, this invention transforms the dimensionality reduction algorithm into a clustering algorithm, resolving the problem of algorithmic fragmentation when clustering high-dimensional data. Specifically, the features extracted by the deep autoencoder are directly related to the final output cluster category and are interpretable from a probabilistic perspective. Secondly, considering the data structure characteristics of electrical signals, this invention introduces a multi-scale residual dilated convolution module for feature extraction and signal reconstruction, improving the neural network's adaptability to electrical signal data.

[0063] The flowchart of the algorithm for training a deep clustering variational autoencoder network using a two-stage training method in this invention is as follows: Figure 2As shown. First, the first stage of training is performed (201). First, the iteration step is initialized (202) and a round of training is started (203). By inputting unlabeled training samples (204), the network reconstructs the signal (205) and outputs the clustering results. The error and loss function are calculated for these results (206). Then, the network parameters are updated by backpropagation using the first stage learning rate α1 (207). It is determined whether to traverse the training set (208). If "no", return to step (204). If it has been completed, the iteration step number is incremented by 1 (209). Then, it is determined whether the number of rounds has reached the number of training rounds in the first stage (210). If "no", return to step (203). If "yes", end the first stage of training and start the second stage of training.

[0064] In the second stage of training, the network parameters are inherited and multiple rounds of training begin. The difference is that the network parameters are updated by backpropagation using the second-stage learning rate α2 (217). When the number of rounds reaches the set target (220), the training ends (221). The loss function used for training is shown in equation (4).

[0065] The advantages and beneficial effects of the deep variational autoencoder algorithm for clustering aircraft electrical signals according to the present invention include:

[0066] (1) Compared with traditional variational autoencoders, the deep clustering variational autoencoder of this invention has better adaptability in completing clustering tasks. First, the clustering variational autoencoder of this invention transforms the dimensionality reduction algorithm into a clustering algorithm, solving the problem of algorithm fragmentation when clustering high-dimensional data. That is, the features extracted by the deep autoencoder are directly related to the final output cluster category and have interpretability from a probabilistic perspective. At the same time, considering the data structure characteristics of electrical signals, the deep clustering variational autoencoder of this invention introduces a multi-scale residual dilated convolution module to extract and reconstruct signals, and combines an attention mechanism module to calculate the weights of electrical signal features at different scales and perform weighted feature fusion, thus having a stronger adaptability to electrical signal data.

[0067] (2) The deep clustering variational autoencoder of the present invention introduces a multi-scale residual dilated convolution module to extract features from the signal. By integrating a variety of advanced neural network technologies, it effectively solves the problem of extracting multi-scale features from high-dimensional signals and overcomes the gradient vanishing problem that occurs in deep networks. At the same time, due to the use of dilated convolution, the number of network parameters and complexity are effectively controlled, the operating efficiency is further improved, and the feature extraction requirements of high-dimensional electrical signals of aircraft are effectively met.

[0068] (3) The deep clustering variational autoencoder of the present invention significantly improves the clustering accuracy of aircraft electrical signals, has stronger adaptability and robustness to electrical signal data, and makes outstanding contributions to the core fault identification of aircraft health management strategies.

Claims

1. A method of electrical signal cluster analysis based on a deep clustering variational autoencoder with attention mechanism, characterized in that The method comprises the following steps: receiving, by a first multi-scale residual dilated convolution module (103), an electrical signal collected by a signal collection and transmission module (102); stacking, by an attention mechanism module (104), the first multi-scale residual dilated convolution module (103) to obtain a feature map from the electrical signal; extracting and reducing the dimension of the obtained feature map by a maximum pooling layer (105); passing the feature map through two parallel first and second fully connected layers (106) and (107) to obtain a feature mean (108) and a feature variance (109), respectively; reconstructing the feature mean and the feature variance into a feature vector by a reconstruction layer (110); inputting the feature vector into a third fully connected layer (111); passing the feature vector through the third fully connected layer (111) into an up-sampling layer (112); performing reverse operations of the maximum pooling layer (105) by the up-sampling layer (112) in combination with a second multi-scale residual dilated convolution module (113); reconstructing the output of the decoder network module by a reconstruction module (114) to completely match the input of the encoder; performing feature transformation on the reconstructed feature vector by a fourth fully connected layer (115); outputting a clustering result by combining the feature vector after the feature transformation by the fourth fully connected layer (115) with a Softmax function by a final layer (116), wherein the first and second multi-scale residual dilated convolution modules are used to perform the following operations: assuming that an original data X satisfies a probability distribution p(X) in its dimensional space, assuming that a current original data sample is a sample x obtained by sampling from the distribution p(X), representing the distribution law of X by constructing a hidden variable Z, i.e.: representing a model for generating a sample X from Z by p(X|Z), and by adjusting the values in the vector Z within the distribution, the generated sample X will also satisfy the distribution of the data set X in its dimensional space, in the encoding process, a variational autoencoder module including the signal collection and transmission module (102), the first multi-scale residual dilated convolution module (103), the attention mechanism module (104), the maximum pooling layer (105), the two parallel first and second fully connected layers (106) and (107) is used to fit two vectors of equal length, representing the mean (108) and the variance (109) of Z, respectively, and the network structure of the variational autoencoder module is the joint distribution p(X, Z) of X and Z, making p(X, Z) as close as possible to an approximate distribution q(X, Z), wherein q is set as a standard normal distribution, to make the mean and the feature variance of Z satisfy the standard normal distribution, the loss function considers the KL divergence between the distribution of Z and the standard normal distribution in addition to the constraint on the reconstruction error, i.e.: where μ is the characteristic mean value, σ 2 is the characteristic variance, and N(0, I) is the standard normal distribution, thus, Z is expanded to (Z, Y), wherein Y is a discrete variable output by the clustering layer and represents the output category, and thus the KL loss of Z is represented as: wherein x represents a sample obtained by sampling from the distribution law of X, and z represents one dimension in the hidden variable Z, to make the loss function easier to calculate: Set p(Z|X) as a normal distribution, and the output is the mean and characteristic variance of Z; Set q(X|Z) as a normal distribution with constant variance; Set q(Z|Y) as a normal distribution with mean equal to the mean of Y output and variance of 1; Set q(Y) as a uniform distribution, so that the distribution of categories is balanced; Use the clustering module to fit p(Y|Z), and represent the loss function as: Where the meanings of the three losses are respectively: reconstruction loss To constrain the similarity of the reconstructed signal and the input signal, the error of the input signal and the reconstructed signal is calculated by mean square error and minimized. kl loss for constraining the distribution of features; Class loss To constrain the correspondence between the output cluster class and the feature vector, the alignment between the cluster class and the feature vector is achieved by limiting the output of the clustering result through cross-entropy error.

2. The electric signal clustering analysis method according to claim 1, characterized in that: In KL loss In the determination, the KL divergence between the distribution p(Y|Z) and the distribution q(Y) is calculated to constrain the distribution of the latent vector.

3. The electrical signal cluster analysis method of claim 1, wherein The first and second multi-scale residual dilated convolution modules each comprise: Two branches with different scale convolution kernels: A small-scale feature extraction branch with a convolution kernel size of 3 for perceiving small-scale features; A large-scale feature extraction branch with a convolution kernel size of 5 for perceiving large-scale features, Through the respective actions of the two branches, the first and second multi-scale residual dilated convolution modules can capture different scale signal features in parallel, Wherein: The second convolution layer of the small-scale feature extraction branch adopts a convolution with an expansion rate of 2, The first and second layers of the large-scale feature extraction branch respectively adopt convolutions with expansion rates of 2 and 5.

4. The electrical signal cluster analysis method according to one of claims 1 to 3, characterized in that Further comprising training by using two-stage training, including: First, first-stage training (201), First iteration step initialization (202), Start a round of training (203), Input unlabeled training samples (204), Reconstruct the sample signal (205) and output the clustering result, Calculate the error and loss function of the clustering result (206), Then update the network parameters by back propagation using the first-stage learning rate α1 (207), Determine whether the training set is traversed (208), "No" returns to the step of inputting unlabeled training samples (204), "Yes" increments the iteration step number by 1 (209), Then determine whether the number of rounds reaches the number of training rounds in the first stage (210), "No" returns to the step of starting a round of training (203), and "Yes" ends the first-stage training and starts the second-stage training, In the second-stage training, the network parameters are inherited and multiple rounds of training are started, the difference being that the network parameters are updated by back propagation using the second-stage learning rate α2 (217), When the number of rounds reaches the set target (220), end the training (221), Wherein the loss function used in training is shown in formula (4).

Images