Intelligent identification method for ultrashort wave signal system based on Masked auto-encoder

Abstract

The invention discloses an ultrashort wave signal system intelligent identification method based on a Masked auto-encoder, and the method comprises the steps: carrying out the preprocessing of ultrashort wave broadband data according to the signal outgoing time, center frequency and bandwidth, carrying out the short-time Fourier transform parameter extraction of ultrashort wave narrowband time-frequency features, and constructing an unsupervised learning data set; a random mask mechanism is redesigned for an ultra-short wave signal system recognition task, and pre-training is carried out on a large-scale ultra-short wave narrow-band time-frequency graph data set through a self-supervised learning mode based on a Masked auto-encoder. A multi-task learning module is designed around the basic structure of a pre-training model, reconstruction learning and classification learning are carried out on a marked specific-system ultrashort wave signal time-frequency graph, and the problem that the model generalization is poor due to the fact that the number of ultrashort wave signal samples is insufficient and distribution is uneven is solved. A method of large-scale self-supervised learning pre-training and small-scale supervised learning fine tuning is adopted to realize specific system ultrashort wave signal identification in a complex electromagnetic environment.

Description

Technical Field

[0001] The present invention belongs to the technical field of space target recognition, and particularly relates to an intelligent recognition method for ultra-short wave signal systems based on Masked autoencoders. Background Art

[0002] The ultra-short wave communication frequency band is between 30 MHz and 300 MHz. Due to its characteristics such as wide frequency band, wide coverage, and strong penetration, it is widely used in tactical communication and is one of the most commonly used communication methods in the current military field. The current battlefield electromagnetic environment is complex, filled with a large amount of electronic interference and noise, and at the same time contains ultra-short wave band communication signals such as conventional signals, frequency hopping, spread spectrum, and radio stations. How to quickly and accurately identify the ultra-short wave signal system from the intercepted signals is crucial and is an important way to obtain intelligence. By accurately identifying the ultra-short wave communication signal system of the opponent, information leakage can be effectively prevented, and important tactical guidance can be provided for counterattacks.

[0003] The types of ultra-short wave reconnaissance equipment are increasing day by day. The traditional operation method of manual analysis and comparison to complete the operation and maintenance of each reconnaissance station has a high cost and insufficient timeliness. It is necessary to rely on the development of intelligent technologies to study ultra-short wave reconnaissance methods based on artificial intelligence to effectively improve the accuracy and timeliness of communication reconnaissance. The patent application with the application number 201811159958.X provides an ultra-short wave specific signal recognition method based on a convolutional neural network. In this method, it is proposed to combine the time-frequency spectrogram of the ultra-short wave signal with the convolutional neural network, and use the obtained signal time-frequency spectrogram to train the optimized convolutional neural network model, and finally realize the recognition of ultra-short wave specific signals. Ma Bo'ang et al. from the 54th Research Institute of CETC proposed in "Ultra-short Wave Time-Frequency Map Classification Method Based on Improved VGG16" to convert the actually collected ultra-short wave blind signals in the electromagnetic battlefield into time-frequency spectrograms, and then combine them with the optimized VGG16 convolutional neural network through transfer learning, and introduce dilated convolution into the network to complete the classification of ultra-short wave blind signals. The above methods all use supervised learning methods to realize the recognition of ultra-short wave specific signals. However, in the actual scenario, the number of intercepted military ultra-short wave signals is small and the interference is serious, making it difficult to construct a large-scale and standard ultra-short wave signal system recognition data set, resulting in insufficient generalization of the ultra-short wave system recognition algorithm. Summary of the Invention

[0004] The object of the present invention is to overcome the deficiencies of the prior art and provide an intelligent identification method for ultra-short wave signal systems based on a Masked autoencoder. By constructing a Masked Encoder model, unsupervised learning is carried out using a large-scale dataset of time-frequency diagrams of unlabeled ultra-short wave communication signals, and then fine-tuning is performed on the time-frequency diagrams of ultra-short wave narrowband signals of specific systems with labeling information. By reconstructing learning to assist in the identification of ultra-short wave systems, the generalization of the model can be improved, and intelligent identification of specific ultra-short wave signals under low signal-to-noise ratio conditions can be achieved.

[0005] The object of the present invention is achieved by the following technical solutions: An intelligent identification method for ultra-short wave signal systems based on a Masked autoencoder, comprising the following steps:

[0006] S1. Preprocess the signal according to the outgoing time, center frequency, and bandwidth of the intercepted ultra-short wave signal. The preprocessed ultra-short wave narrowband signal must contain the content of the synchronization frame and part of the data frame of the signal;

[0007] Then, perform a short-time Fourier transform on the preprocessed ultra-short wave narrowband signal to construct a dataset of time-frequency diagrams of ultra-short wave narrowband signals for unsupervised learning;

[0008] S2. Construct a Masked AutoEncoder intelligent identification model for ultra-short wave systems, and perform unsupervised learning pre-training using the dataset of time-frequency diagrams of ultra-short wave narrowband signals; The Masked AutoEncoder intelligent identification model for ultra-short wave systems sequentially includes a random masking layer, an encoding layer, a decoding layer, and a classification output layer;

[0009] S3. According to the ultra-short wave signals of known systems, perform preprocessing and time-frequency transformation according to the process in step S1 to construct a dataset of time-frequency diagrams of ultra-short wave narrowband signals for supervised learning;

[0010] S4. Fine-tune the pre-trained Masked AutoEncoder intelligent identification model for ultra-short wave systems, and perform training using the dataset constructed in step S3;

[0011] S5. Use the trained Masked AutoEncoder intelligent identification model for ultra-short wave systems to identify the systems of specific ultra-short wave communication signals.

[0012] The beneficial effects of the present invention are as follows: Firstly, the present invention uses time-frequency analysis method to transform the problem of communication IQ signal system identification into an image recognition problem based on deep learning. By using unsupervised learning method, it solves the problem of insufficient effective sample quantity of specific ultra-short wave system signals under actual conditions, improves the utilization rate of unlabeled signal data in the real battlefield environment, and realizes the intelligent identification of specific ultra-short wave system signals by using the method of large-scale unsupervised learning pre-training + small-scale supervised learning fine-tuning. It can effectively solve the common problem of poor generalization of intelligent algorithms in the field of communication reconnaissance, and has strong engineering practical value. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 It is a flow chart of an intelligent identification method for ultra-short wave signal system based on Masked auto-encoder of the present invention;

[0014] Figure 2 It is a narrowband time-frequency diagram of a certain navigation signal in the ultra-short wave band in this embodiment;

[0015] Figure 3 It is a schematic diagram of the Masked AutoEncoder model architecture for unsupervised learning in this embodiment;

[0016] Figure 4 It is a schematic diagram after random masking processing of the narrowband time-frequency diagram of a certain navigation signal in the ultra-short wave band in this embodiment;

[0017] Figure 5 It is a schematic diagram of the structure of the bidirectional Transformer module in this embodiment;

[0018] Figure 6 It is a schematic diagram of the Masked AutoEncoder model architecture for supervised learning in this embodiment. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0019] The technical solution of the present invention will be further described below with reference to the accompanying drawings.

[0020] As Figure 1 shown, an intelligent identification method for ultra-short wave signal system based on Masked auto-encoder of the present invention includes the following steps:

[0021] S1. Obtain the outgoing time, center frequency and bandwidth of the intercepted ultra-short wave signal, and preprocess the signal according to the outgoing time, center frequency and bandwidth of the intercepted ultra-short wave signal. The preprocessed ultra-short wave narrowband signal must contain the content of the synchronization frame and part of the data frames of the signal. The preprocessing includes time-domain interception, mixing and down-conversion; Firstly, intercept the original signal in the time domain according to the start time of the synchronization frame and the end time of the data frame, then mix according to the center frequency of the target signal to ensure that the signal is at zero frequency, and finally perform down-conversion according to the down-sampling multiple.

[0022] In this example, a communication signal receiving device is used to obtain ultra-short wave band broadband communication IQ data

[0023] S = {S1, S2, S3, … S i , …, S n}, where n is the total number of sampling points within a certain sampling time range, the sampling rate is fs, and the center frequency is F c . After digital channelization and signal detection, the emergence times of the complete target signals are t1 and t2, the center frequency is fc, and the bandwidth is bw;

[0024] First, intercept the ultra-short wave signal IQ data in the time domain according to the emergence time of the target signal. The starting sampling point and the ending sampling point positions of the interception are respectively and Then calculate the frequency difference |Fc - fc| between the center frequency of the broadband signal and the center frequency of the target signal. Shift the frequency of the target signal to zero frequency through mixing technology. Finally, calculate the optimal downsampling multiple to construct a low-pass filter to achieve downconversion and obtain a standard ultra-short wave narrowband IQ signal.

[0025] Then perform a short-time Fourier transform on the preprocessed ultra-short wave narrowband signal to construct a time-frequency map dataset of ultra-short wave narrowband signals for unsupervised learning; The specific method is: for the preprocessed ultra-short wave narrowband IQ signal, select appropriate window functions, window lengths, and step sizes to perform a short-time Fourier transform, and extract the time-frequency map features of the signal, such as Figure 2 shown, which is the time-frequency map of a certain navigation signal in the ultra-short wave band. The short-time Fourier transform (STFT) is a key analysis tool in the field of signal processing. Its advantage lies in its localization characteristics. When the traditional Fourier transform processes the entire signal, it cannot distinguish the frequency characteristics of the signal at different time points. However, through setting a short-time window, STFT can intuitively display the parameter change trends and characteristics of the signal, and can characterize parameters such as different modulation methods, synchronization frame styles, time slots, and bandwidths of ultra-short wave signals with different systems through image textures, edge shapes, color gamut distributions, etc., and is suitable for signal analysis of non-stationary features or multi-frequency components.

[0026] S2. Construct a Masked AutoEncoder ultra-short wave system intelligent recognition model, and use the time-frequency map dataset of ultra-short wave narrowband signals for unsupervised learning pre-training;

[0027] An Autoencoder (AE) is a special neural network structure used to achieve non - linear compression, reconstruction of data, and learning of latent representations. The core idea is to use a neural network model to attempt to reconstruct its input and self - train only using this information. An autoencoder usually consists of two parts: an encoder and a decoder. The encoder compresses high - dimensional data into a lower - dimensional hidden representation space through a series of non - linear transformations. In this process, the encoder captures the key features of the input data and expresses them in a concise form. Therefore, the last output layer of the encoder is called the "latent vector", whose size is much smaller than the size of the original input. The decoder then restores the latent vector to the original - dimensional space and attempts to accurately reconstruct the original input data as much as possible. The decoding process is a mapping process from low - dimension to high - dimension, which to a certain extent simulates the behavior of a generative model. The training objective of the autoencoder is to minimize the reconstruction error, that is, the distance or similarity measurement value (such as mean squared error, cross - entropy, etc.) between the input data and the output data of the decoder. During the training process, the network tries to find a way to preserve key information during the compression into the latent vector so that when it is passed to the decoder, the decoder can restore the original input as much as possible. Generally speaking, as a tool for unsupervised learning, the main purpose of the autoencoder is to extract effective feature representations from the original input without relying on labels and verify the applicability of these features through the decoding process. Through training, we can discover the latent structure, reduce the data complexity, and improve the effect of subsequent supervised tasks.

[0028] Therefore, the present invention proposes to construct a specific ultra - short - wave system signal recognition model based on the Masked AutoEncoder model. Through the idea of reconstruction learning in unsupervised learning, a reliable pre - trained model is trained using the time - frequency diagrams of a large number of unlabeled ultra - short - wave system signals. Then, the encoding layer part of the pre - trained Masked AutoEncoder model is used as a feature extractor, and a fully - connected layer is connected as an output classification layer to construct a specific ultra - short - wave signal system recognition model. It is trained using a time - frequency diagram data set of specific - system ultra - short - wave signals with labels to overcome the problem of model over - fitting caused by insufficient sample numbers of specific ultra - short - wave system signals in the real electromagnetic environment, and to achieve the recognition of specific ultra - short - wave system signals under low - signal - to - noise ratio conditions. The Masked AutoEncoder ultra - short - wave system intelligent recognition model of the present invention sequentially includes a random masking layer, an encoding layer, a decoding layer, and a classification output layer, and its structure is as Figure 3 shown; The present invention optimizes the random masking layer and the encoding layer, and the details are as follows:

[0029] (1) The random masking layer in the Masked AutoEncoder model of the present invention draws on and optimizes the original masking mechanism. Since the distinguishable information of different ultra-short wave systems is reflected in the shape of the synchronization frame and the texture of the data frame in the time-frequency diagram, the random masking with a single fixed probability is very likely to cover the key image areas of the synchronization frame and the data frame, resulting in the AutoEncoder model being unable to effectively reconstruct and learn the signal, seriously affecting the convergence speed and reconstruction accuracy of the model. Therefore, the present invention first evenly divides the ultra-short wave narrowband time-frequency diagram into multiple pixel blocks in the horizontal and vertical directions, then randomly masks the divided pixel blocks according to three probabilities of 0.25 / 0.5 / 0.75 respectively, and finally sequentially extracts the unmasked parts in the pixel blocks of the original ultra-short wave narrowband time-frequency diagram processed according to the three random masking probabilities, randomly shuffles them, resizes them in the image height dimension, and then splices them in the channel dimension, and sends them to the encoding layer for feature extraction and compression, as Figure 3 shown in the random masking mechanism. In this embodiment, the input original image size of the random masking layer is 224*224, the number of masking blocks in the random masking layer is 8*8, and the size of a single masking block is 28*28; the random masking layer evenly divides the 224*224 time-frequency diagram into 8*8 pixel blocks in the horizontal and vertical directions, and the size of each block is 28*28 (224 / 8 = 28). The schematic diagram of the random masking process of a certain navigation signal narrowband time-frequency diagram in the ultra-short wave band in this embodiment is as Figure 4 shown. This division method can ensure that the masking operation can cover the key areas in the time-frequency diagram (such as the synchronization frame and the data frame texture), while reducing computational redundancy. Using the random masking layer can reduce the redundant information in the ultra-short wave narrowband time-frequency diagram on the one hand, only learn the non-masked pixel blocks, reducing the computational amount by up to 75% at most, and on the other hand, solve the problem that traditional convolutional neural networks can only learn local features of data, forcing the model to learn the global features of the ultra-short wave narrowband time-frequency diagram and improving the model's reconstruction and learning ability for ultra-short wave signal images.

[0030] (2) The encoding layer includes multiple cascaded bidirectional Transformer modules, and each bidirectional Transformer module sequentially includes an input layer, a positional encoding, a multi-head self-attention mechanism, and a feed-forward neural network.

[0031] 1) The input layer includes two processes: word embedding and character embedding; word embedding maps the vocabulary to a high-dimensional vector space to capture the semantic information of the words; character embedding encodes each character in the word. In ultra-short wave signal processing, the pixel blocks of the time-frequency diagram are regarded as "words", and individual pixels or local features are regarded as "characters". Word embedding maps each pixel block to a high-dimensional vector, and character embedding captures the local details within the pixel block (such as texture, edges);

[0032] 2) Position encoding, which is used to add position encoding to the input word embeddings. Since the Transformer does not contain a recurrent structure, in order to introduce sequence information into the model, it is necessary to add position encoding to the input word embeddings. Common methods include absolute position encoding and relative position encoding, which represent the position of a word relative to the absolute position of the entire sequence or the distance relative to the context words, respectively.

[0033] 3) Multi-Head Self-Attention mechanism: The multi-head self-attention mechanism processes the time-frequency image pixels in two directions, the forward sequence (from left to right) and the reverse sequence (from right to left), to obtain the forward feature vector and the reverse feature vector respectively. Forward processing can capture the signal evolution characteristics in the time dimension, and reverse processing can capture the symmetry characteristics in the frequency dimension. The feature vectors in the two directions are fused through the multi-head self-attention mechanism. Through the bidirectional mechanism, by splicing the outputs in the two directions, the model's ability to express the global features of the time-frequency diagram can be enhanced. In the multi-head self-attention mechanism, the output of each head is calculated through the scaled dot-attention mechanism, and finally, it is spliced into a complete feature through a linear transformation (Output weights); in order to improve the model's learning ability for global features, the present invention captures the dependence relationship between the feature vectors of the time-frequency image pixels of the ultra-short wave signal through the self-attention mechanisms in the two directions, and the structure of the self-attention mechanism is as Figure 5 shown. In this embodiment, the encoder is stacked by 6 identical bidirectional Transformer modules.

[0034] The multi-head self-attention mechanism divides the input feature vector into multiple "heads" (or channels), and each "head" will process the features in both the forward feature vector and the reverse feature vector directions simultaneously. Calculate an attention score for the features in the two directions respectively, and then perform weighted summation on these attention scores through the scaled dot-attention mechanism layer to obtain a new vector representation. This design can simultaneously focus on information at different scales. Then, at the splicing layer, the outputs of all heads are spliced to form a feature representation that fuses bidirectional information. Then, the feature representation is mapped to the target dimension through the linear transformation matrix of the self-attention output.

[0035] Each head is transformed by three matrices Query, Key, and Value, and the similarity is obtained by calculating the dot product of Query and Key divided by sqrt(d). d is the dimension of the feature vector (such as the vector dimension after embedding each pixel block). When calculating the attention score, the dot product result is scaled by dividing by sqrt(d) to prevent gradient vanishing or explosion. Then, multiplying the similarity by the Value matrix is the output of each head.

[0036] 4) Position-wise Feed-Forward Networks: It contains two consecutive fully-connected layers, and ReLU or GELU activation functions are usually used in the fully-connected layers.

[0037] 5) Residual connection and normalization operation (Layer Normalization): Add a normalization layer before and after the position-wise feed-forward network; perform residual addition on the input and output of the multi-head self-attention mechanism and then input the result into the first normalization layer, and the output of the first normalization layer is used as the input of the position-wise feed-forward network; perform residual addition on the input and output of the position-wise feed-forward network and then output it to the second layer normalization layer.

[0038] The decoding layer adopts the structure of the multi-head attention mechanism in Vision Transformer, and the output of the decoding layer is the reconstructed image.

[0039] The size of the classification output layer is the number of specific ultra-short wave communication signal systems.

[0040] S3. According to the known ultra-short wave signals of the system, perform preprocessing and time-frequency transformation according to the process in step S1 to construct a time-frequency map dataset of ultra-short wave narrowband signals for supervised learning.

[0041] S4. Fine-tune the pre-trained ultra-short wave system intelligent recognition model and train it using the dataset constructed in step S3; use the dataset of ultra-short wave signals of a specific system with labels for model fine-tuning training and testing. This method adopts the multi-task learning method to jointly train the ultra-short wave narrowband time-frequency map mask reconstruction learning task and the ultra-short wave specific system recognition task to achieve better performance than training the ultra-short wave specific system recognition task alone. Among them, in the fine-tuning stage, the model simultaneously performs the reconstruction task and the classification task; the reconstruction task restores the masked time-frequency map through the decoding layer, and the classification task predicts the signal system category through the newly added fully-connected layer. The loss functions of the two tasks are dynamically adjusted by weights α and β, and the optimization goal is to minimize the total loss L_Total; as Figure 6 shown;

[0042] The specific method of fine-tuning is: Connect a Dropout layer and a fully-connected layer after the output of the encoding layer as the branch network for the classification task, where the number of neurons in the fully-connected layer is the number of specific ultra-short wave systems.

[0043] In the fine-tuning stage, the trainable parameters of the first 3 bidirectional Transformer modules in the encoding layer are frozen, and a reconstruction learning branch and a classification learning branch are constructed for adaptive joint training. Among them, the mean squared error (MSE) loss function is used for the reconstruction learning loss function, and the cross-entropy loss function after label smoothing is used for the classification loss function. Label smoothing is a regularization technique used in deep learning classification tasks, mainly used to alleviate the overconfidence of the model in a single category. It creates a soft target by smoothing between the actual class label and a uniform distribution to replace the original hard target y hot : where K is the number of specific ultra-short wave system types; δ is a hyperparameter, which takes the value of 0.1 in the present invention. Label smooth can improve the generalization performance of the model in the case of a small dataset. The generalization of the algorithm is improved by combining self-supervised learning and supervised learning; the loss function formula of the model in the fine-tuning stage is as follows:

[0044] L Total = αL MSE + βL LS

[0045] where L Total is the total loss function, α and β are the weights of the reconstruction loss and the classification loss function respectively, and the weights of the two tasks are iteratively optimized through model training to achieve the global optimal effect; L MSE is the root mean square error between the reconstructed ultra-short wave time-frequency map and the original time-frequency map, and L LS is the classification loss function after label smoothing

[0046] S5. Use the trained Masked AutoEncoder ultra-short wave system intelligent recognition model to identify the system of specific ultra-short wave communication signals

[0047] Those of ordinary skill in the art will realize that the embodiments described herein are to assist the reader in understanding the principles of the present invention, and it should be understood that the protection scope of the present invention is not limited to such specific statements and embodiments. Those of ordinary skill in the art can make various other specific deformations and combinations without departing from the essence of the present invention based on the technical revelations disclosed in the present invention, and these deformations and combinations are still within the protection scope of the present invention

Claims

1. An intelligent recognition method for ultra-short wave signal systems based on Masked Autoencoders, characterized in that, It includes the following steps: S1. Preprocess the signal according to the outgoing time, center frequency, and bandwidth of the intercepted ultra-short wave signal. The preprocessed ultra-short wave narrowband signal must contain the content of the signal's synchronization frame and part of the data frame; Then perform short-time Fourier transform on the preprocessed ultra-short wave narrowband signal to construct an ultra-short wave narrowband signal time-frequency map dataset for unsupervised learning; S2. Construct a Masked AutoEncoder ultra-short wave system intelligent recognition model and perform unsupervised learning pre-training using the ultra-short wave narrowband signal time-frequency map dataset; The Masked AutoEncoder ultra-short wave system intelligent recognition model successively includes a random masking layer, an encoding layer, a decoding layer, and a classification output layer; S3. According to the ultra-short wave signal with known system, perform preprocessing and time-frequency transformation according to the process in step S1 to construct an ultra-short wave narrowband signal time-frequency map dataset for supervised learning; S4. Fine-tune the pre-trained ultra-short wave system intelligent recognition model and train it using the dataset constructed in step S3; S5. Use the trained Masked AutoEncoder ultra-short wave system intelligent recognition model to identify the system of specific ultra-short wave communication signals.

2. The intelligent recognition method for ultra-short wave signal systems based on Masked autoencoders according to claim 1, wherein The preprocessing in step S1 includes time-domain interception, mixing, and down-conversion; First, intercept the original signal in the time domain according to the start time of the synchronization frame and the end time of the data frame, then perform mixing according to the center frequency of the target signal to ensure that the signal is at zero frequency, and finally perform down-conversion according to the down-sampling ratio.

3. The intelligent identification method for ultra-short wave signal systems based on Masked autoencoders according to claim 1, characterized in that The random masking layer first evenly divides the ultra-short wave narrowband time-frequency map into multiple pixel blocks in the horizontal and vertical directions, then randomly masks the divided pixel blocks with three probabilities of 0.25 / 0.5 / 0.75 respectively. Finally, randomly shuffle the unmasked parts in the pixel blocks of the original ultra-short wave narrowband time-frequency map processed with the three random masking probabilities, perform Resize in the image height dimension, and then splice them in the channel dimension, and send them to the encoding layer for feature extraction and compression.

4. The intelligent recognition method for ultra-short wave signal systems based on Masked autoencoders according to claim 1, characterized in that, The encoding layer includes multiple cascaded bidirectional Transformer modules, and each bidirectional Transformer module successively includes an input layer, position encoding, multi-head self-attention mechanism, and feed-forward neural network; 1) The input layer includes two processes: word embedding and character embedding; Word embedding is to map the vocabulary to a high-dimensional vector space to capture the semantic information of the word; Character embedding is to encode each character in the word; In the processing of ultra-short wave signals, the pixel blocks of the time-frequency map are regarded as "words", and individual pixels or local features are regarded as "characters"; 2) Position encoding is used to add position encoding to the input word embedding; 3) Multi-Head Self-Attention Mechanism: Process the time-frequency image pixels in two directions, the forward sequence and the reverse sequence, to obtain the forward feature vector and the reverse feature vector respectively; the multi-head self-attention mechanism divides the input feature vector into multiple "heads", and each "head" processes the features in both the forward and reverse directions simultaneously; calculate an attention score for the features in both directions respectively, and then weight and sum these attention scores through a scaled dot-attention mechanism layer to obtain a new vector representation; then concatenate the outputs of all heads in the concatenation layer to form a feature representation that fuses bidirectional information. 4) Feed-Forward Neural Network: Consists of two sequentially connected fully connected layers; add a normalization layer before and after the feed-forward neural network; add the input and output of the multi-head self-attention mechanism through residual addition and then input it into the first normalization layer, and the output of the first normalization layer is used as the input of the feed-forward neural network; add the input and output of the feed-forward neural network through residual addition, and then output it to the second layer normalization layer.

5. The intelligent identification method for ultra-short wave signal systems based on Masked autoencoder according to claim 3, wherein The specific method of fine-tuning in step S4 is: connect a Dropout layer and a fully connected layer after the output of the encoding layer as the branch network for the classification task, where the number of neurons in the fully connected layer is the number of specific ultra-short wave systems. In the fine-tuning stage, freeze the trainable parameters of the first 3 bidirectional Transformer modules in the encoding layer, and construct a reconstruction learning branch and a classification learning branch for adaptive joint training. The reconstruction learning loss function uses the MSE loss function, and the classification loss function uses the cross-entropy loss function after label smoothing; the loss function formula of the model in the fine-tuning stage is as follows: L Total = αL MSE + βL LS where L Total is the total loss function, α and β are the weights of the reconstruction loss and the classification loss functions respectively. The weights of the two tasks are iteratively optimized through model training to achieve the global optimal effect; L MSE is the root mean square error between the reconstructed ultra-short wave time-frequency map and the original time-frequency map, and L LS is the classification loss function after label smoothing.

Images