Open set modulation identification method based on time-frequency domain feature learning and fusion

Abstract

The invention belongs to the technical field of cognitive radio, and particularly relates to an open set modulation identification method based on time-frequency domain feature learning and fusion. The method comprises the following steps of: receiving a signal by adopting a receiving antenna through electromagnetic spectrum monitoring equipment, capturing a space electromagnetic radiation signal, converting the space electromagnetic radiation signal into an electric signal, amplifying the signal through a low-noise amplifier, performing frequency conversion and filtering through an analog receiver, and converting the received signal into an intermediate-frequency signal. Then, the signal is digitalized through AD sampling, and the intermediate frequency signal is converted into a baseband complex signal through frequency mixing and digital filtering in a digital receiver. Then performing time domain characterization calculation and frequency domain characterization calculation on the signal sampling data to form a plurality of time domain characterization vectors and a plurality of frequency domain characterization vectors; and then the vectors are sent to a trained deep neural network model, reasoning is carried out by using the network model, a reasoning result is post-processed, and a signal modulation identification result is finally obtained. According to the invention, the recognition accuracy of unknown signals is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of cognitive radio, and particularly relates to an open-set modulation recognition method based on time-frequency domain feature learning and fusion. Background Technique

[0002] Radio monitoring plays an important role in multiple fields such as the Internet of Things (IoT), spectrum management, electronic warfare, and interference recognition. Electromagnetic signal parameter estimation is a basic function of radio monitoring systems, and the signal modulation mode, as an important parameter, is crucial for the blind signal recognition and blind demodulation functions of radio monitoring systems in non-cooperative wireless environments. To effectively identify signal modulation methods, a key technology called automatic modulation recognition is applied to radio monitoring systems. Generally, automatic modulation recognition is divided into two major categories: rule-based methods and deep learning-based methods. Rule-based methods use pre-involved features and rules for classification, and these methods are often inefficient and not very robust. With the development of artificial intelligence technology, deep learning methods have begun to be introduced into the field of wireless communication for various tasks such as classification, regression, and decision-making. Currently, various deep learning-based automatic modulation recognition methods have been proposed based on structures such as convolutional neural networks (CNNs), recurrent neural networks (RNNs), long short-term memory (LSTM) networks, and complex-valued CNNs.

[0003] Recent deep learning-based automatic modulation recognition methods mainly focus on closed-set classification techniques. However, since radio monitoring systems operate in open wireless environments, these methods encounter limitations in practical applications. First, automatic modulation recognition is an open-set classification problem. In a real wireless environment, the known dataset may not contain all modulation types. Second, for signals of the same modulation and bandwidth, the center frequency and relative sampling rate of the received signal data may vary in different receiving environments. This frequency offset is caused by the Doppler effect. At the same time, in radio monitoring systems, the sampling rate is usually dynamically adjusted according to the estimated signal bandwidth. This adjustment may lead to sampling rate fluctuations, usually due to inaccurate bandwidth estimation, which is a common problem in real-world applications. Therefore, achieving accurate classification under signal examples with different frequency offsets and different sampling rates is a basic ability of automatic modulation recognition methods. Summary of the Invention

[0004] The object of the present invention is to provide a method that can simplify the operation of monitoring equipment, has a fast signal recognition processing speed, a high signal recognition accuracy rate, and can effectively detect unknown signals, aiming at the problems of low signal recognition accuracy rate in existing electromagnetic spectrum monitoring equipment when there are frequency offsets and digital sampling rate offsets in the received signal, and the inadaptability or low detection rate of unknown signal detection.

[0005] The technical solution of the present invention is:

[0006] An open-set modulation recognition method based on time-frequency domain feature learning and fusion, comprising the following steps:

[0007] S1. A broadband electromagnetic spectrum monitoring device receives and captures spatial electromagnetic radiation signals through a receiving antenna and converts them into electrical signals, converts the electrical signals into intermediate-frequency signals through an analog receiver, and then converts the intermediate-frequency signals into baseband complex signals through a digital receiver;

[0008] S2. Convert the baseband complex signal into a time-domain representation and a frequency-domain representation, and each representation is output in the form of a vector. Denote the time-domain representation vector as , where n is the serial number of the time-domain representation result, N is the number of points in the complex signal sequence calculated this time, i represents the serial number of the time-domain representation, and the calculation formula of the time-domain representation is:

[0009] ,

[0010] where, represents the input baseband complex signal sequence, represents the calculation of taking the real part, represents the calculation of taking the imaginary part, represents the calculation of taking the amplitude, represents the calculation of taking the phase, represents the calculation of taking the first-order difference;

[0011] Denote the frequency-domain representation vector as , where k is the serial number of the frequency-domain representation result, and the calculation formula of the frequency-domain representation is:

[0012] ,

[0013] where, represents the fast Fourier transform, represents the element-wise complex multiplication calculation of the complex sequence, and represent the element-wise complex 8th power and 16th power calculations of the complex sequence;

[0014] S3. Construct a modulation feature extraction neural network, which includes two branches with exactly the same structure. The processing of the input data for each branch is as follows: Use a one-dimensional ResNet convolutional neural network to process the input multi-channel feature vector. Through multiple layers of convolution and pooling operations, a tensor is formed. Then, the dimension of the tensor is transformed to obtain a vector sequence with a length of 256 and a single vector dimension of 512, which is defined as the embedded feature Token sequence. Add a class Token with an initial value of 0 and a dimension of 512 at the first position in the Token sequence, and then add the position encoding Token. The sequence number and dimension of the position encoding Token are the same as those of the input Token, obtaining the marked sequence with added position encoding. Send the marked sequence into the Transformer encoder for processing. The Transformer encoder is formed by stacking multiple attention mechanism layers. The processing of the Transformer encoder does not change the length and dimension of the Token sequence. After being processed by the Transformer encoder, the processed Token sequence is obtained. In the processed Token sequence, the first Token is the class token processed by the encoder;

[0015] After the time-domain feature vector and the frequency-domain feature vector are respectively processed by the two branches in the neural network, two processed Token sequences are obtained. Take the first Token of the two processed Token sequences as the time-domain class token and the frequency-domain class token respectively, and splice the time-domain class token and the frequency-domain class token to obtain the modulation feature vector;

[0016] S4. Perform signal classification and unknown class recognition based on the obtained modulation feature vector, specifically:

[0017] Construct different group classifiers according to different modulation methods , where k represents the k-th group classifier. Each group classifier uses a fully connected neural network to process the input feature vector to form a Logtis value. In each group classifier, according to the Logtis value, use the Softmax layer for probability normalization and use the energy calculation function to calculate the energy of the Logtis value respectively. Among them, the probability normalization predicts the class probability of the modulation methods within the group, and the energy of the Logtis value determines whether the input sample is the modulation method within the group; The calculation formula of the Logtis value is:

[0018] ,

[0019] where is the input modulation feature vector, , are the weight parameters and bias parameters in the fully connected network, That is, the calculated Logtis value, where c is the number of modulation methods within the corresponding group;

[0020] The process of probability normalization by the Softmax layer is as follows:

[0021]

[0022] Where and represent the Logits value and the normalized probability value of class c in group ; represents the summation over all classes belonging to the class set ; represents the Logits value of class i during the summation process;

[0023] For group k, the calculation method of the energy value is as follows:

[0024]

[0025] where T is the temperature parameter;

[0026] After each group classifier calculates its corresponding class probability value and energy value, final class judgment and detection of unknown classes are performed through post-processing; the post-processing method is that for the modulation feature vector v, first, it is determined whether the sample belongs to the group class modulation based on the energy value, and the rule is:

[0027] ,

[0028] where is the decision threshold;

[0029] If a sample is determined not to belong to the corresponding group after energy value judgment in all group classifiers, then the sample is determined to be an unknown class; for each group classifier, if it is determined that the sample belongs to this group according to the energy value, then the maximum predicted probability in this group is output, along with its corresponding class serial number

[0030] S5. After obtaining the baseband complex signals of known categories using the method of S1, sample signals are obtained through preprocessing. After processing the sample signals using the method of S2, the modulated feature extraction neural network of S3 is input to obtain the modulated feature vectors for training. The group classifier of S4 is trained using the modulated feature vectors for training. The loss functions used in training include cross-entropy loss and energy loss. For a single group The cross-entropy loss :

[0031]

[0032] where represents the probability value that the input sample is predicted as category c in group , and represents that the true category of the sample in group is c; the cross-entropy of each group is calculated independently and then the losses of all groups are summed as the cross-entropy loss of this sample. Then, the average of the cross-entropy losses of all samples is used as the cross-entropy loss during training. Therefore, the total cross-entropy loss is denoted as:

[0033]

[0034] For the energy loss of a single group , its calculation method is:

[0035]

[0036] where represents the energy output by the classifier for the input sample, and is a hyperparameter during training, meaning that if a sample does not belong to group , then in the classifier corresponding to this group, its energy value should be less than ; the energy loss value of each group is calculated independently and then the losses of all groups are summed as the energy loss of this sample. Then, the average of the energy losses of all samples is used as the energy loss during training. Therefore, the total energy loss is denoted as:

[0037]

[0038] The total loss during training is composed of the weighted sum of the two losses, and the calculation method is:

[0039]

[0040] where is a hyperparameter that controls the weighted ratio of the two losses;

[0041] After training, a trained group classifier is obtained;

[0042] S6. After processing the obtained electromagnetic radiation signal in the manner of S1 - S3, a modulation feature vector is obtained, and then it is input into the trained group classifier for recognition.

[0043] Furthermore, the preprocessing described in S5 is data augmentation processing, including adding random frequency offset, adding random Gaussian noise, and adding random sampling rate transformation to the signal data;

[0044] The enhancement method of adding random frequency offset is:

[0045] , n = 0, 1, 2, …, N - 1,

[0046] where is the random frequency offset value added on the basis of the original signal, is the sampling rate, is the enhanced sample, and N is the number of signal sampling points in the sample;

[0047] The method of adding Gaussian noise is:

[0048]

[0049] where is a complex Gaussian noise with zero mean and variance of ;

[0050] The method of adding resampling is:

[0051]

[0052] where is the resampling function, and its resampling operation is controlled by two parameters: the original sampling rate and the resampled sampling rate, is the original sampling rate, is the resampled sampling rate.

[0053] The beneficial effects of the present invention are as follows: it simplifies the operation of electromagnetic spectrum monitoring equipment and improves the signal recognition speed ability. The present invention is directed to the automatic modulation recognition of magnetic radiation signals by electromagnetic spectrum monitoring equipment. The electromagnetic spectrum monitoring equipment uses a broadband antenna to receive signals, automatically amplifies, analog-converts the frequency, and filters the signals through a radio frequency channel and outputs them to a digital receiving module; the digital receiving automatically performs AD sampling, digital down-conversion, filtering, and decimation on the intermediate-frequency analog signals output from the receiving channel to form a baseband complex digital signal, and then automatically performs various preset time-domain characterizations and frequency-domain characterizations on the baseband complex digital signal to form time-frequency domain characterization data of the signal; the deep neural network model automatically processes the time-frequency domain characterization data to form a prediction result of the neural network; the post-processing model automatically performs judgment processing according to set rules on the pre-processing result output by the neural network to form a final signal category recognition result for output. The entire process does not require manual setting of judgment thresholds and operation of equipment. The machine automatically extracts signal features and judges the signal type, greatly simplifying the design of the monitoring equipment, reducing manual operation, and improving the reliability of engineering implementation.

[0054] It improves the recognition accuracy rate of the signal modulation method by the monitoring equipment. The present invention uses time-frequency domain multi-domain characterization instead of sending baseband dual-channel data into the neural network for processing, and the signal representation method is more abundant. At the same time, combined with a specially designed neural network model based on a dual-transformer structure, it improves the automatic grasping ability of modulation-related features, thereby improving the recognition accuracy rate of the signal modulation method.

[0055] It improves the recognition ability of the monitoring equipment for signals with Doppler frequency offset and sampling rate deviation. The present invention uses various signal time-domain and frequency-domain characterization methods. In the case of signals with Doppler frequency offset and sampling rate deviation, these characterization methods can still correctly represent the modulation characteristics of the signals. Combined with a specific data enhancement mechanism, the deep neural network model learns and processes these characterizations, grasps the modulation characteristics related to anti-frequency offset and anti-sampling rate deviation, and uses a classifier to process the features and form a classification result, improving the signal recognition accuracy rate in signal environments and receiving conditions such as Doppler frequency offset and sampling rate deviation.

[0056] It improves the recognition accuracy rate of unknown signals. The present invention adopts a mechanism combining a clustering classifier and energy constraint. Without using training samples of unknown signals, the neural network can be trained to achieve the detection of unknown category signals. By using a clustering classifier and a comprehensive decision-making mechanism in post-processing, the recognition of unknown categories is realized. Description of the Drawings

[0057] Figure 1 It is a schematic flow diagram of an open-set modulation recognition method based on time-frequency domain feature learning and fusion according to the present invention.

[0058] Figure 2It is the schematic diagram of the neural network structure for extracting modulation features of the present invention.

[0059] Figure 3 It is the schematic diagram of the post-processing for signal classification and unknown class recognition based on feature vectors in the present invention.

[0060] Figure 4 It is the schematic diagram of the training of the neural network for signal modulation mode recognition in the present invention. Specific implementation manners

[0061] The practicality of the present invention will be described below with reference to the accompanying drawings.

[0062] As Figure 1 shown, the process of the method of the present invention is as follows: In the electromagnetic spectrum monitoring scenario, a broadband electromagnetic spectrum monitoring device uses a receiving antenna to receive signals, captures spatial electromagnetic radiation signals and converts them into electrical signals, amplifies the signals through a low-noise amplifier in the analog receiver, and through analog frequency conversion and filtering, converts the received signals into intermediate-frequency signals. Then, in the digital receiver, the signals are digitized through AD sampling, and the intermediate-frequency signals are converted into baseband complex signals by using digital mixing, digital filtering, and decimation. Then, time-frequency domain characterization calculations are performed on the sampled data of the baseband signals, including 5 kinds of time-domain characterization calculations and 5 kinds of frequency-domain characterization calculations, and each characterization is output in the form of a vector. In the present invention, the time-domain characterizations adopted mainly include: the real part of the input complex signal, the imaginary part, the instantaneous amplitude of the signal, the instantaneous phase, and the instantaneous frequency. The 5 kinds of time-domain characterizations are denoted as: . n is the serial number of the time-domain characterization result, N is the number of points of the complex signal sequence calculated this time. i represents the serial number of the time-domain characterization. The calculation formula of the time-domain characterization is expressed as:

[0063] (1)

[0064] Where represents the input complex signal sequence, represents the calculation of finding the real part, represents the calculation of finding the imaginary part, represents the calculation of finding the amplitude, represents the calculation of finding the phase, represents the calculation of finding the first-order difference.

[0065] The frequency-domain features adopted mainly include: signal spectrum, quadratic spectrum, quartic spectrum, octic spectrum, and sixteenth-power spectrum. The 5 kinds of frequency-domain characterizations are denoted as: , k is the serial number of the frequency-domain characterization result. i represents the serial number of the frequency-domain characterization. The calculation formula of the frequency-domain characterization is expressed as:

[0066] (2)

[0067] Where denotes the Fast Fourier Transform, represents the element-wise complex multiplication calculation of complex sequences, and represents the element-wise complex 8th power and 16th power calculations of complex sequences. Ten characterization vectors are fed into the trained neural network model for inference, and the predicted probability results of each group are output according to different groups. The post-processing module comprehensively evaluates the classification probability prediction results of multiple groups, and finally outputs the signal modulation mode recognition result and the unknown signal detection result.

[0068] such as Figure 2 shown, the modulation feature extraction neural network structure is as follows: Its overall structure is to use two neural network branches with the same structure to process the time-domain characterization vectors and frequency-domain characterization vectors respectively, and form time-domain class tokens and frequency-domain class tokens respectively. These two tokens are concatenated to form a modulation feature vector for subsequent classifier processing. The two branches adopt the same neural network structure. Taking the time-domain characterization processing branch as an example, the structure of a single branch is specifically as follows: The size of the input 5-channel characterization vector is 5×W i (typically 5×8192 can be taken). First, a one-dimensional ResNet convolutional neural network is used to process the input multi-channel characterization vector. Through multi-layer convolution and pooling processing, a C f ×W f (typically 512×256 can be taken) tensor is formed. Then, the tensor is dimensionally transformed to form a vector sequence with a length of 256 and a single vector dimension of 512, which is called the time-domain embedded characterization Token sequence. At the first position in this Token sequence, a class Token initialized to 0 with a dimension of 512 is added, and then the position encoding Token (the sequence number and dimension of the position encoding Token are the same as those of the input Token) is added to form a marked sequence with position encoding added. This marked sequence is fed into the Transformer encoder for processing. The encoder is formed by stacking multiple attention mechanism layers. The encoder processing does not change the length and dimension of the Token sequence. After encoder processing, the processed Token is formed. In the processed Token sequence, the first Token is the time-domain class token processed by the encoder (with a dimension of 512). The frequency-domain branch adopts the same structure to process the multi-channel frequency-domain characterization vectors, and the first Token is taken as the frequency-domain class token processed by the encoder (with a dimension of 512). The time-domain class token and the frequency-domain class token are merged and concatenated to form a modulation feature vector (with a dimension of 1024).

[0069] such as Figure 3As shown in the figure, the post-processing for signal classification and unknown class recognition based on feature vectors is designed as follows: Based on the input feature vectors, signal classification and unknown class recognition are performed by means of clustering discrimination. Specifically, clustering is carried out according to different major categories such as different ASK, PSK, MSK, QAM, etc., as follows:

[0070] ●G1: {2ASK, 4ASK};

[0071] ●G2: {BPSK, QPSK, OQPSK, π / 4DQPSK, 8PSK};

[0072] ●G3: {2FSK, 4FSK, MSK};

[0073] ●G4: {16QAM, 32QAM};

[0074] For each group, a fully connected neural network is used to process the input feature vectors to form Logtis values, which are used for processing two branches. The first branch is to use the Softmax layer for probability normalization to form the class probability prediction of the modulation methods within the group. Assume the input feature vector is , where d is the dimension of the feature vector. For each group 's classifier, it first calculates the Logtis value using a fully connected layer, and its formula can be expressed as:

[0075] (3)

[0076] where , are the weight parameters and bias parameters in the fully connected network. That is, the calculated Logtis value, c is the number of modulation methods within the group. For example, c in group G1 is 2. The Softmax layer performs probability normalization, and its calculation process is:

[0077] (4)

[0078] and represent the Logits value and the normalized probability value of class c in group . The second branch of the group classifier is to calculate the energy of the Logtis value using an energy calculation function to determine whether the input sample is a modulation method within the group. For group k, its energy value is calculated as follows:

[0079] (5)

[0080] where T is the temperature parameter, and a typical value can be set to 5.0.

[0081] After each group classifier calculates its corresponding class probability value and energy value, final class judgment and detection of unknown classes are performed through post-processing. The post-processing method is as follows: For the feature vector v, first, it is determined whether the sample belongs to the group class modulation based on the energy value. The rule is:

[0082] (6)

[0083] where is the decision threshold. If a sample does not belong to the corresponding group after energy value judgment in all group classifiers, the sample is determined to be an unknown class. For each group classifier, if it is determined that the sample belongs to this group according to the energy value, the maximum predicted probability value in this group, and its corresponding class serial number are output as the recognition result of the effective modulation method of this group. If multiple groups all output effective modulation method recognition results, then the probability values of these effective modulation recognition results are sorted, and the modulation class corresponding to the group with the maximum probability value is taken as the final modulation method recognition result output.

[0084] As Figure 4 shown, the training process of the signal modulation method recognition neural network mainly includes parts such as signal data augmentation, signal feature calculation, model inference calculation, loss function calculation, and parameter update. First, data augmentation is performed on the original baseband complex signal data (also called signal IQ data) in the training dataset. The augmentation methods include adding random frequency offset, adding random Gaussian noise, and adding random sampling rate transformation to the signal data. For the same signal sample data, the three methods can be combined and superimposed for sample augmentation. That is, a sample can select one or more of the three methods for augmentation to form an augmented sample. The augmentation method of adding random frequency offset is:

[0085] ,n = 0,1,2,…,N - 1 (7)

[0086] where is the random frequency offset value added on the basis of the original signal, is the sampling rate, is the augmented sample, and N is the number of signal sampling points in the sample. Among them, the frequency offset value added to the signal is controlled by a random parameter. The control method is:

[0087] (8)

[0088] where is a random variable subject to a uniform distribution, is the signal bandwidth.

[0089] The way to add Gaussian noise is as follows:

[0090] (9)

[0091] where is complex Gaussian noise with zero mean and variance of . It is controlled by setting the signal-to-noise ratio, and its relationship with the signal-to-noise ratio is:

[0092] (10)

[0093] where is the signal power, is the signal bandwidth.

[0094] The way to increase resampling is:

[0095] (11)

[0096] where is the resampling function, and its resampling operation is controlled by two parameters, the original sampling rate and the resampled sampling rate, is the original sampling rate, is the resampled sampling rate. The control method of

[0097] (12)

[0098] where is a random variable obeying the uniform distribution.

[0099] The enhanced data format is the same as the original data format, that is, it is also a complex signal sequence, including two channels of real part and imaginary part. After data enhancement, 5 kinds of time-domain characterization calculations and 5 kinds of frequency-domain characterization calculations are performed on the enhanced signal samples. 10 characterization vectors are obtained, and these characterization vectors are sent into the feature extraction neural network model. The neural network model performs inference calculations to form feature vectors; the feature vectors are sent into the open-set classifier, and each group classifier outputs the prediction probability values corresponding to each category of each group and an energy value. The prediction probability values and energy values of each group are used as the output of the classifier. Combining with the true class label of this sample, the error between the prediction result and the true label is calculated using the loss function to form a loss value. Then, using this loss value through the feedback propagation mechanism, the learnable parameters of the neural network in the feature extraction neural network and the classifier are updated. There are two types of loss functions in the present invention, cross-entropy loss and energy loss. For a single group the cross-entropy loss , its calculation method is:

[0100] (13)

[0101] where represents the probability value that the input sample is predicted as class c in the group , and represents that the true class of the sample in the group is c.

[0102] For each group, the cross-entropy is calculated independently, and then the losses of all groups are summed as the cross-entropy loss of the sample. Then, the cross-entropy losses of all samples are averaged as the cross-entropy loss during training. Therefore, the total cross-entropy loss is denoted as:

[0103] (14)

[0104] For the energy loss of a single group , its calculation method is:

[0105] (15)

[0106] where represents the energy output by the classifier for the input sample, is a hyperparameter during training. Its meaning is that if a sample does not belong to the group , then in the classifier corresponding to this group, its energy value should be less than . For each group, the energy loss value is calculated independently, and then the losses of all groups are summed as the energy loss of the sample. Then, the energy losses of all samples are averaged as the energy loss during training. Therefore, the total energy loss is denoted as:

[0107] (16)

[0108] The total loss during training is composed of two losses weighted, and the calculation method is:

[0109] (17)

[0110] where is a hyperparameter that controls the weighted ratio of the two losses.

[0111] In an alternative embodiment, the number of elements in the input representation vector of the neural network is 8192, the total number of representation vectors is 10 time-frequency representation data, with a size of 10×8192. After being processed by the time-frequency domain branch of the feature extraction network, the input data respectively form feature tensors with a size of 512×256. The two feature tensors are respectively tokenized, transformed into token sequences with a single token dimension of 512 and a sequence length of 256, and processed using two parallel Transformer encoders. Class tokens with a dimension of 512 are respectively obtained, and the two class tokens are concatenated to form a feature vector with a size of 1024. A classifier is used to output the prediction probabilities and energy values of each group.

Claims

1. An open-set modulation recognition method based on time-frequency domain feature learning and fusion, characterized in that It includes the following steps: S1. The broadband electromagnetic spectrum monitoring device receives and captures the spatial electromagnetic radiation signal through the receiving antenna and converts it into an electrical signal. The electrical signal is converted into an intermediate frequency signal through the analog receiver, and then the intermediate frequency signal is converted into a baseband complex signal through the digital receiver; S2. Convert the baseband complex signal into time-domain and frequency-domain representations, and output each representation in the form of a vector. Denote the time-domain representation vector as , where n is the serial number of the time-domain representation result, N is the number of points in the complex signal sequence for the current calculation, and i represents the serial number of the time-domain representation. The calculation formula for the time-domain representation is: ， Among them, represents the input baseband complex signal sequence, represents the real part calculation, represents the imaginary part calculation, represents the amplitude calculation, represents the phase calculation, represents the first-order difference calculation; Denote the frequency-domain characterization vector as , where k is the serial number of the frequency-domain characterization result, and the calculation formula for frequency-domain characterization is: ， Among them, represents the fast Fourier transform, represents the element-wise complex multiplication calculation of complex sequences, and represents the element-wise complex 8th and 16th power calculations of complex sequences; S3. Build a modulation feature extraction neural network, which includes two branches with exactly the same structure. The processing of the input data for each branch is as follows: Use a one-dimensional ResNet convolutional neural network to process the input multi-channel characterization vector. Through multi-layer convolution and pooling processing, a tensor is formed. Then, the dimension of the tensor is converted to obtain a vector sequence with a length of 256 and a single vector dimension of 512, which is defined as an embedded characterization Token sequence. Add a class Token with an initialization value of 0 and a dimension of 512 at the first position in the Token sequence, and then add a position encoding Token. The sequence number and dimension of the position encoding Token are the same as those of the input Token, obtaining a marked sequence with added position encoding. The marked sequence is sent to the Transformer encoder for processing. The Transformer encoder is formed by stacking multiple attention mechanism layers. The processing of the Transformer encoder does not change the length and dimension of the Token sequence. After being processed by the Transformer encoder, a processed Token sequence is obtained. In the processed Token sequence, the first Token is the class token processed by the encoder; The time-domain characterization vector and the frequency-domain characterization vector are respectively processed by two branches in the neural network to obtain two processed Token sequences. The first Token of each of the two processed Token sequences is taken as the time-domain class token and the frequency-domain class token respectively, and the time-domain class token and the frequency-domain class token are concatenated to obtain a modulation feature vector; S4. Perform signal classification and unknown class recognition based on the obtained modulation feature vector. Specifically: Construct different group classifiers with fundamentally different modulation methods , where k represents the k-th group classifier. Each group classifier uses a fully connected neural network to process the input feature vector to form a Logtis value. In each group classifier, according to the Logtis value, the Softmax layer is used for probability normalization and the energy calculation function is used to calculate the energy of the Logtis value. Among them, the category probability of the in-group modulation method is predicted by probability normalization, and whether the input sample is the modulation method within the group is judged by the energy of the Logtis value; the calculation formula of the Logtis value is: ， Among them is the input modulation feature vector, , are the weight parameters and bias parameters in the fully connected network, that is, the calculated Logtis value, and c is the number of modulation methods within the corresponding group; The process of probability normalization by the Softmax layer is: ， where and represent the Logits value and the normalized probability value of class c in the group , represents the summation over all classes belonging to the class set , represents the Logits value of class i in the summation process; For group k, the energy value is calculated as follows: ， where T is the temperature parameter; After each group classifier calculates its corresponding class probability value and energy value, final class judgment and detection of unknown classes are performed through post-processing; The method of post-processing is that for the modulation feature vector v, first judge whether the sample belongs to the modulation of this group class based on the energy value. The rule is: ， Among them is the decision threshold; If a sample is determined not to belong to the corresponding group after energy value judgment in all group classifiers, then the sample is determined to be of an unknown category; for each group classifier, if it is determined according to the energy value that the sample belongs to this group, then the maximum predicted probability value in the group is output , and its corresponding category serial number , as the effective modulation mode recognition result of this group; if multiple groups all output effective modulation mode recognition results, then sort the probability values of these effective modulation recognition results, and take the modulation category corresponding to the group with the maximum probability value as the final modulation mode recognition result for output; S5. After obtaining the baseband complex signals of known categories by using the method of S1, sample signals are obtained through preprocessing. After processing the sample signals by using the method of S2, they are input into the modulation feature extraction neural network of S3 to obtain the modulation feature vectors for training. The group classifier of S4 is trained by using the modulation feature vectors for training. The loss functions adopted during training include cross-entropy loss and energy loss. For a single group the cross-entropy loss : ， Among them represents the probability value that the input sample is predicted as class c in the group ; represents that the true class of the sample in the group is c; the cross-entropy of each group is calculated independently and then the losses of all groups are summed as the cross-entropy loss of the sample, and then the cross-entropy losses of all samples are averaged as the cross-entropy loss during training. Therefore, the total cross-entropy loss is denoted as:​ ， For a single group the energy loss is calculated as follows: ， Among them represents the energy output by the classifier for the input sample is a hyperparameter during training, meaning that if a sample does not belong to the group , then in the classifier corresponding to that group, its energy value should be less than ; the energy loss value of each group is calculated independently and then the losses of all groups are summed as the energy loss of the sample, and then the average of the energy losses of all samples is used as the energy loss during training. Therefore, the total energy loss is denoted as: ， The total loss during training is composed of the weighted sum of two losses, and the calculation method is: ， wherein is a hyperparameter for controlling the weighted proportion of the two losses; After training, a trained group classifier is obtained; S6. The obtained electromagnetic radiation signal is processed in the manner of S1 - S3 to obtain a modulation feature vector, and then the modulation feature vector is input into the trained group classifier for recognition.

2. The open-set modulation recognition method based on time-frequency domain feature learning and fusion according to claim 1, wherein, The preprocessing described in S5 is data augmentation processing, including adding random frequency offset, adding random Gaussian noise, and adding random sampling rate transformation to the signal data; The enhancement method of adding random frequency offset is: , where n = 0, 1, 2, …, N - 1 where is the random frequency offset value added on the basis of the original signal, is the sampling rate, is the enhanced sample, and N is the number of signal sampling points in the sample; The adding method of Gaussian noise is: ， wherein is a complex Gaussian noise with zero mean and variance ; The method of adding resampling is: ， Among them is a resampling function, and its resampling operation is controlled by two parameters: the original sampling rate and the resampling rate is the original sampling rate is the resampling rate

Images