Deep learning based multi-modal fusion wireless signal automatic modulation recognition method

Abstract

This invention discloses an automatic modulation identification method for multimodal fusion wireless signals based on deep learning, belonging to the field of communication technology. First, a publicly available dataset is selected as the channel input data and merged according to modulation type to form a set of time-domain I / Q data. Then, a constellation diagram is created for each set of time-domain I / Q data, and the data and corresponding constellation diagrams are segmented according to the signal-to-noise ratio to form a training set. Next, a multimodal fusion neural network model based on the time-domain I / Q data and constellation diagram data is constructed and input into the training set for iteration, adjusting the learning rate until the network model reaches stability. Finally, a newly acquired radio signal to be identified is input into the trained multimodal fusion neural network model, which automatically outputs the modulation identification type of the wireless signal. This invention can more fully extract the fusion features of the signal in the time domain and the corresponding constellation diagram to optimize the information loss caused by downsampling, thereby obtaining more accurate signal phase and amplitude information.

Description

Technical Field

[0001] This invention belongs to the field of communication technology, specifically a method for automatic modulation and identification of multimodal fusion wireless signals based on deep learning. Background Technology

[0002] The explosive growth in the number of frequency-using devices has made the already scarce spectrum resources even more strained. As an important step between signal detection and signal demodulation, automatic modulation identification has become an important means to solve the increasingly crowded and complex electromagnetic space, and an important prerequisite for achieving efficient spectrum sensing, spectrum understanding and spectrum utilization in non-cooperative communication scenarios.

[0003] As wireless channels, signal waveforms, and modulation formats become increasingly dynamic and complex, accurate signal parameter estimation and modulation classification become increasingly challenging.

[0004] Cognitive radio can improve spectrum utilization and enable spectrum sharing after modulation identification is completed. However, as channel conditions become more complex and modulation types increase, new challenges arise for modulation identification. The key aspects of signal modulation identification align perfectly with the advantages of deep learning.

[0005] Deep learning-based automatic modulation recognition methods effectively reduce computational complexity, improve model generalization ability, and enhance robustness. To a certain extent, they outperform traditional methods and are more suitable for practical applications, representing a major trend in current automatic modulation recognition technology. Therefore, finding a deep learning-based automatic modulation recognition method for wireless signals under mixed noise conditions is essential. With the diversification of signal patterns, different representations of the same signal also affect signal recognition and classification. By fusing data from different modalities, more advantageous feature maps can be discovered, leading to more accurate classification and a more superior recognition model. Summary of the Invention

[0006] This invention proposes an automatic modulation and identification method for multimodal fusion wireless signals based on deep learning. By designing a multimodal feature fusion neural network model with an attention mechanism module, it can more fully extract the fusion features of the signal in the time domain and the corresponding constellation diagram, thereby optimizing the information loss caused by downsampling and obtaining more accurate signal phase and amplitude information.

[0007] The automatic modulation and identification method for wireless signals includes the following steps:

[0008] Step 1: Select the public dataset RML2016.10B as the channel input data, and according to the modulation type, merge 8 adjacent data of the same modulation type into a set of time domain I / Q data.

[0009] Step 2: Create a constellation diagram for each group of time-domain I / Q data, and cut the data according to the signal-to-noise ratio. The constellation diagrams are also cut one by one to form the training set and the test set.

[0010] Both the training and test sets include: time-domain I / Q data of the signal and one-to-one corresponding constellation diagram data;

[0011] Step 3: Construct a multimodal fusion neural network model based on time-domain I / Q data and constellation diagram data, and input it into the training set for iteration, adjusting the learning rate until the network model reaches stability;

[0012] The multimodal fusion neural network model includes: a time-domain I / Q signal processing layer, a constellation diagram processing layer, a model fusion layer, and a fully connected layer.

[0013] The temporal I / Q signal processing layer extracts I / Q type signal features through three sets of convolutional blocks with different kernels; it consists of 14 convolutional layers and 3 pooling layers.

[0014] The constellation graph processing layer extracts features from the input constellation graph signal using residual convolution stacks. It consists of 16 groups of four types of residual stacks. Each residual stack consists of two 3*3 two-dimensional convolutions. The number of convolution kernels used in each type of residual stack is also different. In addition, the number of residual stacks contained in each type is also different.

[0015] The model fusion layer combines the features extracted from the temporal I / Q signal processing layer and the features extracted from the constellation graph processing layer using the concat method. It then performs channel and spatial attention through the channel attention module and spatial attention module in CBAM, respectively. Finally, the feature maps are further extracted by the convolutional layer and fed into the subsequent fully connected layer.

[0016] The fully connected layer maps the outputs of multiple neurons to probabilities in [0,1] using the SoftMax activation function to achieve multi-label classification; the loss function of this convolutional neural network model is estimated by combining cross-entropy error and the softmax function.

[0017] Step 4: For the newly acquired radio signal to be identified, input it into the trained multimodal fusion neural network model, and automatically output the modulation identification type of the radio signal.

[0018] The advantages of this invention are:

[0019] 1. The multimodal fusion wireless signal automatic modulation identification method based on deep learning has certain advantages compared with the current mainstream methods: the generalization ability of this method is relatively strong, and the modulation classification accuracy of the multimodal fusion neural network based on the attention mechanism in this dataset is better than that of other adaptive modulation coding neural network models and other single data modal models in terms of high-resolution classification among high-order modulation methods.

[0020] 2. A deep learning-based multimodal fusion method for automatic modulation and recognition of wireless signals. This method uses a multimodal fusion approach, where the model input consists of different representations of the same data, namely time-domain I / Q data and constellation diagram data, i.e., numerical data and constellation diagram data. These data have significantly different classification effects under certain modulation methods. By combining the advantages of both through multimodal fusion, the recognition accuracy is improved.

[0021] 3. A deep learning-based method for automatic modulation and recognition of multimodal fusion wireless signals is proposed. Based on the characteristics of different modal data, lightweight convolutional network combination blocks and residual convolutional network combination blocks are used respectively. Each of them uses a small convolutional kernel, which can quickly extract features while reducing the complexity of the model and still maintaining good recognition performance.

[0022] 3. A deep learning-based multimodal fusion automatic modulation recognition method for wireless signals introduces and improves the attention mechanism in automatic modulation recognition. The feature map is multiplied and weighted in each channel and in the convolution operation to recalibrate the original features in terms of channel dimension and space. The attention mechanism weights the acquired importance information, and the resulting feature map is more advantageous than the original image. Attached Figure Description

[0023] Figure 1 This is a flowchart of a deep learning-based method for automatic modulation and identification of multimodal fusion wireless signals.

[0024] Figure 2 The structure diagram of the multimodal fusion neural network model based on time-domain I / Q data and constellation diagram data constructed in this invention is shown below.

[0025] Figure 3 This is a diagram of the residual stack structure used in this invention;

[0026] Figure 4 This is a structural diagram of the CBAM attention mechanism used in this invention; Detailed Implementation

[0027] The specific embodiments of the present invention will be described in further detail below with reference to the accompanying drawings.

[0028] In existing technologies, the proposal of automatic modulation recognition based on multimodal fusion neural networks using attention mechanisms stems from the fact that conventional deep learning methods only analyze and translate one modality of signal data to achieve classification. However, the multimodal fusion wireless signal automatic modulation recognition method proposed in this invention uses temporal I / Q data and constellation diagram data as inputs to the model, extracting features from both separately. The outputs of the two branch analytical networks are then fused through feature fusion. An attention mechanism is introduced to extract the channel and spatial importance weights of the feature maps during convolution, and the original features are weighted for calibration. Simultaneously, a cross-entropy loss function combined with a SoftMax classifier is used to better measure the difference between the estimated and true values, reconstructing more accurate signal values ​​and improving the accuracy of model training. Experimental results demonstrate the feasibility, effectiveness, and superiority of this method.

[0029] The wireless signal automatic modulation and identification method, such as Figure 1 As shown, it includes the following steps:

[0030] Step 1: Select the public dataset RML2016.10B as the channel input data, and according to the modulation type, merge the adjacent 8 groups of data with the same modulation identification as a group of time domain I / Q data.

[0031] For high-order modulation signals, the data size of RML2016.10B is appropriate, which can prevent the model from being too long during training after the time-domain I / Q data is converted into a constellation diagram; and it can simulate wireless communication environments through complex channel conditions and environments.

[0032] The length of each set of time-domain I / Q data is 2*1024;

[0033] Step 2: Create a constellation diagram for each group of time-domain I / Q data, and cut the data according to the signal-to-noise ratio. The constellation diagrams are also cut one by one to form the training set and the test set.

[0034] First, the coefficients of the real and imaginary parts of each set of I / Q data are used as the x-axis and y-axis of the image to create a constellation diagram; each set of data corresponds to one constellation diagram.

[0035] Then, the data is segmented according to the different signal-to-noise ratios of various modulation methods, and the I / Q data is segmented according to the one-to-one correspondence of the constellation diagram.

[0036] Each group of time-domain I / Q data is used to construct a constellation diagram according to the proportion and size of the space in the coordinate system. Each constellation diagram contains 1024 points, so that there is a one-to-one correspondence between the time-domain I / Q data and the constellation diagram data. The overall data is divided into training set and test set in an 8:2 ratio. Both training set and test set include: time-domain I / Q data of the signal and corresponding constellation diagram data.

[0037] Step 3: Construct a multimodal fusion neural network model based on time-domain I / Q data and constellation diagram data, and input it into the training set for iteration, adjusting the learning rate until the network model reaches stability;

[0038] By defining the model's loss function, I / Q data and constellation diagram data feature extraction layers, feature fusion and attention extraction layers, etc., a multimodal fusion neural network model is constructed. The data in the training set is input into the network model, and through continuous training and parameter optimization, a high-quality signal recognition method is obtained.

[0039] like Figure 2 As shown, the multimodal fusion neural network model includes: a time-domain I / Q signal processing layer, a constellation diagram processing layer, a model fusion layer, and a fully connected layer.

[0040] The temporal I / Q signal processing layer extracts I / Q type signal features through three sets of convolutional blocks with different kernels; a CNN architecture is used to extract I / Q features, consisting of 14 convolutional layers and 3 pooling layers;

[0041] The connection relationships are as follows: the first convolutional layer processes the temporal I / Q data from the input layer, with an input dimension of 2×1024, a kernel size of 3*3, and padding size of (3,1); the first convolutional layer is followed by the first pooling layer; then the same second convolutional layer and second pooling layer are connected in sequence; to optimize the extraction of signal features, the first pooling layer reduces the size of the feature map, the second convolution is the same as the first convolution, and the second pooling layer is the same as the first pooling layer.

[0042] Subsequently, the output feature maps are passed sequentially through the 1st, 2nd, and 3rd combination blocks. Each combination block consists of 4 convolutional layers and 4 Batchnorm2d operations. The first and last convolutional layers of the 1st, 2nd, and 3rd convolutional blocks use 1×1 kernel matrices to reduce the dimensionality of the feature map channels, which impacts computational complexity. The 2nd and 3rd convolutional layers use asymmetric kernel matrices of 3×1 and 1×3 sizes, respectively, instead of 3×3, to reduce the number of trainable parameters. The final pooling layer uses a 3*16 convolutional kernel to match the subsequent feature fusion operation.

[0043] Each block consists of convolutional blocks with varying numbers of kernels. Each convolutional block consists of a convolutional layer (conv2d), a batch normalization layer (batchnorm2d), and an activation function (ReLU).

[0044] The activation function, the ReLU function (rectified linear unit), is given by:

[0045] z[i]=y=ReLU(z[i])=max(0,z[i])={0,z[i]<0;z[i],z[i]0}

[0046] Where z[i] represents the feature mapping value, y represents the value after activation by the function, and then put into z[i].

[0047] The constellation graph processing layer extracts features from the input constellation graph signal using residual convolution stacks. A ResNet architecture is used to extract constellation graph features, consisting of 16 sets of four types of residual stacks. Each residual stack consists of two 3x3 two-dimensional convolutions with a stride of 1 and padding of 1. The stack structure is as follows: Figure 3 As shown, the number of convolutional kernels used for each type of residual stack is different, at 16, 32, 32, and 16 respectively. Furthermore, the number of residual stacks included in each type is also different, at 3, 4, 6, and 3 respectively. By using different types and numbers of residual stacks, effective feature maps can be extracted by fully utilizing convolutional operations.

[0048] The model fusion layer fuses the time-domain I / Q data processed by the time-domain I / Q signal processing layer and the constellation map data processed by the constellation map signal processing layer. The previous operation has reduced the data of each type to a suitable feature map size. This layer uses the concat method to perform feature fusion, superimposing the feature map information of the corresponding dimensions to obtain the advantageous information.

[0049] Because the representational information of constellation diagrams and time-domain I / Q data is completely inconsistent, the semantics of the extracted feature information at corresponding channel positions are not similar. The goal is to achieve fusion while significantly increasing content. The formula for the model fusion stage can be expressed as:

[0050] F = F1·F2

[0051] Where F represents the result after feature fusion, F1 is the feature of the time-domain I / Q data processing layer, and F2 is the feature of the constellation diagram.

[0052] Due to the complexity of the feature maps after fusion, the CBAM (Convolutional Block Attention Module) attention mechanism is introduced to improve model performance. The model structure of this mechanism is as follows: Figure 4As shown, CBAM comprises two independent sub-modules: the Channel Attention Module (CAM) and the Spartial Attention Module (SAM). CAM reassigns weights to channels and spatial areas respectively, highlighting important information features. It can assign importance to the model in both channels and space while saving computational resources. Furthermore, during convolution operations, it recalibrates the weights in each channel and different spatial areas, accurately capturing inter-channel dependencies and thus improving the model's ability to handle complex problems. Finally, the feature map after the attention mechanism is passed through a 3x3 convolutional layer to further extract appropriately sized feature maps before being fed into subsequent fully connected layers.

[0053] The formula for CBAM can be summarized as follows:

[0054]

[0055] In the formula, σ is the sigmoid function, W0∈RCr×C, W1∈RC×Cr, the weights W0 and W1 of the MLP are shared, and the ReLU activation function is before W0;

[0056] The fully connected layer uses the SoftMax activation function to map the outputs of multiple neurons to probabilities in [0,1] to achieve multi-label classification; it includes a fully connected (FC) layer with 128 units, a ReLU activation layer, a dropout layer, and the last layer is also a fully connected layer.

[0057] After feature extraction and fusion in the first few layers, the data enters a fully connected (FC) layer with 128 units, a ReLU activation layer, followed by a dropout layer that randomly discards 70% of the neurons, and the last layer is also a fully connected layer with 10 neurons, the same as the number of modulations considered. Softmax activation is used to alleviate overfitting and force the nodes to be more independent than usual. A dropout regularization strategy is used to prevent the complete update of the weights of some nodes.

[0058] The loss function of this convolutional neural network model is estimated by combining cross-entropy error and the softmax function. The specific formula is shown below:

[0059]

[0060] Where m is the number of labels, t[i] is the one-hot encoding of the actual label, and p[i] is the probability of the label, as shown in the formula:

[0061]

[0062] Here, z′[i] represents the feature map after ReLU activation.

[0063] Based on the loss value, the adaptive moment estimation (Adam) optimization algorithm and backpropagation mechanism are used to update the values ​​of trainable parameters in the network, realizing the learning and optimization process of the neural network. The initial learning rate is 0.003 for model optimization, and a total of 50 iterations are performed. The learning rate decreases by 0.1 after 40 iterations. Adam requires fast convergence, can train complex networks with small memory requirements, and is suitable for large datasets and high-dimensional spaces. This model is very suitable for a large number of datasets.

[0064] The corresponding test set and training set data are input into the constructed multimodal fusion neural network model with attention mechanism, and the model is trained until it is stable.

[0065] Step 4: For the newly acquired radio signal to be identified, input it into the trained multimodal fusion neural network model, and automatically output the modulation identification type of the radio signal.

[0066] Example:

[0067] The wireless signal automatic modulation and identification method based on deep learning multimodal fusion includes: a data preprocessing module, a training module, and an output module;

[0068] The data preprocessing module uses 10 modulation types of data from the publicly available dataset RML2016.10B as training data for the model. After cutting and partitioning, the time-domain I / Q data is made into constellation diagram data and matched one by one. Then, the time-domain data and constellation diagram data are input into the model together.

[0069] Specifically, the dataset contains 10 standard modulation methods under 20 signal-to-noise ratio (SNR) conditions (from -20dB to 18dB, in 2dB increments). The modulation methods are as follows:

[0070] 1) Digital modulation: PSK, QPSK, 8PSK, 16QAM, 64QAM, GFSK, CPFSK, 4PAM;

[0071] 2) Analog modulation: WBFM, AM-DSB.

[0072] Each modulation method contains 128 dual-channel I / Q data points at each signal-to-noise ratio value, and each data point contains 1000 samples.

[0073] In preprocessing, the original 100,000 I / Q data points of length 2*128 are converted to data of length 2*1024, resulting in 12,500 I / Q data points. The coefficients of the real and imaginary parts of each I / Q data set are used as the x and y axes to create a constellation diagram. Each set of 1024 points is used to create a constellation diagram. Based on each I / Q data set and the constellation... Figure 1 A one-to-one correspondence was established, resulting in the creation of 12,500 constellation chart data images.

[0074] Then, the signal-to-noise ratio (SNR) data for the 10 modulation schemes were segmented into 20 files based on their respective SNR values ​​for later use. The generated constellation diagram data was also divided into 20 files according to the one-to-one correspondence between the I / Q data.

[0075] The training module trains a multimodal fusion neural network model based on time-domain I / Q data and constellation diagram data, iterates multiple times and adjusts the learning rate until the network model reaches stability, and then saves the model.

[0076] Specifically, it includes:

[0077] The time-domain I / Q data and constellation diagram data of the signal are input into the multimodal fusion neural network model in a one-to-one correspondence. After different networks process the multimodal data and fuse features, and the attention mechanism extracts importance, the modulation signal classification features are obtained and finally output to the fully connected layer.

[0078] The output module loads the multimodal fusion neural network model with attention mechanism saved by the training module, verifies the data in each channel of the RML2016.10B dataset, loads the saved model parameters, and evaluates the recognition accuracy of the model.

[0079] Specifically, it includes the following:

[0080] Extract the trained multimodal fusion neural network model with attention mechanism and parameters, load the model parameters saved in the training module into its network model, verify the loss value and accuracy under each channel from -20dB to 18dB, and judge the performance of the model; then identify the modulation mode under each channel ratio, and output the loss value and accuracy.

[0081] The data processing module, the training module, and the output module are linked sequentially.

[0082] The embodiments described in this specification are merely examples of implementations of the inventive concept. The scope of protection of this invention should not be considered as limited to the specific forms stated in the embodiments. The scope of protection of this invention also extends to equivalent technical means that can be conceived by those skilled in the art based on the inventive concept.

Claims

1. A deep learning-based multi-modal fusion wireless signal automatic modulation recognition method, characterized in that, The specific steps are as follows: For the public dataset, the time-domain I / Q data are divided into groups according to the modulation type, and a corresponding constellation diagram is made for each group of time-domain I / Q data. Then, a multimodal fusion neural network model is constructed, and time-domain I / Q data and constellation diagram data are input for feature extraction. The outputs of the two branch parsing networks are fused through feature fusion. An attention mechanism is introduced to extract the channel and spatial importance weights of the feature maps during convolution, and the original features are weighted to complete calibration. At the same time, the cross-entropy loss function is used in combination with the SoftMax function classifier to better measure the difference between the estimated value and the true value, reconstruct more accurate signal values, and improve the accuracy of model training. Finally, for the newly acquired radio signal to be identified, the trained multimodal fusion neural network model is input, and the modulation identification type of the radio signal is automatically output. The multimodal fusion neural network model includes: a time-domain I / Q signal processing layer, a constellation diagram processing layer, a model fusion layer, and a fully connected layer; The temporal I / Q signal processing layer extracts I / Q type signal features through three sets of convolutional blocks with different kernels; it consists of 14 convolutional layers and 3 pooling layers. The connection relationships of the time-domain I / Q signal processing layer are as follows: The first convolutional layer processes the temporal I / Q data from the input layer, followed by the first pooling layer; then the same second convolutional layer and second pooling layer are connected in sequence. Subsequently, the output feature map is passed through the 1st, 2nd, and 3rd combination blocks in sequence. Each combination block consists of 4 convolutional layers and 4 Batchnorm2d layers. The 2nd and 3rd convolutional layers use asymmetric kernel matrices with kernel sizes of 3×1 and 1×3, respectively. The last pooling layer uses a 3*16 convolutional kernel to match the subsequent feature fusion operation. The constellation map processing layer extracts features from the input constellation map signal using residual convolution stacks. There are 16 groups of four types of residual stacks. Each residual stack consists of two 2D convolutions with 3*3 kernels. The number of convolutional kernels used in each type of residual stack is different. At the same time, the number of residual stacks contained in each type is also different. The model fusion layer combines the features extracted from the temporal I / Q signal processing layer and the features extracted from the constellation diagram processing layer using the concat method. It then extracts attention in the channel and spatial dimensions using the channel attention module and spatial attention module in CBAM, respectively. Finally, the feature map is further extracted by the convolutional layer and fed into the subsequent fully connected layer. Fully connected layers use the SoftMax activation function to map the outputs of multiple neurons to probabilities in [0, 1] to achieve multi-label classification; The loss function of the multimodal fusion neural network model is estimated by combining cross-entropy error and the SoftMax function.

2. The deep learning-based multi-modal fusion wireless signal automatic modulation recognition method of claim 1, wherein, The publicly available dataset is RML2016.10B, which merges eight adjacent sets of data of the same modulation type into a single set of time-domain I / Q data.

3. The deep learning-based multi-modal fusion wireless signal automatic modulation recognition method of claim 1, wherein, The process involves creating a constellation diagram for each group of time-domain I / Q data, segmenting the data according to the signal-to-noise ratio, and correspondingly segmenting the constellation diagrams to form a training set.

4. The method for automatic modulation and identification of multimodal fusion wireless signals based on deep learning as described in claim 1, characterized in that, The formula of the CBAM is where, is a sigmoid function, , , the weights of and are shared, and is a ReLU activation function.

5. The method for automatic modulation and identification of multimodal wireless signals based on deep learning as described in claim 1, characterized in that, The loss function formula is as follows: ,in The number of tags. It is a valid one-bit code of the actual label. It is the probability of the label, calculated using the following formula: ,in, This represents the feature map after ReLU activation.

Images