Knowledge distillation and communication modulation identification method based on hybrid expert model

Abstract

The invention provides a knowledge distillation and communication modulation identification method based on a hybrid expert model, belongs to the field of communication, solves the problems of increasing communication modulation identification models and signal interference, and comprises the following steps: training an initial expert model according to a signal-to-noise ratio cutting training set; obtaining a trained hybrid expert model capable of communication modulation identification; training the initial selection model through the training set and the hybrid expert model; if the hybrid expert model is not weighted, identifying an optimal expert model matched with the training set by using T-softmax to obtain a trained selection model, and integrating to obtain a teacher model; if yes, outputting an output result of the teacher model through the aggregation model; and training the initial student model through the output results of the optimal expert model, the training set and the teacher model to obtain a trained student model capable of communication modulation identification. According to the invention, through the trained student model, light weight of the model is realized, and the recognition rate of a low signal-to-noise ratio is improved.

Description

Technical Field

[0001] The present invention belongs to the field of communication technology, and in particular relates to a knowledge distillation and communication modulation recognition method based on a hybrid expert model. Background Art

[0002] In a communication system, in order to achieve efficient transmission over long distances, the transmitter needs to modulate the signal, which can convert the original information into an analog or digital signal suitable for propagation in the transmission medium. The receiver needs to use demodulation technology to restore the original information.

[0003] In cooperative communication, the transmitter can inform the receiver of the required modulation style in order to demodulate the data. However, in non-cooperative communication, the receiver cannot determine the modulation mode of the signal, so it must determine its modulation mode by analyzing the characteristics of the signal in order to perform subsequent signal demodulation operations and applications, and perform spectrum monitoring at the same time to ensure the orderly use of the wireless spectrum.

[0004] like Figure 1 This is a schematic diagram of the existing communication modulation recognition process system. The signal is collected by the antenna to the computer at the receiving end. After signal detection and preprocessing, the preprocessed signal is obtained. It is input into the modulation recognition model, and the signal features are extracted and recognized, so as to achieve the purpose of identifying the signal modulation method.

[0005] Since O'shea published Convolutional radio modulation recognition networks in 2016, the networks for communication modulation recognition have become increasingly diversified and their accuracy has been continuously improved. Various deep learning structures have gradually been introduced to improve the accuracy of signal recognition.

[0006] However, with the continuous emergence of the above-mentioned models, their technical recognition rates continue to rise, but they have the following defects: First, the communication modulation recognition model under normal circumstances is relatively large, which requires a lot of computing power and cannot meet the increasing demand for lightweight deployment; Second, the recognition rate of the lightweight communication modulation recognition commonly used in the existing technology is still low under low signal-to-noise ratio conditions. Summary of the invention

[0007] The present invention provides a knowledge distillation and communication modulation recognition method based on a hybrid expert model. The technical solution disclosed in the present invention is lightweight. Compared with a common convolutional neural network model, the student model finally trained can reach less than 30% of its parameters, solving the problem of the increasing size of the communication modulation recognition model. At the same time, the present invention sets a hybrid expert model for signal-to-noise ratio division of different communication signals, and integrates it with a selection model, solving the problem that the signal feature extraction of a high signal-to-noise ratio is interfered by a signal of a low signal-to-noise ratio.

[0008] To achieve the above object, the present invention proposes a knowledge distillation and communication modulation recognition method based on a hybrid expert model, comprising:

[0009] Step 1: Use a non-repetitive random sampling method to extract signals in the wireless communication field data set according to a preset ratio, and cut them into a training set, a validation set, and a test set;

[0010] Step 2: construct the first training module through the Adam optimizer and the cross entropy loss function, and construct the second training module through the Adam optimizer and the knowledge distillation loss function; construct the initial expert model, the initial selection model and the initial student model respectively by adding batch normalization and the fully connected layer in the convolution layer; the number of layers corresponding to the convolution layer and the fully connected layer in the initial expert model, the initial student model and the initial selection model gradually decreases; set the association points associated with the middle layer of the initial expert model between the layers of the initial student model through the hook function in pytorch;

[0011] Step 3: According to the signal-to-noise ratio in the training set, the training set is cut to obtain multiple training subsets; using the first training module, the initial expert model is trained through the multiple training subsets respectively to obtain multiple trained expert models capable of communication modulation recognition, and the multiple expert models constitute a hybrid expert model;

[0012] Step 4: Use the first training module to train the initial selection model through the training set and the hybrid expert model. If the selection model does not weight the hybrid expert model, use the T-softmax function to identify the optimal expert model that matches the training set to obtain a trained selection model; integrate the hybrid expert model with the trained selection model to obtain a teacher model; if the selection model weights the hybrid expert model, use the aggregation model to weight the selection model output weight and the expert model output weight, and output the recognition result with the highest probability, which is the output result of the teacher model;

[0013] Step 5: Use the second training module to train the initial student model through the association points of the optimal expert model and the training set to obtain the trained middle layer of the student model capable of communication modulation recognition; then use the second training module to train the output result of the initial student model through the output result of the teacher model and the training set to form a trained student model capable of communication modulation recognition with the trained middle layer;

[0014] Step six, iterate step five. After reaching the preset number of times, the internal data of the student model is locked, and the student model is verified using the verification set. If the knowledge distillation loss function in the second training module converges and the mean square error value is overfitted, the verification is stopped to obtain a qualified student model; the test set is input into the qualified student model, and the communication modulation recognition result is output.

[0015] In step 2, the method for constructing the first training module includes:

[0016] Create the Adam optimizer in Python using the torch.optim.adam function;

[0017] The cross entropy loss function value of the cross entropy loss function is input into the Adam optimizer, and the model to be trained is trained by making the cross entropy loss function value gradient decrease through the Adam optimizer, and the learning rate of each parameter in the model to be trained is dynamically adjusted to optimize the convergence of the cross entropy loss function, thereby obtaining the first training module constructed;

[0018] Among them, the cross entropy loss function is:

[0019] CE = -Σ-tilog(pi);

[0020] Among them, CE is the cross entropy loss function value; ti represents the correct answer, and pi represents the predicted value.

[0021] In step 2, the method for constructing the second training module includes:

[0022] Create the Adam optimizer in Python using the torch.optim.adam function;

[0023] The knowledge distillation loss function value of the knowledge distillation loss function is input into the Adam optimizer, and the knowledge distillation loss function value is gradient-decreased to train the to-be-trained model through the Adam optimizer, and the learning rate of each parameter in the to-be-trained model is dynamically adjusted to optimize the convergence of the knowledge distillation loss function, thereby obtaining the constructed second training module;

[0024] Among them, the knowledge distillation loss function is:

[0025] FKD_LOSS=a*MSE_Loss_result+b*MSE_Loss_hook1+c*MSE_Loss_hook2;

[0026] Among them, FKD_LOSS is the value of the knowledge distillation loss function, a represents the weight of the difference between the features of the student model's self-learning results, b represents the weight of the difference between the features of the convolutional layers, and c represents the weight of the difference between the features of the fully connected layers, which plays a role in reconciling the order of magnitude and weighting; MSE_Loss_result represents the MSE of the student model's self-learning; MSE_Loss_hook1 represents the MSE of the inter-layer features of the same dimension between the convolutional layers, and MSE_Loss_hook2 represents the MSE of the inter-layer features of the same dimension extracted between the fully connected layers. Among them, MSE is the mean square error value, and the mean square error function is expressed as follows:

[0027]

[0028] Among them, Za represents the true value, za represents the predicted value, and n represents the total number of executed predictions.

[0029] In step 2, the construction methods of the initial expert model, the initial selection model and the initial student model include:

[0030] Set the nn.nn.covd function in pytorch to determine the number of dimensions of the convolutional layer;

[0031] The output convolution is flattened and reduced in dimension through the torch.reshape function;

[0032] Then, the intermediate layer is constructed by using the nn.Linear function, nn.RELU function and nn.BatchNorm2d function. After the intermediate layer is fully connected, positively correlated and batch normalized, an initial expert model, an initial selection model and an initial student model are constructed respectively. The number of layer dimensions corresponding to the convolutional layer and the fully connected layer in the initial expert model, the initial student model and the initial selection model is gradually reduced. And the hook function in pytorch is used to set the association points associated with the intermediate layer of the initial expert model between the layers of the initial student model.

[0033] In step 3, the training set is cut according to the signal-to-noise ratio in the training set to obtain multiple training subsets, including:

[0034] The training sets corresponding to the lowest and highest signal-to-noise ratio segments in the training set are cut, and the training sets corresponding to the remaining middle segments are evenly cut to obtain multiple training subsets.

[0035] In step 3, using the first training module, training the initial expert model through multiple training subsets respectively to obtain multiple trained expert models capable of communication modulation recognition, the method for multiple expert models to form a mixture of experts model includes:

[0036] Inputting multiple training subsets into the initial expert model respectively, and outputting the predicted probability values of batch signals;

[0037] Taking the class label corresponding to the maximum predicted probability value of the signal as the predicted class label, inputting the true class label and the predicted class label of the signal into the cross-entropy loss function in the first training module, and calculating the cross-entropy loss function value;

[0038] Inputting the cross-entropy loss function value into the Adam optimizer, training the initial expert model by making the cross-entropy loss function value decline in gradient through the Adam optimizer, dynamically adjusting the learning rate of each parameter in the initial expert model, optimizing the convergence of the cross-entropy loss function, obtaining multiple trained expert models capable of communication modulation recognition corresponding to multiple training subsets, and multiple expert models form a mixture of experts model.

[0039] In step 4, using the first training module, the method for training the initial selection model through the training set and the mixture of experts model includes:

[0040] Extracting the features of each expert model in the mixture of experts model respectively through the convolutional layer of the initial selection model, and at the same time extracting the features of the input training set;

[0041] Using the first training module, calculating the cross-entropy loss function value between the features of each expert model in the mixture of experts model and the features of the training set, training the initial selection model by making the cross-entropy loss function value decline in gradient through the Adam optimizer, and dynamically adjusting the learning rate of each parameter in the initial selection model, optimizing the convergence of the cross-entropy loss function.

[0042] In step 4, the T-softmax function is:

[0043]

[0044] where pT-i is the recognition rate distribution of knowledge distillation results; i is a single signal target; T is the temperature parameter, and the value range is 0 < T < 1; pi is the recognition rate of the original teacher's single output signal target; pj is the recognition rate of all signal targets output by the original teacher; n is the total number of predictions;

[0045] The aggregation model is:

[0046]

[0047] Among them, A is the output result of the teacher model; k is the number of expert models in the teacher model; N is the total number of expert models in the teacher model; Bk is the output weight of the kth expert model; PBk is the output weight of the kth expert model corresponding to the selection model.

[0048] In step 5, the method of using the second training module to train the initial student model through the associated points of the optimal expert model and the training set to obtain the intermediate layer of the trained student model capable of communication modulation recognition includes:

[0049] The features of the training set are extracted through the convolutional layer of the initial student model. The initial student model extracts the output results of the intermediate layer of the optimal expert model in the teacher model through the association points. The second training module is used to calculate the knowledge distillation loss function value of the output results of the intermediate layer of the optimal expert model and the features of the training set. The initial student model is trained by gradient descent of the knowledge distillation loss function value through the Adam optimizer. The learning rate of each parameter in the initial student model is dynamically adjusted to optimize the convergence of the knowledge distillation loss function, and the trained intermediate layer of the student model capable of communication modulation recognition is obtained.

[0050] In step 5, the method of using the second training module to train the output result of the initial student model through the output result of the teacher model and the training set, and forming a trained student model capable of communication modulation recognition with the trained intermediate layer includes:

[0051] The features of the training set are extracted through the convolution layer of the initial student model, and the output results of the teacher model are extracted through the convolution layer of the initial student model. The second training module is used to calculate the knowledge distillation loss function value of the output results of the teacher model and the features of the training set. The initial student model is trained by gradient descent of the knowledge distillation loss function value through the Adam optimizer. The learning rate of each parameter in the initial student model is dynamically adjusted to optimize the convergence of the knowledge distillation loss function value. The trained student model capable of communication modulation recognition is composed of the intermediate layer of the trained student model capable of communication modulation recognition.

[0052] Beneficial effects of the present invention:

[0053] 1. The present invention discloses a knowledge distillation and communication modulation identification method based on a hybrid expert model, which adopts a hybrid expert model to identify the modulation mode of communication signals, further refines the data set in the field of wireless communication, and makes each expert model in the integrated learning recognize the signal features more fully, and the signal feature extraction of high signal-to-noise ratio will not be interfered by the signal of low signal-to-noise ratio.

[0054] 2. The present invention adopts feature-based knowledge distillation to perform lightweight communication signal modulation recognition. By learning the inter-layer relationship of the optimal expert model in the teacher model, the inter-layer information of the communication signal is further utilized. The core of this patent adopts a hybrid expert model integrated selection model as a lightweight teacher model for the communication modulation recognition network, and uses the selection model therein to evaluate the recognition effect of the hybrid expert model, thereby achieving the purpose of integrating the hybrid expert model, and then using the hybrid expert model to guide the training of a single student model in turn, which greatly simplifies the computational complexity of the model and solves the problem that the signal feature extraction of high signal-to-noise ratio will be interfered by the signal of low signal-to-noise ratio to a certain extent, so that the present invention achieves the lightweight of the model as a whole. Compared with the common convolutional neural network model, the student model finally trained can reach less than 30% of its parameters, solving the problem of the increasing size of the model.

[0055] 4. This patent optimizes the selection model in the expert model by introducing the T-softmax function, so that the serial operations sorted on the CPU are converted into parallel operations on the GPU, which increases its reasoning speed in one step, optimizes network performance, and reduces network training time. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 It is a system diagram of the communication modulation recognition process of the signal.

[0057] Figure 2 This is a schematic diagram of the response-based knowledge distillation learning model structure.

[0058] Figure 3 Schematic diagram of the feature-based knowledge distillation learning model structure.

[0059] Figure 4 It is a schematic diagram of the structure of a single expert model.

[0060] Figure 5 It is a schematic diagram of the structure of the hybrid expert model.

[0061] Figure 6 It is a schematic diagram of the selected model structure.

[0062] Figure 7 It is a flow chart of a knowledge distillation and communication modulation recognition method based on a hybrid expert model.

[0063] Figure 8 It is a schematic diagram of the modulation recognition accuracy of each network model under different signal-to-noise ratios. DETAILED DESCRIPTION

[0064] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.

[0065] The inventors found in the research process that the key technology for lightweight model construction in signal communication modulation recognition is knowledge distillation. The knowledge distillation algorithm is as follows: Figure 2 , Figure 3 As shown, it is divided into response-based and feature-based knowledge distillation. Both are achieved by constructing a "teacher model" with a large scale and high recognition rate and a "student model" with a small scale and low recognition rate, calculating the mean square error (MSE) of the two, and then optimizing the algorithm model through gradient descent iteration, so that the "student model" learns the "teacher model" to achieve the effect of model lightweighting. At the same time, in previous studies, in order to solve the problem of how to select the "educational qualifications" of the expert model in the case of multiple expert models, and to achieve the lightweighting of the modulation recognition model, the present invention chooses to introduce a selection model in the hybrid expert model to better integrate knowledge distillation of the network, thereby achieving the purpose of lightweighting. Below, the technical solution of the present invention is described in detail through specific embodiments.

[0066] The embodiment of the present invention provides a knowledge distillation and communication modulation recognition method based on a hybrid expert model, comprising:

[0067] Step 1: Use a non-repetitive random sampling method to extract signals in the wireless communication field data set according to a preset ratio, and cut them into a training set, a validation set, and a test set;

[0068] The preset ratio is preferably 7:2:1. Specifically, the input data array is set by using the cPickle.load function in the cPickle library in the computer programming language python, the np.zeros in the numpy library, and the torch.tensor function in pytorch, and the format of the key (modulation mode, signal-to-noise ratio) in the data set is converted to a larger data of (number of data items, (modulation mode, signal-to-noise ratio), data sampling points), and then the modulation 10 classifications are converted into a 10-bit binary Onehot vector form.

[0069] Step 2: Construct the first training module through the Adam optimizer and the cross entropy loss function, and construct the second training module through the Adam optimizer and the knowledge distillation loss function; construct the initial expert model, the initial selection model and the initial student model respectively by adding batch normalization and the fully connected layer in the convolution layer; the number of layer dimensions corresponding to the convolution layer and the fully connected layer in the initial expert model, the initial student model and the initial selection model gradually decreases; set the association points associated with the middle layer of the initial expert model between the layers of the initial student model through the hook function in pytorch; extract the output between the expert model layers through the hook function.

[0070] like Figure 4 As shown, a single expert model constructed by the present invention can be composed of 4 convolutional layers and 7 fully connected layers. At the same time, batch normalization is added to the convolutional layers to enhance the feature extraction of the convolutional layers and enhance their convergence. Each individual expert model is mainly responsible for learning a signal data set with a fixed signal-to-noise ratio. Figure 6 As shown, the selected model has, for example, 2 convolutional layers and 3 fully connected layers, and the dimensions of the five layers are all small.

[0071] Step 3: According to the signal-to-noise ratio in the training set, the training set is cut to obtain multiple training subsets; using the first training module, the initial expert model is trained through the multiple training subsets respectively to obtain multiple trained expert models capable of communication modulation recognition, and the multiple expert models constitute a hybrid expert model;

[0072] In the learning of data sets by traditional signal modulation recognition technology, the data sets are not divided. Since signal modulation recognition relies on the extraction and recognition of signal features, when the signal-to-noise ratio span is large, the learning of low signal-to-noise ratio may interfere with the learning under high signal-to-noise ratio conditions, resulting in a decrease in the signal recognition rate. In order to improve the signal recognition rate, this step divides the data set based on the signal-to-noise ratio.

[0073] Step 4: Use the first training module to train the initial selection model through the training set and the hybrid expert model. If the selection model does not weight the hybrid expert model, use the T-softmax function to identify the optimal expert model that matches the training set to obtain a trained selection model; integrate the hybrid expert model with the trained selection model to obtain a teacher model; if the selection model weights the hybrid expert model, use the aggregation model to weight the selection model output weight and the expert model output weight, and output the recognition result with the highest probability, which is the output result of the teacher model;

[0074] In order to enable a single expert model to better learn datasets under similar signal-to-noise ratio conditions, the present invention proposes to use a mixture of experts model, that is, in the case of parallel multi-expert models, a selection model is further introduced to integrate the multi-experts, so as to realize the identification of signal modulation modes.

[0075] However, the sorting of the original mixture of experts model is based on the pre-weighting of the selection network, which belongs to CPU serial operation in the sorting problem and consumes a large amount of computing time. Therefore, as Figure 5 shown, the present invention combines the temperature parameter T-softmax function to further improve the selection of the mixture of experts model by adjusting the probability gap between different categories output by the mixture of experts model. In the knowledge distillation method based on response, the temperature parameter T is greater than 0, which serves to reduce the probability gap between different categories output by the mixture of experts model, enabling the student model to learn more knowledge from the teacher model. On the contrary, in the current selection of the mixture of experts model, it is necessary to select 0 < T < 1, which can further exacerbate the probability gap between different categories output by the mixture of experts model, making the weight assigned to the discarded mixture of experts model approach 0, and the weight of the adopted mixture of experts model approach 1, so as to achieve the effect of both being able to select the "optimal" mixture of experts model and performing parallel operations, further accelerating the inference speed. Then, the initial student model is trained through the optimal expert model, enabling the student model to learn the convolutional layer, fully connected layer, and output of the optimal expert model respectively, and iteratively adjusting the relevant coefficients through the Adam optimizer by itself to achieve the purpose of learning.

[0076] Step Five: Use the second training module to train the initial student model through the correlation points of the optimal expert model and the training set to obtain the intermediate layer of the trained student model capable of communication modulation recognition; then use the second training module to train the output result of the initial student model through the output result of the teacher model and the training set, and combine it with the trained intermediate layer to form the trained student model capable of communication modulation recognition, as Figure 7 shown.

[0077] Step Six: Iterate Step Five. After reaching the preset number of times, the internal data of the student model is locked, and the student model is verified using the validation set. If the knowledge distillation loss function in the second training module converges and the mean square error value shows overfitting, stop the verification to obtain a qualified student model; input the test set into the qualified student model to output the communication modulation recognition result.

[0078] After this step is iterated multiple times, setting the learning parameter of the student model to the teacher model to gradually decrease and the learning parameter of the student model to the dataset to gradually increase can enable the student model to independently learn the modulation mode.

[0079] In step 2, the method for constructing the first training module includes:

[0080] Create the Adam optimizer in Python using the torch.optim.adam function;

[0081] The cross entropy loss function value of the cross entropy loss function is input into the Adam optimizer, and the model to be trained is trained by making the cross entropy loss function value gradient decrease through the Adam optimizer, and the learning rate of each parameter in the model to be trained is dynamically adjusted to optimize the convergence of the cross entropy loss function, thereby obtaining the first training module constructed;

[0082] Among them, the cross entropy loss function is:

[0083] CE = -Σ-tilog(pi);

[0084] Among them, CE is the cross entropy loss function value; ti represents the correct answer, and pi represents the predicted value.

[0085] In step 2, the method for constructing the second training module includes:

[0086] Create the Adam optimizer in Python using the torch.optim.adam function;

[0087] The knowledge distillation loss function value of the knowledge distillation loss function is input into the Adam optimizer, and the knowledge distillation loss function value is gradient-decreased to train the to-be-trained model through the Adam optimizer, and the learning rate of each parameter in the to-be-trained model is dynamically adjusted to optimize the convergence of the knowledge distillation loss function, thereby obtaining the constructed second training module;

[0088] Among them, the knowledge distillation loss function is:

[0089] FKD_LOSS=a*MSE_Loss_result+b*MSE_Loss_hook1+c*MSE_Loss_hook2;

[0090] Among them, FKD_LOSS is the value of the knowledge distillation loss function, a represents the weight of the difference between the features of the student model's self-learning results, b represents the weight of the difference between the features of the convolutional layers, and c represents the weight of the difference between the features of the fully connected layers, which plays a role in reconciling the order of magnitude and weighting; MSE_Loss_result represents the MSE of the student model's self-learning; MSE_Loss_hook1 represents the MSE of the inter-layer features of the same dimension between the convolutional layers, and MSE_Loss_hook2 represents the MSE of the inter-layer features of the same dimension extracted between the fully connected layers. Among them, MSE is the mean square error value, and the mean square error function is expressed as follows:

[0091]

[0092] Among them, Za represents the true value, za represents the predicted value, and n represents the total number of executed predictions.

[0093] In step 2, the construction methods of the initial expert model, the initial selection model and the initial student model include:

[0094] Set the nn.nn.covd function in pytorch to determine the number of dimensions of the convolutional layer;

[0095] The output convolution is flattened and reduced in dimension through the torch.reshape function;

[0096] Then, the intermediate layer is constructed by using the nn.Linear function, nn.RELU function and nn.BatchNorm2d function. After the intermediate layer is fully connected, positively correlated and batch normalized, an initial expert model, an initial selection model and an initial student model are constructed respectively. The number of layer dimensions corresponding to the convolutional layer and the fully connected layer in the initial expert model, the initial student model and the initial selection model is gradually reduced. And the hook function in pytorch is used to set the association points associated with the intermediate layer of the initial expert model between the layers of the initial student model.

[0097] In step 3, the training set is cut according to the signal-to-noise ratio in the training set to obtain multiple training subsets, including:

[0098] The training sets corresponding to the lowest and highest signal-to-noise ratio segments in the training set are cut, and the training sets corresponding to the remaining middle segments are evenly cut to obtain multiple training subsets.

[0099] When setting the signal-to-noise ratio grouping, it is also necessary to note that when the signal-to-noise ratio span of the group in the signal-to-noise ratio grouping is too large, the network will have increased interference, which will lead to poor recognition rate performance, so that the selection model will not tend to be selected, resulting in poor network recognition effect; similarly, if the setting is too detailed, it will lead to an increase in the amount of calculation of the selection model, resulting in a waste of model calculation resources. Therefore, according to the scheme disclosed in the present invention, the RML2016.10b data set can be divided into 6 expert models according to the signal-to-noise ratio, and the 4 lowest signal-to-noise ratio segments (-20dB, -18dB, -16dB, -14dB) and the highest 4 segments (12dB, 14dB, 16dB, 18dB) are respectively set to 1 expert model for learning and recognition, and the rest are evenly distributed, and every 3 segments correspond to 1 expert model, so that it can learn modulation classification and recognition within a relatively reasonable range.

[0100] Since the overall recognition rate of the four groups with the lowest signal-to-noise ratio is low, such as Figure 8As shown in the figure, it contains less information, and is divided into a group mainly for low signal-to-noise ratio recognition. The overall characteristics of the top four groups are obvious. Unifying them to one expert for learning is conducive to increasing the accuracy of feature extraction, improving the role of experts under low signal-to-noise ratios, and making up for the problem of unclear recognition at some junction signal-to-noise ratios. Each group is set up with one expert model for learning and recognition, and the rest are evenly divided into one expert model each, so that they can learn modulation classification and recognition within a relatively reasonable range.

[0101] In step 3, the first training module is used to train the initial expert model through multiple training subsets to obtain multiple trained expert models capable of communication modulation recognition. The method of forming a hybrid expert model with multiple expert models includes:

[0102] Input multiple training subsets into the initial expert model respectively, and output the predicted probability values ​​of batch signals;

[0103] The category label corresponding to the maximum predicted probability value of the signal is used as the predicted category label, the real category label and the predicted category label of the signal are input into the cross entropy loss function in the first training module, and the cross entropy loss function value is calculated;

[0104] The cross entropy loss function value is input into the Adam optimizer, and the initial expert model is trained by making the cross entropy loss function value gradient descend through the Adam optimizer. The learning rate of each parameter in the initial expert model is dynamically adjusted to optimize the convergence of the cross entropy loss function, and multiple trained expert models that can recognize communication modulation corresponding to multiple training subsets are obtained. Multiple expert models constitute a hybrid expert model.

[0105] In step 4, the method of using the first training module to train the initial selection model through the training set and the hybrid expert model includes:

[0106] The features of each expert model in the mixed expert model are extracted through the convolution layer of the initial selection model, and the features of the input training set are extracted at the same time;

[0107] Using the first training module, the cross entropy loss function value of the features of each expert model in the hybrid expert model and the features of the training set is calculated. The initial selection model is trained by gradient descent of the cross entropy loss function value through the Adam optimizer. The learning rate of each parameter in the initial selection model is dynamically adjusted to optimize the convergence of the cross entropy loss function.

[0108] In step 4, the T-softmax function is:

[0109]

[0110] Among them, pT-i is the recognition rate distribution of knowledge distillation results; i is a single signal target; T is the temperature parameter, and the value range is 0 < T < 1; pi is the recognition rate of a single output signal target of the original teacher; pj is the recognition rate of all signal targets output by the original teacher; n is the total number of executions of predictions;

[0111] The selection model learns from the input data to select which expert the data matches for the best learning effect. That is, the smaller the value of the calculated cross-entropy loss function, the closer the distributions of the two probabilities. Select the expert model corresponding to the minimum value to integrate multiple experts; at the same time, classify its integration situation: if weighted, directly output the weights and perform weighted averaging with the experts to output the recognition result. If not selected, use the above T-softmax function. By presetting the value of T in the range of (0,1), make its output behave as a selection.

[0112] Among them, the aggregation model is:

[0113]

[0114] Among them, A is the output result of the teacher model; k is the number of expert models in the teacher model; N is the total number of expert models in the teacher model; Bk is the output weight of the k-th expert model; PBk is the output weight of the k-th expert model corresponding to the selection model.

[0115] In step five, the method of using the second training module to train the initial student model through the correlation points of the optimal expert model and the training set to obtain the intermediate layer of the trained student model capable of communication modulation recognition includes:

[0116] Extract the features of the training set through the convolutional layer of the initial student model. The initial student model extracts the output result of the intermediate layer of the optimal expert model in the teacher model through the correlation points. Use the second training module to calculate the knowledge distillation loss function value between the output result of the intermediate layer of the optimal expert model and the features of the training set. Use the Adam optimizer to make the gradient of the knowledge distillation loss function decrease to train the initial student model, dynamically adjust the learning rate of each parameter in the initial student model, and optimize the convergence of the knowledge distillation loss function to obtain the intermediate layer of the trained student model capable of communication modulation recognition.

[0117] In step five, then use the second training module to train the output result of the initial student model through the output result of the teacher model and the training set, and the method of forming the trained student model capable of communication modulation recognition together with the trained intermediate layer includes:

[0118] The features of the training set are extracted through the convolution layer of the initial student model, and the output results of the teacher model are extracted through the convolution layer of the initial student model. The second training module is used to calculate the knowledge distillation loss function value of the output results of the teacher model and the features of the training set. The initial student model is trained by gradient descent of the knowledge distillation loss function value through the Adam optimizer. The learning rate of each parameter in the initial student model is dynamically adjusted to optimize the convergence of the knowledge distillation loss function value. The trained student model capable of communication modulation recognition is composed of the intermediate layer of the trained student model capable of communication modulation recognition.

[0119] In order to better illustrate the effect of the present invention, the experimental verification results of a knowledge distillation and communication modulation recognition method based on a hybrid expert model proposed by the present invention are disclosed. Figure 8 shown.

[0120] This experiment lists the mixed expert model, the single teacher model, the student model without knowledge distillation, the student model with knowledge distillation for the single teacher model, the student model after FKD with serial operation, and the student model after FKD with parallel operation. Figure 8 It can be found that the accuracy of the model is significantly improved by using the weighting-selection mechanism of the selection network in the hybrid expert model using the technical solution disclosed in the present invention. At the same time, compared with the serial network, the parallel network introduces the T-softmax function, so that the weight assigned to the discarded network approaches 0, and the weight of the adopted network approaches 1, and some critical weighting points will turn a single selection into multiple selections, which can not only improve the training rate, but also slightly improve the accuracy of the model, solving the problem of slow serial training.

[0121] Beneficial effects of the embodiments of the present invention:

[0122] 1. The present invention discloses a knowledge distillation and communication modulation identification method based on a hybrid expert model, which adopts a hybrid expert model to identify the modulation mode of communication signals, further refines the data set in the field of wireless communication, and makes each expert model in the integrated learning recognize the signal features more fully, and the signal feature extraction of high signal-to-noise ratio will not be interfered by the signal of low signal-to-noise ratio.

[0123] 2. The present invention adopts feature-based knowledge distillation to perform lightweight communication signal modulation recognition. By learning the inter-layer relationship of the optimal expert model in the teacher model, the inter-layer information of the communication signal is further utilized. The core of this patent adopts a hybrid expert model integrated selection model as a lightweight teacher model for the communication modulation recognition network, and uses the selection model therein to evaluate the recognition effect of the hybrid expert model, thereby achieving the purpose of integrating the hybrid expert model, and then using the hybrid expert model to guide the training of a single student model in turn, which greatly simplifies the computational complexity of the model and solves the problem that the signal feature extraction of high signal-to-noise ratio will be interfered by the signal of low signal-to-noise ratio to a certain extent, so that the present invention achieves the lightweight of the model as a whole. Compared with the common convolutional neural network model, the student model finally trained can reach less than 30% of its parameters, solving the problem of the increasing size of the model.

[0124] 4. This patent optimizes the selection model in the expert model by introducing the T-softmax function, so that the serial operations sorted on the CPU are converted into parallel operations on the GPU, which increases its reasoning speed in one step, optimizes network performance, and reduces network training time.

[0125] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A knowledge distillation and communication modulation recognition method based on a hybrid expert model, characterized in that: include: Step 1: Use a non-repetitive random sampling method to extract signals in the wireless communication field data set according to a preset ratio, and cut them into a training set, a validation set, and a test set; Step 2: construct the first training module through the Adam optimizer and the cross entropy loss function, and construct the second training module through the Adam optimizer and the knowledge distillation loss function; construct the initial expert model, the initial selection model and the initial student model respectively by adding batch normalization and the fully connected layer in the convolution layer; the number of layers corresponding to the convolution layer and the fully connected layer in the initial expert model, the initial student model and the initial selection model gradually decreases; set the association points associated with the middle layer of the initial expert model between the layers of the initial student model through the hook function in pytorch; Step 3: According to the signal-to-noise ratio in the training set, the training set is cut to obtain multiple training subsets; Using a first training module, the initial expert model is trained through multiple training subsets to obtain multiple trained expert models capable of communication modulation recognition, and the multiple expert models form a hybrid expert model; Step 4: Use the first training module to train the initial selection model through the training set and the hybrid expert model. If the selection model does not weight the hybrid expert model, use the T-softmax function to identify the optimal expert model that matches the training set to obtain a trained selection model; integrate the hybrid expert model with the trained selection model to obtain a teacher model; if the selection model weights the hybrid expert model, use the aggregation model to weight the selection model output weight and the expert model output weight, and output the recognition result with the highest probability, which is the output result of the teacher model; Step 5: Use the second training module to train the initial student model through the associated points of the optimal expert model and the training set to obtain the middle layer of the trained student model capable of communication modulation recognition; Then, the second training module is used to train the output results of the initial student model through the output results of the teacher model and the training set, and the output results of the initial student model are combined with the trained intermediate layer to form a trained student model capable of communication modulation recognition; Step six, iterate step five. After reaching the preset number of times, the internal data of the student model is locked, and the student model is verified using the verification set. If the knowledge distillation loss function in the second training module converges and the mean square error value is overfitted, the verification is stopped to obtain a qualified student model; the test set is input into the qualified student model, and the communication modulation recognition result is output.

2. The method for knowledge distillation and communication modulation recognition based on a hybrid expert model as claimed in claim 1, characterized in that: In step 2, the method for constructing the first training module includes: Create the Adam optimizer in Python using the torch.optim.adam function; The cross entropy loss function value of the cross entropy loss function is input into the Adam optimizer, and the model to be trained is trained by making the cross entropy loss function value gradient decrease through the Adam optimizer, and the learning rate of each parameter in the model to be trained is dynamically adjusted to optimize the convergence of the cross entropy loss function, thereby obtaining the first training module constructed; Among them, the cross entropy loss function is: CE = -Σ-tilog(pi); Among them, CE is the cross entropy loss function value; ti represents the correct answer, and pi represents the predicted value.

3. The method for knowledge distillation and communication modulation recognition based on a hybrid expert model as claimed in claim 1, characterized in that: In step 2, the method for constructing the second training module includes: Create the Adam optimizer in Python using the torch.optim.adam function; The knowledge distillation loss function value of the knowledge distillation loss function is input into the Adam optimizer, and the knowledge distillation loss function value is gradient-decreased to train the to-be-trained model through the Adam optimizer, and the learning rate of each parameter in the to-be-trained model is dynamically adjusted to optimize the convergence of the knowledge distillation loss function, thereby obtaining the constructed second training module; Among them, the knowledge distillation loss function is: FKD_LOSS=a*MSE_Loss_result+b*MSE_Loss_hook1+c*MSE_Loss_hook2; Among them, FKD_LOSS is the value of the knowledge distillation loss function, a represents the weight of the difference between the features of the student model's self-learning results, b represents the weight of the difference between the features of the convolutional layers, and c represents the weight of the difference between the features of the fully connected layers, which plays a role in reconciling the order of magnitude and empowering; MSE_Loss_result represents the MSE of the student model's self-learning; MSE_Loss_hook1 represents the MSE of the inter-layer features of the same dimension between convolutional layers, and MSE_Loss_hook2 represents the MSE of the inter-layer features of the same dimension extracted between fully connected layers. MSE is the mean square error value, and the mean square error function is expressed as follows: Among them, Za represents the true value, za represents the predicted value, and n represents the total number of executed predictions.

4. The method for knowledge distillation and communication modulation recognition based on a hybrid expert model as claimed in claim 1, characterized in that: In step 2, the construction methods of the initial expert model, the initial selection model and the initial student model include: Set the nn.nn.covd function in pytorch to determine the number of dimensions of the convolutional layer; The output convolution is flattened and reduced in dimension through the torch.reshape function; Then, the intermediate layer is constructed by using the nn.Linear function, nn.RELU function and nn.BatchNorm2d function. After the intermediate layer is fully connected, positively correlated and batch normalized, an initial expert model, an initial selection model and an initial student model are constructed respectively. The number of layer dimensions corresponding to the convolutional layer and the fully connected layer in the initial expert model, the initial student model and the initial selection model is gradually reduced. And the hook function in pytorch is used to set the association points associated with the intermediate layer of the initial expert model between the layers of the initial student model.

5. The method for knowledge distillation and communication modulation recognition based on a hybrid expert model as claimed in claim 1, characterized in that: In step 3, the training set is cut according to the signal-to-noise ratio in the training set to obtain multiple training subsets, including: The training sets corresponding to the lowest and highest signal-to-noise ratio segments in the training set are cut, and the training sets corresponding to the remaining middle segments are evenly cut to obtain multiple training subsets.

6. The method for knowledge distillation and communication modulation recognition based on a hybrid expert model as claimed in claim 5, characterized in that: In step 3, the first training module is used to train the initial expert model through multiple training subsets to obtain multiple trained expert models capable of communication modulation recognition. The method of forming a hybrid expert model with multiple expert models includes: Input multiple training subsets into the initial expert model respectively to output the predicted probability values of batch signals; Take the class label corresponding to the maximum predicted probability value of the signal as the predicted class label, and input the true class label and the predicted class label of the signal into the cross-entropy loss function in the first training module to calculate the cross-entropy loss function value; Input the cross-entropy loss function value into the Adam optimizer. Through the Adam optimizer, make the cross-entropy loss function value gradient descent to train the initial expert model, dynamically adjust the learning rate of each parameter in the initial expert model, optimize the convergence of the cross-entropy loss function, and obtain multiple trained expert models capable of communication modulation recognition corresponding to multiple training subsets. The multiple expert models form a mixture of experts model.

7. The method for knowledge distillation and communication modulation recognition based on a hybrid expert model as claimed in claim 1, characterized in that: In step four, the method of using the first training module to train the initial selection model through the training set and the mixture of experts model includes: Extract the features of each expert model in the mixture of experts model through the convolutional layer of the initial selection model, and at the same time extract the features of the input training set; Use the first training module to calculate the cross-entropy loss function value between the features of each expert model in the mixture of experts model and the features of the training set, and make the cross-entropy loss function value gradient descent through the Adam optimizer to train the initial selection model, dynamically adjust the learning rate of each parameter in the initial selection model, and optimize the convergence of the cross-entropy loss function.

8. The method for knowledge distillation and communication modulation recognition based on a hybrid expert model as claimed in claim 1, characterized in that: In step four, the T-softmax function is: where pT-i is the recognition rate distribution of knowledge distillation results; i is a single signal target; T is the temperature parameter, and the value range is 0 < T < 1; pi is the recognition rate of the original teacher's single output signal target; pj is the recognition rate of all signal targets output by the original teacher; n is the total number of predictions made; The aggregation model is: where A is the output result of the teacher model; k is the number of expert models in the teacher model; N is the total number of expert models in the teacher model; Bk is the output weight of the k-th expert model; PBk is the output weight of the k-th expert model corresponding to the selection model.

9. The method for knowledge distillation and communication modulation recognition based on a hybrid expert model as claimed in claim 1, characterized in that: In step five, the method of using the second training module to train the initial student model through the associated points of the optimal expert model and the training set to obtain the intermediate layer of the trained student model capable of communication modulation recognition includes: Extract the features of the training set through the convolutional layer of the initial student model. The initial student model extracts the output result of the intermediate layer of the optimal expert model in the teacher model through the associated points. Use the second training module to calculate the knowledge distillation loss function value between the output result of the intermediate layer of the optimal expert model and the features of the training set, and make the knowledge distillation loss function value gradient descent through the Adam optimizer to train the initial student model, dynamically adjust the learning rate of each parameter in the initial student model, and optimize the convergence of the knowledge distillation loss function to obtain the intermediate layer of the trained student model capable of communication modulation recognition.

10. The method for knowledge distillation and communication modulation recognition based on a hybrid expert model according to claim 9, characterized in that: In step five, then use the second training module to train the output result of the initial student model through the output result of the teacher model and the training set, and form the trained student model capable of communication modulation recognition with the trained intermediate layer. The method includes: The features of the training set are extracted through the convolution layer of the initial student model, and the output results of the teacher model are extracted through the convolution layer of the initial student model. The second training module is used to calculate the knowledge distillation loss function value of the output results of the teacher model and the features of the training set. The initial student model is trained by gradient descent of the knowledge distillation loss function value through the Adam optimizer. The learning rate of each parameter in the initial student model is dynamically adjusted to optimize the convergence of the knowledge distillation loss function value. The trained student model capable of communication modulation recognition is composed of the intermediate layer of the trained student model capable of communication modulation recognition.

Images