Incremental spectrum prediction method and system based on bidirectional knowledge distillation

Abstract

The invention discloses an incremental spectrum prediction method and system based on bidirectional knowledge distillation, and the system comprises a teacher model and two student models, the teacher model is trained based on historical spectrum data, and serves a prediction task of historical data; the first student model reconstructs spectrum potential features through reverse knowledge distillation, and potential feature differences are obtained based on output of the first student model and the teacher model to realize spectrum drift detection; and when drifting occurs, the lightweight second student model rapidly deploys a new prediction task by performing knowledge distillation on the teacher model and combining drifting data. The method is based on bidirectional knowledge distillation, the first student model is based on reverse knowledge distillation, the output characteristic difference is amplified when the frequency spectrum data generates drift, so that accurate frequency spectrum drift detection is realized, the second student model utilizes forward knowledge distillation, the training cost is greatly reduced, and the training efficiency is improved. And high-precision prediction and robustness can be maintained for historical data and drift data at the same time.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of spectrum prediction, more particularly, to an incremental spectrum prediction method based on bidirectional knowledge distillation and an incremental spectrum prediction system based on bidirectional knowledge distillation. BACKGROUND

[0002] With the diversification of wireless services and the densification of communication devices, the electromagnetic environment is increasingly complex and crowded. Under this background, the contradiction between the scarce spectrum resources and the surging communication demand poses a severe challenge to spectrum management. Spectrum prediction technology predicts the spectrum state by mining the law of historical measurement results, thereby realizing optimal management.

[0003] Current spectrum prediction research mainly falls into two paradigms: model-driven methods and data-driven methods. Model-driven methods such as autoregressive models and hidden Markov models can only mine the evolution law of a single dimension and are limited by prior assumptions, resulting in poor actual prediction performance.

[0004] Compared with model-driven methods, data-driven methods based on deep learning show greater prediction potential. Long short-term memory networks can perform multi-step, different time resolution spectrum prediction, but only utilize the spectrum time dimension features. The hybrid architecture of convolutional neural networks and long short-term memory networks can simultaneously extract spatial and temporal features, and is particularly suitable for multi-channel spectrum sharing scenarios. However, due to factors such as the displacement of transceiver devices and dynamic access of users, the distribution of spectrum measurement data continues to evolve, and the measurement results will drift over time, causing the spectrum parameters to deviate from the historical evolution law, which makes it difficult for the prediction model to maintain robustness. SUMMARY

[0005] Existing spectrum prediction does not consider the problem of spectrum prediction under spectrum drift conditions, making it difficult for the prediction model to maintain robustness.

[0006] To solve the above problems, the technical scheme provided by the present application is as follows: a bidirectional knowledge distillation-based incremental spectrum prediction method, comprising the following steps: using a neural network as a teacher model, collecting historical spectrum data to construct training samples and training the teacher model, and freezing the training parameters of the teacher model after training; using an architecture symmetrical to the teacher model to construct a first student model and using the training samples for reverse knowledge distillation training; collecting to-be-measured spectrum data and inputting the to-be-measured spectrum data into the teacher model and the first student model, respectively, detecting whether the to-be-measured spectrum data has generated spectrum drift according to the difference value of the output features of the teacher model and the first student model; if the to-be-measured spectrum data has not generated the spectrum drift, completing spectrum prediction of the to-be-measured spectrum data based on the teacher model; if the to-be-measured spectrum data has generated the spectrum drift, performing knowledge distillation on the teacher model to form a second student model, and performing online knowledge distillation training on the second student model by collecting part of the spectrum data that has generated drift, and completing incremental spectrum prediction of the to-be-measured spectrum data by the second student model after training.

[0007] As a further improvement of the present application, the teacher model specifically comprises a cascaded CNN convolutional layer (Convolutional Neural Network), a BiLSTM (Bidirectional Long Short-Term Memory) layer and an LSTM (Long Short-Term Memory) prediction layer, wherein: the CNN convolutional layer is used to extract frequency domain features of the spectrum; the BiLSTM layer is used to extract time domain features of the spectrum; and the LSTM prediction layer is used to integrate the frequency domain features and the time domain features and output a prediction vector.

[0008] As a further improvement of the present application, according to the difference value of the output features of the teacher model and the first student model for the to-be-measured spectrum data, whether the to-be-measured spectrum data has generated spectrum drift is detected, specifically comprising: calculating the difference value d of the output features of the first student model and the teacher model; setting a spectrum drift threshold τ according to the historical spectrum data and a three-sigma criterion; if d≤τ exists, it is detected that the to-be-measured spectrum data has not generated the spectrum drift; and if d>τ exists, it is detected that the to-be-measured spectrum data has generated the spectrum drift.

[0009] As a further improvement of the present application, the loss function of the first student model comprises: a reconstruction loss wherein,

[0010]

[0011] In the formula, respectively represent the features extracted by the kth module of the teacher model and the first student model, P represents the cosine similarity loss, H k and W k represent the height and width of the feature map S, h represents the hth row of the feature map S, and w represents the wth column of the feature map S.

[0012] As a further improvement of the present application, the second student model is constructed using the following formula to construct a knowledge distillation loss function:

[0013] wherein,

[0014] In the formula, represents the knowledge distillation loss function, and a represents a balance factor, represents the MSE loss of the teacher model, represents the output of the teacher model, represents the output of the second student model, represents the MSE loss of the second student model, and X' represents the generated drift spectrum data.

[0015] To solve the above problems, in a second aspect, the present application provides a technical scheme of an incremental spectrum prediction system based on bidirectional knowledge distillation, which comprises: a first training module, which is used to adopt a neural network as a teacher model, collect historical spectrum data to construct training samples, and train the teacher model; after training, the training parameters of the teacher model are frozen; a second training module, which is used to construct a first student model using an architecture symmetrical to the teacher model and perform reverse knowledge distillation training using the training samples; a judgment module, which is used to collect to-be-tested spectrum data and input the to-be-tested spectrum data into the teacher model and the first student model, respectively, and detect whether the to-be-tested spectrum data has generated a spectrum drift according to the difference value of the output features of the to-be-tested spectrum data in the teacher model and the first student model; and a prediction module, which is used to complete prediction of the to-be-tested spectrum data based on the teacher model if the to-be-tested spectrum data has not generated the spectrum drift, and perform knowledge distillation on the teacher model to form a second student model, and perform online knowledge distillation training on the second student model by collecting part of the spectrum data that has generated a drift, and complete incremental spectrum prediction of the to-be-tested spectrum data through the second student model after training.

[0016] As a further improvement of the present application, the teacher model specifically comprises a cascaded CNN convolution layer, a BiLSTM layer, and an LSTM prediction layer, wherein: the CNN convolution layer is used to extract frequency domain features of the spectrum; the BiLSTM layer is used to extract time domain features of the spectrum; and the LSTM prediction layer is used to integrate the frequency domain features and the time domain features and output a prediction vector.

[0017] As a further improvement of the present application, the determining module is specifically configured to: calculate a difference value d of the first student model output feature and the teacher model output feature; set a spectrum shift threshold value τ according to the historical spectrum data and a three-sigma criterion; if there is d≤τ, it is detected that the to-be-tested spectrum data does not produce the spectrum shift; if there is d>τ, it is detected that the to-be-tested spectrum data produces the spectrum shift.

[0018] As a further improvement of the present application, the loss function of the first student model comprises: a reconstruction loss wherein,

[0019]

[0020] wherein, represents a knowledge distillation loss function, and α represents a balance factor, represents an MSE loss of the teacher model, represents an output of the teacher model, represents an output of the second student model, represents an MSE loss of the second student model, and X represents the spectrum data producing the shift.

[0021] As a further improvement of the present application, the second student model adopts the following formula to construct a knowledge distillation loss function:

[0022] wherein, represents a knowledge distillation loss function, and α represents a balance factor, represents an MSE loss of the teacher model, represents an output of the teacher model, represents an output of the second student model, represents an MSE loss of the second student model, and X' represents the spectrum data producing the shift.

[0023] To solve the above problems, in a third aspect, the present application provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the incremental spectrum prediction method based on bidirectional knowledge distillation of the first aspect.

[0024] To solve the above problems, in a fourth aspect, the present application provides a non-transitory computer readable storage medium, which stores a computer program, and the program is executed by a processor to implement the incremental spectrum prediction method based on bidirectional knowledge distillation of the first aspect.

[0025] Advantages

[0026] Compared with the prior art, the technical scheme provided by the application has the following beneficial effects:

[0027] The increment spectrum prediction designed by the application is based on bidirectional knowledge distillation, a first student model is based on reverse knowledge distillation, and when spectrum data drift occurs, the output feature difference with a teacher model is amplified, thereby realizing accurate spectrum drift detection; a second student model utilizes forward knowledge distillation and adopts a network architecture that is completely consistent with that of the teacher model but has more refined layers, thereby significantly reducing the model parameter quantity, greatly reducing the training cost, and being capable of maintaining high-precision prediction for historical data and drift data, and realizing lower prediction model complexity and higher prediction accuracy and robustness under the condition of spectrum drift. BRIEF DESCRIPTION OF DRAWINGS

[0028] Figure 1 is a flowchart of the increment spectrum prediction method based on bidirectional knowledge distillation according to an embodiment of the application;

[0029] Figure 2 is a prediction framework schematic diagram of the increment spectrum prediction method based on bidirectional knowledge distillation according to an embodiment of the application;

[0030] Figure 3 is a spectrum drift detection performance analysis diagram according to an embodiment of the application;

[0031] Figure 4 is a GSM1800 downlink band prediction error comparison diagram under different numbers of increment samples according to an embodiment of the application;

[0032] Figure 5 is a TV band prediction error comparison diagram under different numbers of increment samples according to an embodiment of the application;

[0033] Figure 6 is a GSM1800 band prediction error comparison diagram under different distillation loss control coefficients according to an embodiment of the application;

[0034] Figure 7 is a TV band prediction error comparison diagram under different distillation loss control coefficients according to an embodiment of the application;

[0035] Figure 8 is a block schematic diagram of the increment spectrum prediction system based on bidirectional knowledge distillation according to an embodiment of the application. DETAILED DESCRIPTION

[0036] For further understanding of the application, the application is described in detail in combination with the drawings and embodiments.

[0037] The structure, proportion, size, etc. shown in the drawings of the specification are only used to cooperate with the content disclosed in the specification, to be understood and read by those skilled in the art, and are not used to limit the conditions of the implementation of the present application, so they do not have technical significance. Any modification of the structure, change of the proportion relationship or adjustment of the size, without affecting the effect and purpose that can be achieved by the present application, should still fall within the scope of the technology disclosed by the present application. At the same time, the terms such as "upper", "lower", "left", "right", "middle" and the like used in the specification are only for the convenience of clear description, and are not used to limit the scope of implementation, and the change or adjustment of the relative relationship is also considered as the implementation of the present application without changing the technical content.

[0038] Figure 1 is a flowchart of a bidirectional knowledge distillation-based incremental spectrum prediction method according to an embodiment of the present application, as shown in Figure 1 The method comprises the following steps:

[0039] Step S1, a neural network is used as a teacher model, historical spectrum data is collected to construct training samples and train the teacher model, and the training parameters of the teacher model are frozen after training.

[0040] Specifically, the historical spectrum data can include the spectrum intensity of the television broadcast band 470-566MHz and the GSM1800 downlink band 1810-1895MHz, the sampling interval is 5 seconds, and the frequency resolution is 200kHz. The training sample contains 9 consecutive time points, and the prediction target of the 10th time point is a 425 or 480-dimensional spectrum vector. The dimension of the spectrum intensity matrix is 9x425 or 9x480.

[0041] In an embodiment of the present application, the teacher model specifically includes a cascaded CNN convolution layer, a BiLSTM layer and an LSTM prediction layer, wherein the CNN convolution layer is used to extract the frequency domain features of the spectrum; the BiLSTM layer is used to extract the time domain features of the spectrum; and the LSTM prediction layer is used to integrate the frequency domain features and the time domain features and output a prediction vector. The cascade can be a combination structure of series or parallel.

[0042] Specifically, the teacher model adopts CNN, BiLSTM and LSTM, which is used for prediction of old tasks. CNN is used to capture the frequency features of spectrum data, while BiLSTM is responsible for extracting time features. Finally, LSTM is used to integrate time-frequency features for prediction. In order to optimize the prediction accuracy, the teacher model selects the mean squared error (MSE) loss to adjust the parameters of the teacher model, as shown below

[0043]

[0044] wherein, MSE loss of the teacher model, The predicted output of the teacher model. Based on the task requirements, the historical spectrum data is selected as the training data set of the teacher model. Therefore, the teacher model exhibits accurate prediction performance on non-drift spectrum data.

[0045] Step S2, a first student model is constructed using an architecture symmetrical to the teacher model and is trained using the training samples for reverse knowledge distillation.

[0046] Specifically, for the challenge of efficiently detecting spectrum drift, the present application proposes an innovative encoder-decoder structure, which optimizes the parameters θ2 of the first student model through reverse distillation technology. The teacher model serves as the encoder E in the first student model architecture, and the first student model serves as the decoder D, both of which exhibit a completely symmetrical topological structure. This symmetry ensures the consistency of the feature representation dimensions between the first student model and the teacher model. During training, reverse distillation technology is used to extract knowledge from the teacher model and transfer it to the first student model. The predicted output generated by the teacher model serves as the input of the first student model, and through reverse distillation, the first student model can reconstruct high-dimensional features from the teacher model output. At the same time, a reconstruction loss is introduced to enhance the feature reconstruction capability of the first student model, and its calculation formula is defined as follows

[0047]

[0048] wherein S and S respectively represent the features extracted by the kth module of the teacher model and the first student model, P is the cosine similarity loss, H k and W k represent the height and width of the feature map S, h represents the hth row of the feature map S, and w represents the wth column of the feature map S.

[0049] To enhance the feature distinction between modules, the cosine similarity loss P of all module feature maps S is aggregated to construct the reconstruction loss The calculation formula of this loss P is as follows:

[0050]

[0051] The first student model takes the multi-scale features output by the CNN convolution layer, BiLSTM layer, and LSTM prediction layer of the teacher model as the supervision target and is trained by minimizing the feature cosine similarity loss.

[0052] During the entire training process, the parameters of the teacher model remain frozen, and the training data set remains completely consistent with the old task. In the testing phase, thanks to the reverse distillation technology, the first student model can learn the ability of the teacher model to extract features from historical spectrum data.

[0053] Step S3, collect the to-be-tested spectrum data and input the to-be-tested spectrum data into the teacher model and the first student model respectively, and detect whether the to-be-tested spectrum data has generated spectrum drift according to a difference value of output features of the to-be-tested spectrum data in the teacher model and the first student model.

[0054] Specifically, for historical data, the teacher model and the first student model will extract highly similar feature representations; but when processing untrained drift data, the difference value d of the features will be significantly amplified. The significant change in feature difference is an effective indicator for achieving accurate spectrum drift detection.

[0055] In one specific embodiment of the present application, detecting whether the to-be-tested spectrum data has generated spectrum drift according to a difference value of output features of the to-be-tested spectrum data in the teacher model and the first student model specifically includes: calculating a difference value d of the first student model output feature and the teacher model output feature; setting a spectrum drift threshold τ according to historical spectrum data and a three-sigma criterion; if there is d≤τ, detecting that the to-be-tested spectrum data has not generated spectrum drift; if there is d>τ, detecting that the to-be-tested spectrum data has generated spectrum drift.

[0056] Specifically, the process of calculating the drift threshold τ is: extracting feature vectors of historical data in the teacher model CNN convolution layer, BiLSTM layer and LSTM prediction layer; calculating a difference value d of the first student model reconstruction feature and the teacher feature; setting a threshold τ according to a three-sigma criterion, μ is the mean of d, σ is the standard deviation, and λ=2.

[0057] Wherein, the process of calculating the feature difference value d is: extracting the teacher model CNN convolution layer, BiLSTM layer and LSTM prediction layer features and the corresponding output layer features of the first student model to calculate d, d represents the sum of cosine similarities between different layer features.

[0058] Step S4, if the to-be-tested spectrum data has not generated spectrum drift, completing spectrum prediction of the to-be-tested spectrum data based on the teacher model.

[0059] Step S5, if the to-be-tested spectrum data has generated spectrum drift, performing knowledge distillation on the teacher model to form a second student model, and performing online knowledge distillation training on the second student model by collecting part of the spectrum data generating drift, and completing incremental spectrum prediction of the to-be-tested spectrum data through the second student model after the training is completed.

[0060] Specifically, to alleviate the negative impact of spectrum drift on prediction robustness, the present application employs a knowledge distillation technique to transfer the prediction ability of the teacher model on historical spectrum data to the second student model. Then, 50 sets of drift data are collected from the to-be-tested spectrum data to construct a training set and perform online knowledge distillation training on the second student model. After the training is completed, the second student model takes over the prediction task and starts rollback monitoring, and switches back to the teacher model when the verification set MSE is greater than 130% of the teacher model.

[0061] The present application focuses on incremental spectrum prediction, which refers to the change in the data distribution of historical spectrum data (spectrum drift), so that incremental prediction needs to be adapted to the new data distribution.

[0062] The prediction task performed by the teacher model is based on historical spectrum data (corresponding to the old task), and when spectrum drift occurs, the change in statistical parameters will cause the original task to fail. Therefore, a new prediction task needs to be constructed to enable the second student model to adapt to both drift data and historical data, thereby enhancing prediction robustness. The knowledge distillation loss function of the second student model is constructed as follows:

[0063]

[0064] wherein, represents the knowledge distillation loss function, and a represents a balance factor, represents the MSE loss of the teacher model, represents the output of the teacher model, represents the output of the second student model, represents the MSE loss of the second student model, and X' represents the drift spectrum data. The value of a is between 0.1 and 0.9.

[0065] In the knowledge distillation loss function, the former represents the distillation loss, which extracts training knowledge from the teacher model, and the latter extracts the frequency rule from the new data.

[0066] The application of the second student model has dual advantages: on the one hand, with the help of knowledge distillation technology, the second student model adopts a network architecture that is completely consistent with the teacher model but has a more streamlined number of layers, which significantly reduces the number of model parameters, not only greatly reducing the training cost, but also ensuring the real-time performance of spectrum prediction. On the other hand, the ultimate goal of this prediction task is to enable the second student model to achieve incremental prediction while maintaining the performance of new and old tasks. Knowledge distillation technology enables the second student model to maintain high-precision prediction on both historical data and drift data, thereby eliminating the need to retain the historical spectrum data set of the old task.

[0067] The online knowledge distillation training comprises a total loss function, a distillation loss and an optimizer; the total loss function is L = a x distillation loss + (1-a) x MSE loss; the preposition coefficient of the distillation loss is controlled to constrain the probability distribution of the second student model; and the optimizer is an SGD with a learning rate of 0.01 and a momentum of 0.9, and training is performed for 3 rounds.

[0068] The parameter quantity of the second student model is 0.14M, which is 45% of the parameter quantity of the teacher model; the single round training time is ≤0.03 seconds; and the prediction response time is ≤1.99 seconds.

[0069] As shown in Figure 2 The prediction framework T-2S of the incremental spectrum prediction method based on bidirectional knowledge distillation designed by the application comprises three modules: a teacher model and two student models. The teacher model is trained based on historical spectrum data and serves the prediction task of the historical data; the first student model reconstructs the spectrum potential features through reverse knowledge distillation, and realizes spectrum drift detection based on the difference between the potential features obtained by the first student model and the teacher model; when drift occurs, the lightweight second student model quickly deploys a new prediction task by performing knowledge distillation on the teacher model and combining the drift data.

[0070] In order to verify the effectiveness and robustness of the method designed by the application, real data from different frequency bands, including television broadcast and GSM1800 downlink frequency bands, are collected for experimental verification, and the spectrum state under different scenes is changed to simulate spectrum drift. The detailed description of the data set is shown in Table 1. In addition, there are 300 training samples for each frequency band, and the test set contains 100 historical samples and 100 drift samples. The designed method predicts the spectrum state at the next time based on the historical data of the past 9 time points.

[0071] Table 1

[0072] Parameter Value Frequency band Television broadcast, GSM 1800 downlink frequency band Location Site A, B, Jiangsu, Nanjing Monitoring time 2024.11.21-2024.11.23 Frequency resolution 200 KHz Time resolution 5 seconds Sample dimension (time, frequency, scene) 4400×480×2,4550×425×2

[0073] Experimental analysis setting:

[0074] In order to further explore the performance of the designed model, experiments focusing on spectrum drift detection and prediction performance analysis are designed. The prediction accuracy is quantified by three evaluation indicators: mean squared error (MSE), root mean squared error (RMSE) and mean absolute error (MAE), of which the mean squared error is selected as the main indicator for performance comparison. The spectrum drift detection capability is evaluated by simultaneously measuring the detection probability and false alarm probability under different thresholds.

[0075] As shown in Figure 3As shown, the false alarm probability of spectral drift presents a continuous downward trend with the increase of λ, while the detection probability presents a change feature of first rising and then falling. It is worth noting that when λ = 2, the average detection performance reaches the best state, and more than 90% of the detection probability is achieved in two frequency bands, while ensuring that the false alarm probability is always less than 5%. The results show that the data distribution of the spectral measurement result will shift in different scenarios. Therefore, detecting spectral drift is crucial to maintaining the robustness of the spectral prediction model.

[0076] In subsequent experiments, in order to facilitate the evaluation of the change of prediction accuracy, all spectral drift samples are assumed to be effectively detected. In order to highlight the advantages of the proposed method, the performance changes between CNN-LSTM, CNN-BiLSTM, single teacher model, and joint training method and the designed method of the application are compared and analyzed.

[0077] Table 2 shows that the designed method of the application significantly improves the prediction accuracy during spectral drift. In the GSM1800 downlink frequency band, the prediction error MSE is reduced by 27.4%, 26.3% and 24.7% compared with CNN-LSTM, CNN-BiLSTM and TM respectively; while in the TV (television broadcast) frequency band, the reduction is 58.9%, 45.2% and 44.5%. This is because spectral drift changes the distribution of historical spectral data, making it difficult for traditional prediction methods to maintain high accuracy. In further research, the influence of the number of available spectral samples for incremental learning on prediction performance is quantitatively analyzed, and the balance between historical task knowledge and new task adaptation is adjusted by adjusting factor α to optimize accuracy.

[0078] In addition, when comparing T-2S with T-2S (joint training), the experimental results show that the prediction error is still reduced by 17.4% and 29.3% respectively. This phenomenon is due to two factors: on the one hand, there is an imbalance between the training data of historical samples and drift samples, resulting in better prediction results for the model on the side with more training samples; on the other hand, the model is difficult to achieve balanced prediction accuracy between different data distributions.

[0079] Table 2

[0080]

[0081] In addition, the application studies how the number of spectral samples available for incremental learning affects prediction performance. At the same time, the control factor α is analyzed to adjust the trade-off between historical task knowledge and new task adaptation to optimize prediction accuracy.

[0082] Figure 4 and Figure 5It is shown that the mean square error (MSE) of the old task increases while the MSE of the new task decreases as the increment of the available sample size of the increment learning. The average prediction performance is continuously improved, but the improvement rate gradually slows down. When the available sample size is 50 compared to 30 and 10, Figure 5 The average MSE of the GSM1800 downlink band increases by 1.0% and 8.7% respectively, Figure 6 The average MSE of the TV broadcast band decreases by 3.8% and 11.6% respectively. This phenomenon is mainly caused by the increase in the amount of spectral drift data. More new task data enriches the prior information, thereby enhancing the robustness of the model, but it will lead to a significant decrease in the prediction ability of historical spectral data.

[0083] As Figure 6 and Figure 7 It is shown that different values of the parameter α have a significant impact on the prediction performance of the second student model. When α is close to 0.5, the model achieves the best balance between old and new tasks. As Figure 7 and Figure 8 It is shown that in the GSM band, the MSE when α = 0.5 is reduced by 12.6% and 10.8% compared to α = 0.3 and α = 0.7 respectively; in the TV band, it is reduced by 1.7% and 9.8% respectively. Experiments show that an increase in the value of α improves the performance of the model on the old task, while a decrease in the value of α is beneficial to the new task. This phenomenon is due to the role of α in regulating the weight distribution between old and new tasks during training - a lower α value means that the model pays more attention to learning the distribution characteristics of spectral drift data, thereby enhancing the adaptability to the new task.

[0084] Computational complexity analysis

[0085] Spectrum prediction imposes strict requirements on real-time performance. To solve this problem, the method proposed in the invention optimizes the computational efficiency through knowledge distillation technology, thereby compressing the complex prediction model. This method can also maintain the prediction accuracy of the old task without relying on a large amount of historical data, further reducing the consumption of computing resources and improving real-time performance. To verify the computational complexity of T-2S, key indicators such as the number of model parameters and running time are shown, and are compared with the joint training scheme.

[0086] Table 3

[0087]

[0088] Table 3 shows that in the T-2S framework, the parameter amount of the teacher model, the first student model and the second student model is 0.31M, 0.14M and 0.14M respectively. It is worth noting that by adopting the knowledge distillation technology, the second student model realizes about 50% parameter amount compression (from 0.31M to 0.14M) compared with the teacher model, which significantly reduces the parameter size of the model. Compared with the prediction methods such as CNN-LSTM and CNN-BiLSTM, the T-2S system not only contains a spectrum prediction module, but also integrates a drift detection module. Although this dual-component architecture introduces additional computational parameters, the increase in overall complexity is still within an acceptable range.

[0089] Table 3 data shows that the offline training time of each round of the teacher model and the first student model is 0.23 seconds and 0.35 seconds respectively, while the second student model needs to perform online training of 0.03 seconds per round. The reason why the training time of the first student model increases compared with the teacher model is that the prediction output of the teacher model needs to be called during the training process of the first student model. In contrast, the second student model realizes the shortest training time because it only needs to optimize a small sample.

[0090] In addition, the TM (joint training) mode has the longest online training time, reaching 0.26 seconds per round. Compared with joint training, the incremental training scheme can reduce the running time by about 88%. This is because joint training needs to process historical data and drift data synchronously, but Table 3 shows that its prediction accuracy does not significantly improve - this phenomenon is due to the imbalance between the amount of new and old data samples, which causes the prediction performance of the model on any subset of data to degrade. The method proposed in the present application effectively alleviates this problem by introducing a balance factor α to dynamically adjust the performance of the dual task, making the T-2S framework exhibit advantages in both training efficiency and prediction performance.

[0091] In summary, according to the incremental spectrum prediction method based on bidirectional knowledge distillation according to the embodiments of the present application, based on bidirectional knowledge distillation, the first student model is based on reverse knowledge distillation, which amplifies the output feature difference with the teacher model when the spectrum data drifts, thereby realizing accurate spectrum drift detection, and the second student model uses forward knowledge distillation and adopts a network architecture identical to the teacher model but with more refined layers, which significantly reduces the model parameter amount, not only greatly reduces the training cost, but also can maintain high-precision prediction for historical data and drift data, and can realize lower prediction model complexity and higher prediction accuracy and robustness under spectrum drift conditions.

[0092] Corresponding to the above-mentioned incremental spectrum prediction method based on bidirectional knowledge distillation, the present application also proposes an incremental spectrum prediction system based on bidirectional knowledge distillation.

[0093] Figure 8is a block schematic diagram of a bidirectional knowledge distillation based incremental spectrum prediction system according to an embodiment of the present application, as shown in the figure, the system comprises a first training module, a second training module, a judgment module and a prediction module. Figure 8

[0094] The first training module is configured to use a neural network as a teacher model, collect historical spectrum data to construct training samples and train the teacher model, and freeze the training parameters of the teacher model after training.The second training module is configured to use an architecture symmetrical to the teacher model to construct a first student model and perform reverse knowledge distillation training on the collected historical spectrum data. The judgment module is configured to input the to-be-tested spectrum data into the teacher model and the first student model, and detect whether the to-be-tested spectrum data has generated spectrum drift according to the difference value of the output features of the to-be-tested spectrum data in the teacher model and the first student model. The prediction module is configured to complete spectrum prediction of the to-be-tested spectrum data based on the teacher model if the to-be-tested spectrum data has not generated spectrum drift.

[0095] As a further improvement of the present application, the teacher model specifically comprises a cascaded CNN convolution layer, a BiLSTM layer and an LSTM prediction layer, wherein: the CNN convolution layer is configured to extract frequency domain features of the spectrum; the BiLSTM layer is configured to extract time domain features of the spectrum; and the LSTM prediction layer is configured to integrate the frequency domain features and the time domain features and output a prediction vector.

[0096] As a further improvement of the present application, the judgment module is specifically configured to: calculate the difference value d of the output features of the first student model and the teacher model; set a spectrum drift threshold τ according to the historical spectrum data and a three-sigma criterion; if d≤τ exists, it is detected that the to-be-tested spectrum data has not generated spectrum drift; and if d>τ exists, it is detected that the to-be-tested spectrum data has generated spectrum drift.

[0097] As a further improvement of the present application, the loss function of the first student model comprises: a reconstruction loss L RD , wherein,

[0098]

[0099] In the formula, S represent the features extracted by the kth module of the teacher model and the first student model respectively, P represents a cosine similarity loss, H k and W k represent the height and width of the feature map S, h represents the hth row of the feature map S, and w represents the wth column of the feature map S.

[0100] As a further improvement of the present application, the second student model adopts the following formula to construct the knowledge distillation loss function:

[0101] wherein, formula

[0102] , represents the knowledge distillation loss function, and alpha represents a balance factor, represents the MSE loss of the teacher model, represents the output of the teacher model, represents the output of the second student model, represents the MSE loss of the second student model, and X' represents the generated drift spectrum data.

[0103] The incremental spectrum prediction system based on bidirectional knowledge distillation according to the embodiment of the present application, based on bidirectional knowledge distillation, the first student model is based on reverse knowledge distillation, when the spectrum data generates drift, amplifies the output feature difference with the teacher model, so as to realize accurate spectrum drift detection, the second student model uses forward knowledge distillation, adopts a network architecture completely consistent with the teacher model but with more simplified layers, significantly reduces the model parameter quantity, not only greatly reduces the training cost, and can keep high-precision prediction for historical data and drift data at the same time, can realize lower prediction model complexity and higher prediction accuracy and robustness under the condition of spectrum drift.

[0104] In addition, the present application also proposes a computer device, comprising a memory, a processor and a computer program stored on the memory and executable on the processor, when the processor executes the program, the incremental spectrum prediction method based on bidirectional knowledge distillation of the first aspect is realized.

[0105] The present application also proposes a non-transitory computer readable storage medium, which stores a computer program, and the program is executed by a processor to realize the incremental spectrum prediction method based on bidirectional knowledge distillation of the first aspect.

[0106] According to the computer device and the non-transitory computer readable storage medium, when the processor executes the computer program stored thereon, based on bidirectional knowledge distillation, the first student model is based on reverse knowledge distillation, and when the spectrum data produces drift, the output feature difference with the teacher model is amplified, thereby realizing accurate spectrum drift detection; the second student model uses forward knowledge distillation, adopts a network architecture that is completely consistent with the teacher model but has more simplified layers, significantly reduces the model parameter quantity, not only greatly reduces the training cost, but also can maintain high-precision prediction on historical data and drift data, and can realize lower prediction model complexity and higher prediction accuracy and robustness under the condition of spectrum drift. The above describes the present application and its embodiments in a schematic manner, which is not restrictive, and the embodiments shown in the drawings are only one of the embodiments of the present application, and the actual structure is not limited thereto. Therefore, if a person of ordinary skill in the art is inspired thereby and designs a similar structure and embodiment without departing from the spirit of the present application, it shall belong to the protection scope of the present application.

Claims

1. An incremental spectrum prediction method based on bidirectional knowledge distillation, characterized in that, Includes the following steps: A neural network is used as the teacher model. Historical spectrum data is collected to construct training samples and train the teacher model. After training is completed, the training parameters of the teacher model are frozen. A first student model is constructed using an architecture symmetrical to the teacher model, and the training samples are used for reverse knowledge distillation training. The spectrum data to be tested is collected and input into the teacher model and the first student model respectively. Based on the difference value of the output features of the spectrum data to be tested in the teacher model and the first student model, it is detected whether the spectrum data to be tested has generated spectrum drift. If the spectrum data to be tested does not produce the spectrum drift, then the spectrum prediction of the spectrum data to be tested is completed based on the teacher model; If the spectrum data to be tested has a spectral drift, the teacher model is subjected to knowledge distillation to form a second student model. The second student model is then trained online by collecting some of the drifted spectrum data. After training, the second student model is used to perform incremental spectrum prediction on the spectrum data to be tested.

2. The incremental spectrum prediction method based on bidirectional knowledge distillation according to claim 1, characterized in that, The teacher model specifically includes cascaded CNN convolutional layers, BiLSTM layers, and LSTM prediction layers, wherein: The CNN convolutional layer is used to extract frequency domain features of the spectrum; The BiLSTM layer is used to extract the temporal features of the spectrum; The LSTM prediction layer is used to integrate the frequency domain features and the time domain features and output a prediction vector.

3. The incremental spectrum prediction method based on bidirectional knowledge distillation according to claim 1, characterized in that, Based on the difference in output features between the test spectrum data and the first student model, the system detects whether the test spectrum data has experienced spectral drift, specifically including: Calculate the difference value d between the output features of the first student model and the output features of the teacher model; The spectral drift threshold τ is set based on the historical spectral data and the three sigma criterion. If d ≤ τ, then the measured spectrum data has not produced the spectrum drift. If d > τ, then the measured spectrum data has been found to have a spectral drift.

4. The incremental spectrum prediction method based on bidirectional knowledge distillation according to claim 1, characterized in that, The loss function of the first student model includes: reconstruction loss. in, In the formula, Let H and P represent the features extracted by the k-th module of the teacher model and the first student model, respectively. Let P represent the cosine similarity loss. k and W k The height and width of feature map S are represented by h, where h represents the h-th row of feature map S and w represents the w-th column of feature map S.

5. The incremental spectrum prediction method based on bidirectional knowledge distillation according to claim 1, characterized in that, The second student model uses the following formula to construct the knowledge distillation loss function: In the formula, This represents the knowledge distillation loss function, where α represents the balance factor. This represents the MSE loss of the teacher model. This represents the output of the teacher model. This represents the output of the second student model. X' represents the MSE loss of the second student model, and X′ represents the generated drift spectrum data.

6. An incremental spectrum prediction system based on bidirectional knowledge distillation, characterized in that, include: The first training module is used to use a neural network as a teacher model, collect historical spectrum data to construct training samples and train the teacher model, and freeze the training parameters of the teacher model after training is completed. The second training module is used to construct a first student model using an architecture symmetrical to the teacher model and to perform reverse knowledge distillation training using the training samples. The judgment module is used to collect the spectrum data to be tested and input it into the teacher model and the first student model respectively, and detect whether the spectrum data to be tested has generated spectrum drift based on the difference value of the output features of the spectrum data to be tested in the teacher model and the first student model. The prediction module is used to perform spectrum prediction on the spectrum data to be tested based on the teacher model if the spectrum data to be tested does not produce the spectrum drift; if the spectrum data to be tested produces the spectrum drift, the teacher model is subjected to knowledge distillation to form a second student model, and the second student model is trained online by collecting some of the drifted spectrum data. After training, the incremental spectrum prediction on the spectrum data to be tested is performed through the second student model.

7. The incremental spectrum prediction system based on bidirectional knowledge distillation according to claim 6, characterized in that, The teacher model specifically includes cascaded CNN convolutional layers, BiLSTM layers, and LSTM prediction layers, wherein: CNN convolutional layers are used to extract frequency domain features of the spectrum; BiLSTM layers are used to extract temporal features of the spectrum; The LSTM prediction layer is used to integrate the frequency domain features and the time domain features and output a prediction vector.

8. The incremental spectrum prediction system based on bidirectional knowledge distillation according to claim 6, characterized in that, The judgment module is specifically used for: Calculate the difference value d between the output features of the first student model and the output features of the teacher model; The spectral drift threshold τ is set based on the historical spectral data and the three sigma criterion. If d ≤ τ, then the measured spectrum data has not produced the spectrum drift. If d > τ, then the measured spectrum data has been found to have a spectral drift.

9. A computer device, characterized in that, The method includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the incremental spectrum prediction method based on bidirectional knowledge distillation according to any one of claims 1-5.

10. A non-transitory computer-readable storage medium, characterized in that, It stores a computer program that, when executed by a processor, implements the incremental spectrum prediction method based on bidirectional knowledge distillation according to any one of claims 1-5.

Images