An informer-based spectrum prediction model and method

Abstract

The present application relates to a kind of based on Informer's spectrum prediction model and method, the model includes the high dynamic system spectrum time series processing module, Informer model and frequency domain attention calculation module interconnected;The method includes the following steps: entering high dynamic system spectrum time series processing module, the original sequence is normalized to obtain corresponding mean, standard deviation and sequence after transformation, after normalization Sequence is embedded to obtain to be sent into encoder, after calculation, obtain original feature map P, carry out the calculation of frequency domain attention using DCT, obtain enhanced vector r, and it is spliced with original feature map P in frequency dimension, obtain the feature map P' that is strengthened by frequency domain attention, again into decoder to obtain the initial prediction result y' of output, y' is denormalized to obtain the final prediction output y.The present application reduces the memory requirement, provides the condition for input longer sequence, has more comprehensive learning to spectrum data, with the characteristics of high accuracy, strong applicability.

Description

Technical Field

[0001] This invention belongs to the field of spectrum management, specifically relating to an Informer-based spectrum prediction model and method. Background Technology

[0002] In intelligent communication systems, spectrum management is a crucial aspect. The spectrum scheduling process follows these steps: spectrum sensing, spectrum analysis, spectrum decision-making, and spectrum adjustment. In today's complex communication environment, comprehensive and real-time spectrum monitoring is relatively resource-intensive and time-consuming, especially in highly dynamic radio environments where the complexity of spectrum sensing is further enhanced. Furthermore, real-time comprehensive spectrum monitoring followed by computational analysis to identify usable spectrum gaps is time-consuming and may lead to conflicts during spectrum access due to monitoring and computational delays, ultimately degrading communication quality. Therefore, spectrum prediction is proposed to reduce computational latency, lower monitoring costs, and provide more data support for subsequent spectrum management.

[0003] Spectrum prediction analyzes historical spectrum usage to predict future spectrum occupancy, thereby enabling flexible use of spectrum resources and improving system communication quality. Especially in highly dynamic electromagnetic environments such as the Internet of Things (IoT), where latency requirements for spectrum switching and information transmission quality are higher, spectrum prediction plays a crucial role and has greater potential to improve the overall communication quality of communication systems. Therefore, spectrum prediction is a highly valuable strategy in intelligent communication, particularly in highly dynamic electromagnetic environments.

[0004] With the rise of new application scenarios such as the Internet of Vehicles, intelligent communication has emerged. The non-renewable nature of spectrum resources requires systems to utilize them more efficiently. Spectrum prediction has great potential in reducing spectrum switching latency and resource consumption. Therefore, in modern communication environments, especially in highly dynamic electromagnetic environments, spectrum prediction is a very valuable research topic.

[0005] Currently, the closest model for sequence prediction is the Transformer model, proposed by Google. This model is based on a full attention mechanism and has shown powerful capabilities in sequence modeling. However, it still has some limitations. (1) High computational complexity: The model uses a full attention mechanism, which requires calculating the attention scores for all positions and times in the input sequence. Assuming the length of the input sequence is L, then a total of L needs to be calculated. 2 The computational complexity of full attention is O(L^2). 2(1) In some experiments with a large amount of data, the requirements for equipment may be high, and there may even be a risk of experimental failure, which limits the application of the model in real-world scenarios. (2) In the spectrum time series, Transformer focuses more on the relationship between the sequence in the time dimension, but information such as interference is difficult to distinguish in the time dimension, but it is easier to distinguish in the frequency domain. The model lacks learning of the frequency domain of the data. (3) In a highly dynamic electromagnetic environment, the stability of the spectrum data may also be poor. Transformer lacks consideration for the processing of such data, which may affect the prediction results. Summary of the Invention

[0006] To address the issues of high computational complexity and lack of frequency domain learning in existing spectrum prediction technologies for long sequences, this invention proposes an Informer-based spectrum prediction model and algorithm. This model reduces memory requirements, provides better conditions for inputting longer sequences, and enables more comprehensive learning of spectrum data, resulting in high accuracy and strong applicability.

[0007] An Informer-based spectrum prediction model includes a high dynamic system spectrum time series processing module, an Informer model, and a frequency domain attention calculation module, wherein the Informer model is connected to both the high dynamic system spectrum time series processing module and the frequency domain attention calculation module.

[0008] Furthermore, the normalization unit of the aforementioned high dynamic system spectrum time series processing module is connected to the encoder of the Informer model, the encoder of the Informer model is connected to the frequency domain attention calculation module, the frequency domain attention calculation module is connected to the decoder of the Informer model, and the decoder of the Informer model is connected to the denormalization unit of the high dynamic system spectrum time series processing module.

[0009] An Informer-based spectrum prediction method includes the following steps:

[0010] Step 1: Enter the high dynamic system spectrum time series processing module. First, normalize the original sequence to obtain the mean, standard deviation and transformed sequence. Then, calculate the non-stationary factor through a multilayer perceptron.

[0011] Step 2: Obtain the normalized sequence through embedding. Will The encoder of the Informer model is fed into the model, and the original feature map P is obtained through the encoder's calculation.

[0012] Step 3: Calculate frequency domain attention on the original feature map P using DCT to obtain the enhancement vector r, and concatenate it with the original feature map P in the frequency dimension to obtain the feature map P' enhanced by frequency domain attention;

[0013] Step 4: Input the frequency domain attention-enhanced feature map P' into the decoder of the Informer model for calculation to obtain the initial prediction result y'.

[0014] Step 5: Perform a denormalization operation on the initial prediction result y' to obtain the final prediction output y.

[0015] Furthermore, step 1 above specifically includes:

[0016] 1) The original sequence Normalization is performed to calculate the mean μ of the original sequence. X and standard deviation σ X , and the transformed sequence X';

[0017]

[0018]

[0019]

[0020] 2) The obtained mean μ X and standard deviation σ X Together with the original sequence X, it is fed into a multilayer perceptron unit to obtain the non-stationary factors τ and Δ in the high dynamic system spectral time series processing module, for use in subsequent sequence recovery, logτ = MLP(σ X ,X), Δ=MLP(μ X ,X).

[0021] Furthermore, step 2 above specifically includes:

[0022] 1) Embed the normalized sequence to obtain

[0023]

[0024] 2) Embedded data The encoder fed into the Informer model first calculates the matrices Q', K', and V' required for attention calculation in each layer of the encoder stack. Then, it calculates sparse attention based on these three matrices and non-stationary factors, selects important attention, replaces the remaining query values ​​with the mean, and performs attention distillation. After N calculations, the original feature map P is obtained.

[0025] Furthermore, in the Informer-based spectrum prediction method described above, if the encoder has N layers, the following steps will be performed N times, specifically:

[0026] 2.1) Calculate Q', K', V' based on linear properties;

[0027]

[0028] 2.2) Calculation

[0029] 2.2.1) Randomly select U dot product pairs from K' to form

[0030] 2.2.2) Calculate the sample score:

[0031]

[0032] 2.2.3) Select Zhongyu The U queries with the largest difference in mean are formed

[0033] 2.2.4) Replace the remaining query values ​​with the mean;

[0034] 2.3) Utilization V',τ,Δ are used to calculate destationary sparse attention;

[0035]

[0036] Where d represents the dimension of the sequence;

[0037] 2.4) Calculate residual connectivity and perform layer normalization;

[0038] 2.5) Train the feedforward network;

[0039] 2.6) Recalculate the residual connectivity and perform layer normalization;

[0040] 2.7) Perform attention distillation

[0041]

[0042] Furthermore, step 3 above specifically includes:

[0043] 1) Calculate a one-dimensional convolution on the original feature map P to obtain V;

[0044] 2) Perform DCT transformation on V.

[0045] Freq = DCT(V) = stack([Freq 0 ,Freq1 ,...,Freq n-1 ]);

[0046] 3) Calculate the frequency domain attention enhancement vector:

[0047] F c -att=σ(W2δ(W1Freq))

[0048] Where W1 and W2 are learnable parameters obtained through training, δ represents the ReLU activation function, and σ represents the Sigmoid activation function;

[0049] 4) Calculate the enhancement vector r through one-dimensional convolution:

[0050] r = P * (F c -att);

[0051] 5) Concatenate P and r along the frequency dimension to calculate the input feature map P' of the enhanced decoder, P' = torch.cat(P,r).

[0052] Furthermore, step 4 above specifically involves inputting the frequency-domain attention-enhanced feature map P' into the decoder of the Informer model for computation, while simultaneously inputting the sequence X into the decoder. de ={X token ,X0},X token Part X0 is the start token, and part X0 indicates the length of the prediction result. After passing through the decoder and the fully connected layer, the output is the initial prediction result y' that was originally set to 0. The decoder structure is a masked multi-head sparse attention mechanism and a multi-head attention mechanism.

[0053] Furthermore, the aforementioned Informer-based spectrum prediction method is calculated at each layer of the decoder stack:

[0054] 1) Computation of multi-head sparse attention mechanisms for occlusion;

[0055] 2) Calculate residual connectivity and perform layer normalization;

[0056] 3) Train the feedforward network;

[0057] 4) Recalculate the residual connectivity and perform layer normalization.

[0058] Furthermore, in step 5 above, the method for calculating the predicted output y is as follows:

[0059]

[0060] The beneficial effects of this invention are:

[0061] 1. The basic model of this invention is the Informer model, which is a computationally less computationally intensive model based on the attention mechanism. Through sparse attention and attention distillation mechanisms, the computational complexity of the model is reduced from O(L...). 2 The time complexity has been reduced to O(LlogL), which lowers memory requirements and provides better conditions for inputting longer sequences;

[0062] 2. This invention incorporates an attention mechanism for frequency domain information, and based on the discovery that GAP is the lowest frequency component of DCT, it adopts the DCT calculation method when calculating frequency domain attention, enabling the model to learn more comprehensively from the spectral data and reduce errors;

[0063] 3. In a highly dynamic electromagnetic environment, this invention preprocesses the collected spectral data to enhance the stationarity of the sequence, thereby providing better attention learning conditions for the model. Furthermore, it restores the non-stationarity of the decoder output, reducing prediction errors and improving the applicability of the model. Attached Figure Description

[0064] Figure 1 This is a schematic diagram of the spectrum prediction model based on Informer in this embodiment.

[0065] Figure 2 This is a schematic diagram showing the connection between the Informer model and the frequency domain attention calculation module in this embodiment.

[0066] Figure 3 This is a flowchart illustrating the frequency domain attention calculation module in this embodiment.

[0067] Figure 4 This is a schematic diagram of the data embedding method in this embodiment. Detailed Implementation

[0068] To further illustrate the technical means and effects of the present invention in achieving its intended purpose, the specific implementation methods, structural features and effects of the present invention are described in detail below with reference to the accompanying drawings and embodiments.

[0069] This embodiment provides a spectrum prediction model based on Informer, see attached... Figure 1-3The spectrum prediction model includes a high-dynamic system spectrum time series processing module, an Informer model, and a frequency domain attention calculation module. The Informer model is connected to both the high-dynamic system spectrum time series processing module and the frequency domain attention calculation module. The normalization unit of the high-dynamic system spectrum time series processing module is connected to the encoder of the Informer model, the encoder of the Informer model is connected to the frequency domain attention calculation module, the frequency domain attention calculation module is connected to the decoder of the Informer model, and the decoder of the Informer model is connected to the denormalization unit of the high-dynamic system spectrum time series processing module.

[0070] This embodiment also provides an Informer-based spectrum prediction method, which includes the following steps:

[0071] Step 1: Enter the high dynamic system spectrum time series processing module. First, normalize the original sequence to obtain the mean, standard deviation and transformed sequence. Then, calculate the non-stationary factor through a multilayer perceptron.

[0072] 1) The original sequence Normalization is performed to calculate the mean μ of the original sequence. X and standard deviation σ X , and the transformed sequence X'.

[0073]

[0074]

[0075]

[0076] 2) The obtained mean μ X and standard deviation σ X Together with the original sequence X, it is fed into a multilayer perceptron unit to obtain the non-stationary factors τ and Δ in the high dynamic system spectral time series processing module, for use in subsequent sequence recovery, logτ = MLP(σ X ,X), Δ=MLP(μ X ,X).

[0077] Step 2: Obtain the normalized sequence through embedding. Will The encoder in the Informer model is fed into the model, and the encoder calculates the original feature map P.

[0078] The encoder structure in this embodiment consists of two stack layers. In each stack layer, a certain number of important attentions are first obtained through a multi-head sparse attention mechanism. Then, the attentions are filtered through a distillation layer to obtain effective attentions. This is also the key to reducing computational complexity and memory requirements in this algorithm.

[0079] 1) Embed the normalized sequence to obtain

[0080]

[0081] Data embedding methods are referenced in the appendix. Figure 4 .

[0082] 2) Embedded data The encoder fed into the Informer model first calculates the matrices Q', K', and V' required for attention calculation in each layer of the encoder stack. Then, it calculates sparse attention based on these three matrices and non-stationary factors, selects important attention, replaces the remaining query values ​​with the mean, and performs attention distillation. After N calculations, the original feature map P is obtained.

[0083] If the encoder has N layers, then the following steps will be performed N times, specifically:

[0084] 2.1) Calculate Q', K', V' based on linear properties;

[0085]

[0086] 2.2) Calculation

[0087] 2.2.1) Randomly select U dot product pairs from K' to form

[0088] 2.2.2) Calculate the sample score:

[0089]

[0090] 2.2.3) Select Zhongyu The U queries with the largest difference in mean are formed

[0091] 2.2.4) Replace the remaining query values ​​with the mean;

[0092] 2.3) Utilization V',τ,Δ are used to calculate destationary sparse attention;

[0093]

[0094] Where d represents the dimension of the sequence;

[0095] 2.4) Calculate residual connectivity and perform layer normalization;

[0096] 2.5) Train the feedforward network;

[0097] 2.6) Recalculate the residual connectivity and perform layer normalization;

[0098] 2.7) Perform attention distillation

[0099]

[0100] Step 3: Calculate frequency domain attention for the original feature map P using DCT. The frequency domain attention module is calculated because traditional channel attention intelligently learns the lowest frequency attention, so DCT is used to calculate it to obtain the enhancement vector r. Then, it is concatenated with the original feature map P in the frequency dimension to obtain the feature map P' enhanced by frequency domain attention.

[0101] 1) Calculate a one-dimensional convolution on the original feature map P to obtain V;

[0102] 2) Perform DCT transformation on V.

[0103] Freq = DCT(V) = stack([Freq 0 ,Freq 1 ,...,Freq n-1 ]);

[0104] 3) Calculate the frequency domain attention enhancement vector:

[0105] F c -att=σ(W2δ(W1Freq))

[0106] Where W1 and W2 are learnable parameters obtained through training, δ represents the ReLU activation function, and σ represents the Sigmoid activation function;

[0107] 4) Calculate the enhancement vector r through one-dimensional convolution:

[0108] r = P * (F c -att);

[0109] 5) Concatenate P and r along the frequency dimension to calculate the input feature map P' of the enhanced decoder, P' = torch.cat(P,r).

[0110] Step 4: Input the frequency-domain attention-enhanced feature map P' into the decoder of the Informer model for computation, and simultaneously input the sequence X into the decoder. de ={Xtoken ,X0},X token Part X0 is the start token, and part X0 indicates the length of the prediction result. After passing through the decoder and the fully connected layer, the output is the initial prediction result y' that was originally set to 0. The decoder structure is a masked multi-head sparse attention mechanism and a multi-head attention mechanism. The purpose of masking is to prevent the prediction position from being affected by the subsequent sequence, causing autoregression.

[0111] Computation is performed at each level of the decoder stack:

[0112] 1) Computation of multi-head sparse attention mechanisms for occlusion;

[0113] 2) Calculate residual connectivity and perform layer normalization;

[0114] 3) Train the feedforward network;

[0115] 4) Recalculate the residual connections again and perform layer normalization.

[0116] Step 5: Perform a denormalization operation on the initial prediction result y' to obtain the final prediction output y.

[0117] The method for calculating the predicted output y is as follows:

[0118]

[0119] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. All content that does not depart from the technical solution of the present invention should be included within the protection scope of the present invention.

Claims

1. A spectrum prediction model based on Informer, characterized in that, The system includes a high-dynamic system spectrum time series processing module, an Informer model, and a frequency domain attention calculation module. The Informer model is connected to both the high-dynamic system spectrum time series processing module and the frequency domain attention calculation module. The normalization unit of the high-dynamic system spectrum time series processing module is connected to the encoder of the Informer model. The encoder of the Informer model is connected to the frequency domain attention calculation module. The frequency domain attention calculation module is connected to the decoder of the Informer model. The decoder of the Informer model is connected to the denormalization unit of the high-dynamic system spectrum time series processing module. The high dynamic system spectrum time series processing module is used to process the original sequence Normalization is performed using normalization units, and the mean of the original sequence is calculated. and standard deviation and the transformed sequence ; And used to obtain the mean and standard deviation Same as the original sequence Together, they are fed into a multilayer perceptron unit to obtain the non-stationary factor in the high dynamic system spectrum time series processing module. and For use in subsequent sequence recovery. , ; The encoder of the Informer model is used to receive the normalized sequence after embedding. The original feature map is calculated. ; The frequency domain attention calculation module is used to process the original feature map. Calculate one-dimensional convolution to obtain ;right conduct Transformation, ; Calculate the frequency domain attention enhancement vector: ; These are all learnable parameters obtained through training. Represents the ReLU activation function. This represents the Sigmoid activation function; the enhancement vector is calculated through one-dimensional convolution. : ;Will and By concatenating the features along the frequency dimension, the input feature map of the enhanced decoder is calculated. , ; The decoder of the Informer model is used to receive the feature map enhanced by the frequency domain attention calculation module. The initial prediction result of the output is calculated. ; The denormalization unit of the high dynamic system spectrum time series processing module is used to process the initial prediction results. ,according to Then perform denormalization to obtain the final prediction output. .

2. A spectrum prediction method based on Informer, characterized in that, Includes the following steps: Step 1: Enter the high dynamic system spectrum time series processing module. First, normalize the original sequence to obtain the mean, standard deviation, and transformed sequence. Then, calculate the non-stationary factor through a multilayer perceptron. Specifically, this includes: 1) normalizing the original sequence... Normalization is performed to calculate the mean of the original sequence. and standard deviation and the transformed sequence ; 2) The obtained mean and standard deviation Same as the original sequence Together, they are fed into a multilayer perceptron unit to obtain the non-stationary factor in the high dynamic system spectrum time series processing module. and For use in subsequent sequence recovery. , ; Step 2: Obtain the normalized sequence through embedding. ,Will The encoder, fed into the Informer model, calculates and obtains the original feature map. ; Step 3: Process the original feature map Frequency domain attention is calculated using DCT to obtain the enhancement vector r, which is then compared with the original feature map. The features are concatenated along the frequency dimension to obtain a frequency-domain attention-enhanced feature map. Specifically, this includes: 1) For the original feature map Calculate one-dimensional convolution to obtain ; 2) To conduct Transformation, ； 3) Calculate the frequency domain attention enhancement vector: in, These are all learnable parameters obtained through training. Represents the ReLU activation function. This represents the Sigmoid activation function; 4) Calculate the enhancement vector through one-dimensional convolution. : ； 5) and By concatenating the features along the frequency dimension, the input feature map of the enhanced decoder is calculated. , ; Step 4: Enhance the feature map with frequency domain attention The input is fed into the decoder of the Informer model for computation, yielding the initial prediction output. ; Step 5: Analyze the initial prediction results Perform denormalization to obtain the final prediction output. , where the predicted output The calculation method is as follows: 。 3. The spectrum prediction method based on Informer according to claim 2, characterized in that, Step 2 specifically involves: 1) Embed the normalized sequence to obtain ; 2) Embedded data The encoder fed into the Informer model first calculates the matrix required for attention in each layer of the encoder stack. The calculation is performed, and then sparse attention is calculated based on the three matrices and non-stationary factors. Important attention is selected, and the remaining query values ​​are replaced with the mean. Attention distillation is then performed. After N calculations, the original feature map P is obtained.

4. The spectrum prediction method based on Informer according to claim 3, characterized in that, If the encoder has N layers, then the following steps will be performed N times, specifically: 2.1) Calculation based on linearity property ; ， ， 2.2) Calculation : 2.2.1) From Random selection Composed of dot product pairs ; 2.2.2) Calculate the sample score: ； 2.2.3) Select Zhongyu The largest difference in mean A query, composed of ; 2.2.4) Replace the remaining query values ​​with the mean; 2.3) Utilization Calculate destable sparse attention; in, Indicates the dimension of the sequence; 2.4) Calculate residual connectivity and perform layer normalization; 2.5) Train the feedforward network; 2.6) Recalculate the residual connectivity and perform layer normalization; 2.7) Perform attention distillation. 。 5. The spectrum prediction method based on Informer according to claim 2, characterized in that, Specifically, step 4 involves processing the frequency-domain attention-enhanced feature map. The input sequence is passed to the decoder of the Informer model for computation. , Part of it is a start token. The length of the partial flag prediction result is output through the decoder and fully connected layer. The positions where the values ​​were originally set to 0 are the initial prediction results output. The decoder is structured as a masked multi-head sparse attention mechanism and a multi-head attention mechanism.

6. The spectrum prediction method based on Informer according to claim 5, characterized in that, Computation is performed at each level of the decoder stack: 1) Computation of multi-head sparse attention mechanisms for occlusion; 2) Calculate residual connectivity and perform layer normalization; 3) Train the feedforward network; 4) Recalculate the residual connections again and perform layer normalization.

Images