Text enhanced multi-mode spectrum occupancy prediction system and method
By combining time-frequency graphs and textual information, the multimodal spectrum occupancy prediction system solves the problem of insufficient single-modal prediction accuracy in existing technologies, and achieves more efficient and robust spectrum occupancy prediction.
Patent Information
- Application Number
- CN202511316843.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-09-16
- Publication Date
- 2026-02-06
AI Technical Summary
Existing spectrum occupancy prediction methods rely too heavily on single-mode data, resulting in insufficient prediction accuracy in highly dynamic and complex environments, and lacking a deep understanding of spectrum usage patterns.
A text-enhanced multimodal spectrum occupancy prediction system is adopted. Features are extracted through a time-frequency graph encoder and a text encoder. A hierarchical fusion strategy is used to combine text features and time-frequency graph features, and a lightweight convolutional network is used for prediction.
It significantly improves the accuracy and robustness of spectrum occupancy prediction, enhances the prediction system's ability to understand spectrum usage patterns, reduces the impact of noise interference, and improves computational efficiency.
Smart Images
Figure CN121486834A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to a kind of multi-modal spectrum occupation prediction system and method based on text enhancement, belong to the technical field of spectrum resource management. BACKGROUND
[0002] With the rapid development of wireless communication technology, spectrum resources are increasingly strained, which has become one of the key bottlenecks restricting the rapid development of modern communication systems. In order to cope with the problem of increasingly scarce spectrum resources, dynamic spectrum access (DSA) technology has emerged and is widely recognized as an important means to improve spectrum utilization. DSA technology realizes the dynamic sharing of spectrum between primary users (PUs) and secondary users (SUs), greatly improving the utilization efficiency of spectrum resources and providing strong support for various wireless communication applications.
[0003] As an important supporting technology of dynamic spectrum access system, spectrum prediction technology effectively predicts the occupation state of spectrum in a certain future time by in-depth analysis and learning of historical spectrum occupation data. This prediction capability enables secondary users to identify potential available frequency bands in advance when making spectrum selection and access decisions, avoiding unnecessary interference with primary users during transmission, while significantly reducing the risk of delay and network congestion in the communication link.
[0004] Currently, significant progress has been made in the study of spectrum occupation prediction, and various mainstream methods have been derived. Through extensive exploration and practice, the existing spectrum occupation prediction methods can be mainly divided into three categories: spectrum occupation prediction methods based on traditional autoregressive (AR) and moving average (MA) models, spectrum occupation prediction methods based on recurrent neural networks (RNN), and spectrum occupation prediction methods based on the Transformer architecture.
[0005] wherein the prediction method based on the autoregressive moving average (ARMA) model combined with a low-pass filter models the stationary characteristics of channel measurement data to achieve preliminary prediction of spectrum occupancy. This method first extracts the average received power and idle state probability from large-scale measurement data, uses the ARMA model to fit and filter the time series characteristics of the data, and determines the occupancy of the wireless channel in different time periods, thus achieving end-to-end modeling of the prediction process and reducing the training complexity of the model. For example, H. Mosavat-Jahromi, Y. Li, L. Cai, J. Pan, et al. published a paper entitled "Prediction and Modeling of Spectrum Occupancy for Dynamic Spectrum Access Systems". This method analyzes the statistical characteristics of historical spectrum data without relying on explicit compression or error correction coding, but directly maps the measurement data to the predicted value of spectrum occupancy, and then uses a feedback mechanism to jointly optimize the model parameters. However, in this article, the optimization objective of the ARMA model is limited to the mean square error loss function for numerical recovery accuracy, and the prediction stability and applicability in highly dynamic wireless spectrum and unbalanced duty cycle are not fully considered. In the high dynamic scenario, this method still has the problems of prediction bias and data distortion.
[0006] The spectrum occupancy prediction method based on recurrent neural network utilizes the excellent time series modeling capability of recurrent neural network (RNN) to capture the time dependence in the data, and realizes accurate prediction of future spectrum occupancy. K. Umebayashi, Y. Kasahara, H. Iwata, A. Al-Tahmeesschi, et al. published a paper entitled "Spectrum Occupancy Prediction based on adaptive Recurrent Neural Networks". This paper studies the wireless local area network spectrum occupancy prediction task based on recurrent neural network (RNN), and proposes a new duty cycle (DC) prediction paradigm. This paradigm addresses the spectrum sensing problem in dynamic spectrum sharing and cognitive radio networks by taking the DC time series reflecting the spectrum usage trend as input, predicting the future duty cycle through RNN, and designing a prediction strategy based on the cooperative work of double RNN, respectively constructing independent RNN models for high and low duty cycles. However, it lacks an effective transition mechanism for states in the middle or with frequent changes, limiting its adaptability in complex environments for spectrum occupancy prediction. At the same time, the prediction method based on RNN has insufficient time series modeling capability for long-term prediction tasks, and the prediction accuracy decreases significantly with the prediction time.
[0007] The spectrum occupancy prediction method based on the Transformer architecture effectively captures long-range dependencies using self-attention mechanisms, efficiently models complex spectrum data, and thus improves prediction accuracy. In the paper "Spectrum Prediction With Deep 3D Pyramid Vision Transformer Learning" by G. Pan, Q. Wu, B. Zhou, J. Li, et al., the spectrum occupancy prediction task based on historical time-frequency maps is studied, and a prediction paradigm based on the Vision Transformer (ViT) module and the feature pyramid structure is proposed. This paradigm blocks and embeds the historical time-frequency map sequence, uses sliding windows to replace global attention calculations with local attention calculations, thus achieving more efficient global feature extraction, and uses feature pyramids and skip connections to achieve multi-scale feature extraction, thus improving prediction accuracy. However, this prediction paradigm only focuses on the time-frequency features and spatial features of the historical time-frequency map sequence, ignoring the semantic information inherent in the time-frequency map, which significantly reduces the prediction performance when the time-frequency map is disturbed by noise or data is missing. However, the above three spectrum occupancy prediction paradigms are all based on data-driven prediction methods, but none of them use multi-modal feature information to improve the accuracy of spectrum occupancy prediction. SUMMARY
[0008] The present application aims to address the problem of excessive reliance on single modal data-driven in the process of time series modeling and feature extraction in the existing spectrum occupancy prediction technology, and proposes a multi-modal spectrum occupancy prediction system and method based on text enhancement. The system uses a time-frequency map encoder and a text encoder at the encoding end to extract features from historical time-frequency maps, global text descriptions, and local text descriptions, respectively, and further uses a hierarchical fusion strategy to fuse the two types of text features and time-frequency map features at different stages. Through multi-modal feature interaction, the complementary information of time-frequency map and text features can be achieved, and a lightweight three-dimensional convolutional network is used at the prediction end to predict the future spectrum occupancy based on the fused features. This system combines text descriptions with time-frequency maps, significantly improving the spectrum occupancy prediction accuracy in high dynamic environments, and has good robustness.
[0009] The technical scheme adopted by the present application to solve its technical problems is: a multi-modal spectrum occupation prediction system based on text enhancement, which comprises an embedding module, an encoding module, a gated attention fusion module, a multi-head self-attention fusion module and a spectrum occupation prediction module. The embedding module uses a three-dimensional convolution kernel to segment the historical time-frequency graph, converts it into a picture block (patch) for subsequent feature extraction; the encoding module comprises two sub-modules of text encoding and time-frequency graph encoding, the text encoding uses a pre-trained model BERT-base-uncased to extract features of local text description and global text description, and the time-frequency graph encoder uses a stack of three-dimensional Swin Transformer blocks and patch merging layers to realize multi-scale feature extraction of the time-frequency graph; the gated attention fusion module fuses the local text description and the time-frequency graph block in the early stage of feature extraction, guiding the model to focus on the significant features in each spectrum graph; the multi-head self-attention fusion module fuses the global text features after the spectrum graph feature extraction, so that the model understands the global regularity of the time-frequency graph at a more abstract feature level; the spectrum occupation prediction module uses a stack of three-dimensional convolution layers to form a lightweight model, and according to the high-dimensional fusion features obtained by the encoding and hierarchical fusion module, fully utilizes the spatial, temporal and frequency information contained therein to predict the future spectrum occupation.
[0010] Further, the system of the present application combines text knowledge with time-frequency graphs, and by utilizing the rich spectrum occupation situation semantic information in the text knowledge, assists the multi-modal model in feature extraction, time series modeling and spectrum occupation situation prediction. Compared with other spectrum occupation prediction frameworks, the present application focuses on introducing text knowledge in the process of time-frequency graph feature extraction, so that the prediction system better understands the spectrum usage rules of a certain frequency band in a certain time period. And by designing a lightweight prediction model, both the spectrum occupation prediction accuracy and the calculation efficiency are taken into account.
[0011] The present application also provides an implementation method of a multi-modal spectrum occupation prediction system based on text enhancement, which comprises the following steps:
[0012] Step 1: slice and extract features of the time-frequency graph sequence and the text description respectively, and fuse the features by using a gated attention mechanism, which comprises the following steps:
[0013] Step 1-1: slice the time-frequency graph sequence, and encode the slices to obtain time-frequency graph embedding vectors X 1:L ;
[0014] Step 1-2: text knowledge feature extraction; if the text branch input of the model is a text description D, to realize the spectrum occupation prediction enhanced by the text, the features of the text D need to be extracted first Two kinds of text descriptions are defined in the present application, one is the global text description D g suitable for all time-frequency maps, and the other is the local text description D 1:L of each time-frequency map. We use the pre-trained model BERT-base-uncased with frozen parameters to extract features of the global text description and the local text description. The process is as follows:
[0015]
[0016] where E BERT (·) is the BERT encoder, and are the global text feature and the local text feature sequence respectively.
[0017] Step 1-3: Use the gated attention module to fuse the local text features obtained in the above step 1-2 with the time-frequency map embedding vector to guide the model to focus on important features of each time-frequency map. Record this step as:
[0018]
[0019] where is the feature sequence after fusion by the gated attention module, X 1:L is the input time-frequency map embedding sequence, and σ and W g are the sigmoid activation function and the gating weight respectively.
[0020] Step 2: After completing the text knowledge feature extraction in the above step 1, time-frequency map feature extraction and multi-modal feature hierarchical fusion are performed in sequence. Specifically, the following steps are included:
[0021] Step 2-1: The feature after fusion by the gated attention module is input into the time-frequency map encoder to obtain a high-dimensional feature sequence.
[0022] Step 2-2: After completing the time-frequency map feature extraction, the high-dimensional feature is fused with the global text feature by using the multi-head self-attention module, and the feature is screened by the bottleneck layer to obtain a high-dimensional fusion feature.
[0023] Step 3: Spectrum occupancy prediction. The high-dimensional fusion feature obtained in the above step 2 is input into the lightweight prediction network to generate the spectrum occupancy of future time slots. First, the spatial feature processing module is used to reduce the dimension of the high-dimensional feature and extract features, and then the time and frequency dimensions are reduced and the resolution is restored by using the time-frequency feature, and finally the reconstructed image of the future time slots is obtained.
[0024] Further, the step 2 of the application comprises: the time-frequency graph feature extraction and fusion module is composed of a time-frequency graph encoder, a multi-head self-attention fusion module and a bottleneck layer. The backbone network of the time-frequency graph encoder is a three-stage three-dimensional SwinTransformer, and the number of three-stage three-dimensional Swin Transformer blocks is 2, 4 and 2 respectively. The features fused through the gated attention module are input into the time-frequency graph encoder, and the obtained high-dimensional feature sequence is The encoding process of the time-frequency graph encoder can be represented as:
[0025]
[0026] wherein E SpecEnc represents a three-dimensional Swin Transformer architecture-based time-frequency graph encoder.
[0027] The multi-head self-attention fusion module utilizes queries, keys and values, combines the window division mechanism of the three-dimensional Swin Transformer, calculates the attention weights of the high-dimensional feature sequence and the global text description features, and captures the dependency relationship in the frequency and time dimensions. After the time-frequency graph feature extraction, the multi-head self-attention fusion module is used to fuse the high-dimensional features and the global text features, and the bottleneck layer is used for feature screening to obtain high-dimensional fusion features The bottleneck layer is composed of two consecutive three-dimensional Swin Transformers, keeps the hidden layer dimension unchanged, and enhances the feature quality after the global text fusion by using the consecutive multi-head self-attention mechanism. This process can be represented as:
[0028]
[0029] wherein f MHSA is a multi-head self-attention module, f BTN is a bottleneck layer composed of Swin Transformer blocks.
[0030] Further, the step 3 of the present application comprises: the spectrum occupation prediction process undergoes spatial feature processing, frequency dimension refining and time dimension refining in turn. The spatial feature processing module is composed of three groups of combined networks containing three-dimensional convolution, batch normalization and ReLU activation function, which performs dimension reduction processing and feature extraction on high-dimensional fusion features; the frequency dimension refining module is composed of two groups of combined networks containing three-dimensional convolution layer, batch normalization layer and ReLU activation function, and one three-dimensional convolution layer, which compresses the frequency dimension of the spatial feature, and maps the spatial feature to the target frequency resolution by using bilinear interpolation; the time dimension refining module is composed of a group of combined networks containing three-dimensional padding layer, three-dimensional convolution layer, batch normalization layer and ReLU activation function, and an independent three-dimensional convolution layer, which maps the spatial feature to the target time resolution, and linearly weights and combines adjacent time frames to weight and smooth the time boundary. Finally, the output is limited to the [0, 1] interval by the Sigmoid activation function, representing the occupation of the frequency point in the future time slot This process can be represented as:
[0031]
[0032] Where f TimeRef , f FreqRef and f SpatialProc represent the time, frequency and spatial dimension processing modules respectively, and sigma is the Sigmoid activation function.
[0033] Beneficial effects:
[0034] 1、The present application couples the time-frequency diagram with the text knowledge, and simultaneously performs spectrum occupation prediction as a data-driven multi-modal model, fully utilizes multi-modal information, and avoids the interference of periodic cross-frequency noise in the historical time-frequency diagram on feature extraction of the model. The test results show that the overall prediction accuracy of the multi-modal spectrum occupation prediction scheme proposed in the present application is improved by 3.17% compared with the optimal baseline, and the recall rate is improved by 7.09% compared with the optimal baseline.
[0035] 2、The present application designs a multi-modal feature hierarchical fusion strategy suitable for multi-modal spectrum prediction tasks, fully utilizes multi-granularity semantic information, realizes hierarchical feature enhancement, and provides richer feature expression for spectrum occupation prediction. The ablation experiment results show that the hierarchical fusion strategy proposed in the present application improves the recall rate by 13.02% compared with the simple feature fusion strategy.
[0036] 3、The present application designs a binary cross entropy loss function enhanced by a dynamic penalty factor for spectrum data with different duty cycles, which effectively reduces the false positive rate of future spectrum in the occupied state. The experimental results show that the introduction of the enhanced binary cross entropy loss function in the present application makes the F1 score improve by 5.37% compared with the optimal baseline. BRIEF DESCRIPTION OF DRAWINGS
[0037] Figure 1 System diagram of the present application.
[0038] Figure 2 Comparison chart of the prediction accuracy of spectrum occupancy within the next 20 seconds of the present application and other prior art.
[0039] wherein, Figure 2 (a) the vertical coordinate is the prediction accuracy (Accuracy) diagram, Figure 2 (b) the vertical coordinate is the recall (Recall) diagram, Figure 2 (c) the vertical coordinate is the precision (Precision) diagram, Figure 2 (d) the vertical coordinate is the F1-score (F1-score) diagram.
[0040] Figure 3 Visualization chart of the prediction results of spectrum occupancy of the present application.
[0041] wherein, Figure 3 (a) is the real-time frequency chart, Figure 3 (b) is the predicted spectrum occupancy diagram, Figure 3 (c) is the prediction result correctness diagram. DETAILED DESCRIPTION
[0042] In order to make the purpose, technical scheme and advantages of the present application more clear and obvious, the present application will be further described in detail below in combination with specific embodiments and with reference to the accompanying drawings. It should be understood that these descriptions are only exemplary and are not intended to limit the scope of the present application. In addition, in the following description, the description of well-known structures and technologies is omitted to avoid unnecessary confusion of the concept of the present application.
[0043] As Figure 1As shown, the present application provides a multi-modal spectrum occupancy prediction system based on text enhancement, which comprises an embedding module, an encoding module, a gated attention fusion module, a multi-head self-attention fusion module and a spectrum occupancy prediction module. The embedding module uses a three-dimensional convolution kernel to segment the historical time-frequency graph, converts it into a picture block for subsequent feature extraction; the encoding module includes two sub-modules of text encoding and time-frequency graph encoding, the text encoding extracts features of global text and local text to obtain semantic information in the text, and the time-frequency graph encoder extracts multi-scale features of the time-frequency graph sequence; the gated attention fusion module fuses the local text description and the time-frequency graph block in the early stage of feature extraction to capture independent features in each spectrum graph; the multi-head self-attention fusion module fuses the global text features after the spectrum graph feature extraction to capture the global regularity of the time-frequency graph sequence; the spectrum occupancy prediction module makes full use of the high-dimensional fusion features obtained by the encoding and hierarchical fusion module to predict the future spectrum occupancy.
[0044] The system of the present application combines text knowledge with time-frequency graph, and by utilizing the rich spectrum occupancy situation semantic information in the text knowledge, assists the multi-modal model in feature extraction, time series modeling and spectrum occupancy situation prediction. Compared with other spectrum occupancy prediction frameworks, the present application focuses on introducing text knowledge in the time-frequency graph feature extraction process, so that the prediction system better understands the spectrum usage rules of a certain frequency band in a certain time period. And by designing a lightweight prediction model, both the spectrum occupancy prediction accuracy and the calculation efficiency are taken into account.
[0045] The present application also provides an implementation method of a multi-modal spectrum occupancy prediction system based on text enhancement, which comprises the following steps:
[0046] Step 1, real frequency modulation (FM) dataset acquisition and multi-modal annotation expansion.
[0047] The frequency modulation (FM) spectrum data is usually expressed in the form of a single in-phase / quadrature (I / Q) signal or time-frequency graph in traditional research. Although such single-modal data can reflect the basic use of the spectrum, it has limitations in describing complex wireless environments and capturing signal semantic information. With the continuous integration of wireless communication and artificial intelligence technology, single-modal data has been difficult to meet the needs of information richness and semantic levels for high-level applications such as spectrum occupancy prediction. Therefore, the present application uses a multi-modal FM dataset to organically combine text description with time-frequency graph data, which not only can fully exploit the structural features of spectrum use, but also can improve the understanding ability of the model for dynamic changes of the spectrum.
[0048] The real spectrum data set used in the experimental verification of the present application is Real-world Spectrum, which contains 10800 frequency modulation sample sets, the sampling time is September 23, 2022, 17:20-20:20, and each sample set contains 32508 groups of I / Q signal samples. First, the I / Q signal samples per second are converted into time-frequency diagrams by using short-time Fourier transform (STFT). The window function of STFT is Hanning window, and the window length is 256. The conversion process can be represented as follows:
[0049]
[0050] Where ω(·) represents the Hanning window function. The amplitude square of each element of the time-frequency matrix is calculated to obtain 10800 RGB time-frequency diagrams with a width of 256 and a height of 256. Then, the multi-modal expansion of the time-frequency diagram data set is carried out in two steps. First, a global text description suitable for all time-frequency diagrams in the data set is introduced, which contains the use of a certain frequency band specified by the International Telecommunication Union (ITU) in the division of radio frequency, and also includes the description of the historical frequency trend of the region; the second step is to introduce the local text description corresponding to each time-frequency diagram. The text is used to describe the detailed information of each time-frequency diagram, including whether there is periodic cross-frequency noise and the specific frequency use of each frequency band.
[0051] Step 2, text encoding and gating fusion process.
[0052] In the text-enhanced multi-modal spectrum occupation prediction process, the system receives a sequence of historical time-frequency diagrams and related text descriptions. First, the collected time-frequency diagrams and text descriptions are respectively embedded and semantic feature extraction is performed, the purpose of which is to obtain the time-frequency structure information of the spectrum and the text semantic information. This process mainly relies on the cooperative work of two algorithms: on the one hand, three-dimensional convolution is used to slice the time-frequency diagram; on the other hand, the BERT-base-uncased model is used to uniformly encode the global spectrum allocation trend and the local text description of the interference phenomenon, to generate a semantic feature vector. Subsequently, in the initial stage of feature extraction, the local text description and the time-frequency diagram slice are fused to guide the model to focus on the representative key information in each time-frequency diagram. The gating attention fusion module developed by the system dynamically adjusts the weight of each modal feature by perceiving the context correlation, realizing the adaptive and fine-grained fusion of local text features and time-frequency diagram embeddings, and effectively promoting the interaction and cooperation of multi-modal information.
[0053] Step 3, time-frequency diagram encoding and multi-head self-attention fusion process.
[0054] After completing the multi-modal feature extraction of the historical time-frequency map sequence and multi-source text description, the application further designs a feature fusion and encoding process to realize efficient spectrum occupation prediction. First, the multi-modal features obtained by the gating attention fusion module are input into the stacked three-dimensional Swin Transformer encoding structure, and multi-level feature extraction is performed in the time, frequency and spatial dimensions in turn. After each encoding stage, the system will integrate and reduce the dimension of the features through the patch merging operation, thereby effectively expanding the receptive field of the model and improving the feature expression capability.
[0055] After the time-frequency map feature extraction is completed, the system fuses the global text description information with the obtained time-frequency features, enabling the model to understand the overall distribution and trend of the spectrum data at a higher level. For this purpose, a multi-head self-attention (MHSA) module is introduced in the feature fusion process, which can effectively integrate features for global text description, capture long-distance dependency and global semantic structure, and further strengthen information interaction between different modalities.
[0056] The fused high-dimensional feature set finally enters the bottleneck layer, which is composed of two three-dimensional Swin Transformer modules in series, dedicated to compressing and simplifying the fused features. Through this process, the model can convert the massive multi-modal information into compact feature representation, providing computationally efficient and semantically rich input for the subsequent spectrum occupation prediction task.
[0057] Step 4, spectrum occupation prediction process.
[0058] In the spectrum occupation prediction process of the application, the receiving end first obtains the reconstructed feature vector after channel transmission and demodulation, and makes decoding decision of the spectrum occupation state according to it. For the spatial dimension feature, the system designs a three-level three-dimensional convolution (Conv3D) module in series, which is used to extract and fuse spatial structure information layer by layer; while for the frequency and time dimension variation features, two layers of Conv3D structure are used for fine processing respectively, realizing in-depth modeling of the spectrum dynamic behavior. Finally, the system uses the sigmoid activation function to normalize all the fused features, and directly outputs the spectrum occupation prediction result.
[0059] In order to further improve the training efficiency and prediction performance of the method, the traditional binary cross entropy (BCE) loss function is optimized. First, the label smoothing mechanism is introduced in the loss calculation, which effectively alleviates the influence of prediction overfitting and label noise, and enhances the generalization ability of the model. Secondly, considering the uneven distribution of actual spectrum occupation samples, a class balance weight is designed to give reasonable importance to different class samples, thereby balancing the loss contribution. Through the above multi-level loss design, the detection ability of the model for sparse occupation signals is significantly improved. The improved loss function calculation process is as follows:
[0060]
[0061] Wherein is the BCE loss function, and y b,t,f,p respectively represent the predicted value and the true value, ω b,t,f,p and π b,t,f,p respectively represent the class weight and the false positive penalty coefficient, Ω is the tensor space, b, t, f, p are respectively four dimensions of the tensor space.
[0062] The effect of the present application will be further described below in combination with simulation experiments. Specifically, it includes:
[0063] 1) Simulation conditions and parameter settings
[0064] The simulation experiment of the present application is carried out on a simulation platform of Python3.9, PyTorch2.1.0. The computer CPU model is Intel Platinum 8352V, the number is 16, the GPU model is Nvidia Geforce RTX 4090, and the video memory is 24GB. The three-dimensional Swin Transformer block number of the time-frequency diagram encoder used in the present application is 2, 4 and 2 respectively, the attention head number is 4, 8 and 16 respectively, and the three-dimensional Swin Transformer block number of the bottleneck layer is 2. The hidden layer dimension of the BERT-base-uncased model after encoding is 128, and the hidden layer dimension of the multi-head self-attention module is 384. The learning rate is set to 0.0005, the batch size is set to 1, a total of 30 iteration rounds are trained, and the early stopping patience value is 5.
[0065] 2) Simulation content
[0066] Figure 2 is the performance comparison of the present application and other spectrum occupation prediction methods on the Real-world Spectrum multi-modal data set. The horizontal coordinate is the future timestamp (s). Figure 2 (a) The vertical coordinate is the prediction accuracy (Accuracy), Figure 2(b) the vertical coordinate is precision, Figure 2 (c) the vertical coordinate is recall, Figure 2 (d) the vertical coordinate is F1-score, which evaluates the prediction performance from the overall prediction accuracy and the sub-class prediction accuracy, and more comprehensively reflects the prediction performance. The circular broken line represents the performance of the present application, the square broken line represents the performance of the traditional 3D-SwinLinear method, the triangular broken line represents the performance of the traditional ConvLSTM method, and the diamond broken line represents the performance of the traditional NN-ResNet method. As can be seen from Figure 2 (a), the present application has maintained a leading precision performance throughout the entire prediction time domain, and in particular in the 16th second ahead of prediction, it has improved by 3.17% compared with the baseline model. Further observation of the recall index, Figure 2 (b) shows that the present application has achieved almost perfect recall performance, which is significantly better than other comparison methods, fully embodying its high detection capability for actual occupancy state and effective inhibition of missed detection phenomenon; from Figure 2 (c), it can be seen that the present application also has a strong performance advantage, significantly reduces the false positive rate, and performs more stably than the traditional model; finally, Figure 3 (d), the F1 score further confirms that the present application has achieved an optimal balance between precision and recall, and its F1 score has improved by an average of 0.0537 compared with the baseline.
[0067] Figure 3 is a visualization result graph of the present application in the spectrum occupancy prediction task. From Figure 3 (a) the target real-time frequency graph and Figure 3 (b) the model prediction result, it can be seen that the present application can accurately restore the actual time-frequency occupancy distribution. Further, (c) shows the visualization effect of prediction correctness, in which gray and green correspond to the correct judgment of the model for idle and occupied time-frequency points, and blue and red reflect the misjudgment of idle and occupied states. Comprehensive analysis shows that the present application significantly improves the detection accuracy of actual occupied time-frequency points while maintaining a low error prediction rate, and exhibits superior spectrum occupancy recognition capability.
[0068] In the art, the above-mentioned is only a preferred example of the present application, and is not used to limit the present application. Any modification, replacement, improvement and polishing made by any person skilled in the art within the spirit and concept of the present application shall be covered within the protection scope of the present application technology.
Claims
1. A method for implementing a text-enhanced multimodal spectrum occupancy prediction system, characterized in that, The method includes the following steps: Step 1: Slice and extract features from the time-frequency graph sequence and text description respectively, and use a gated attention mechanism to fuse features; Step 1-1: Slice the time-frequency graph sequence and encode the slices to obtain the time-frequency graph embedding sequence; Step 1-2: Extract features from the global and local text descriptions to obtain global and local text vectors; Steps 1-3: The time-frequency image embedding vector and its corresponding local text vector are fused using a gated attention module to obtain a gated fused feature sequence; Step 2: After completing the text knowledge feature extraction in Step 1 above, time-frequency graph feature extraction and multimodal feature hierarchical fusion are performed sequentially. Step 2-1: The features fused by the gated attention module are passed through the time-frequency map encoder to obtain a high-dimensional feature sequence; Step 2-2: Use a multi-head self-attention module to fuse the high-dimensional feature sequence with the global text features, and use a bottleneck layer to filter features to obtain a high-dimensional fused feature sequence; Step 3: The high-dimensional fusion features obtained in Step 2 are used to generate the spectrum occupancy of several future time slots through a lightweight prediction network; the model learns by minimizing the loss function, and the network parameters are updated through the inverse gradient until the algorithm converges.
2. The implementation method of a text-enhanced multimodal spectrum occupancy prediction system according to claim 1, characterized in that, Steps 1-2 include: defining a global text description applicable to the entire time-frequency graph sequence and a local text description corresponding to each time-frequency graph as follows: Global text description D g Includes the usage of a specific frequency band as defined by the International Telecommunication Union (ITU) in its radio frequency allocation, as well as a description of historical frequency usage trends in the region; partial text description D 1:L It contains detailed information for each time-frequency plot, including whether there is periodic cross-frequency noise and the specific frequency usage of each frequency band; The BERT-base-uncased pre-trained model with frozen parameters is used to extract features from both global and local text descriptions. The text encoding process can be represented as follows: Where E BERT (·) represents the BERT encoder. and These are global text feature sequences and local text feature sequences, respectively.
3. The implementation method of a text-enhanced multimodal spectrum occupancy prediction system according to claim 1, characterized in that, Steps 1-3 include: the gated attention fusion module uses the time-frequency map embedding as the query vector and the local text features as the key and value vectors, calculates the correlation between the time-frequency map embedding and the local text features, and selectively focuses on the parts related to the local text description to achieve feature enhancement under the guidance of local text description. The early feature fusion process can be represented as: in X refers to the feature sequence fused by the gating attention module. 1:L The input time-frequency graph embedding sequence, σ and W g These refer to the sigmoid activation function and the gating weights, respectively.
4. The implementation method of a text-enhanced multimodal spectrum occupancy prediction system according to claim 1, characterized in that, Step 2 includes: the time-frequency map feature extraction and fusion module consists of a time-frequency map encoder, a multi-head self-attention fusion module, and a bottleneck layer. The backbone network of the time-frequency map encoder is a three-stage 3D Swing Transformer with 2, 4, and 2 blocks in each stage. The features fused by the gated attention module are then passed through the time-frequency map encoder to obtain a high-dimensional feature sequence. The encoding process of a time-frequency encoder can be represented as follows: Where E SpecEnc This represents a 3D time-frequency encoder based on the Swing Transformer architecture. The multi-head self-attention fusion module utilizes queries, keys, and values, combined with the windowing mechanism of the 3D Swing Transformer, to calculate the attention weights between high-dimensional feature sequences and global textual description features. This captures the dependencies in the frequency and temporal dimensions. After extracting time-frequency graph features, the multi-head self-attention module fuses the high-dimensional features with the global textual features, and then performs feature filtering through a bottleneck layer to obtain the high-dimensional fused features. The bottleneck layer consists of two consecutive 3D SwinTransformers, maintaining the hidden layer dimension unchanged. It utilizes a continuous multi-head self-attention mechanism to enhance the feature quality after fusion with the global text. This process can be represented as: Where f MHSA It is a multi-head self-attention module, f BTN It is a bottleneck layer composed of Swing Transformer blocks.
5. The implementation method of a text-enhanced multimodal spectrum occupancy prediction system according to claim 1, characterized in that, Step 3 includes: the spectrum occupancy prediction process undergoes spatial feature processing, frequency dimension refinement, and time dimension refinement sequentially. The spatial feature processing module consists of three sets of combined networks containing 3D convolution, batch normalization, and ReLU activation functions. This module performs dimensionality reduction and feature extraction on high-dimensional fused features. The frequency dimension refinement module consists of two sets of combined networks containing 3D convolutional layers, batch normalization layers, and ReLU activation functions, as well as one 3D convolutional layer. This module compresses the frequency dimension of spatial features and maps the spatial features to the target frequency resolution using bilinear interpolation. The time dimension refinement module consists of a set of combined networks containing 3D padding layers, 3D convolutional layers, batch normalization layers, and ReLU activation functions, as well as one independent 3D convolutional layer. This module maps spatial features to the target time resolution and performs weighted smoothing of the time boundary by linearly weighting and combining adjacent time frames. Finally, the output is restricted to the [0,1] interval by the Sigmoid activation function, representing the occupancy status of this frequency point in future time slots. This process can be represented as: Where f TimeRef f FreqRef and f SpatialProc These represent the time, frequency, and spatial dimension processing modules, respectively, with σ being the Sigmoid activation function.
6. A text-enhanced multimodal spectrum occupancy prediction system, characterized in that, The system includes an embedding module, an encoding module, a gated attention fusion module, a multi-head self-attention fusion module, and a spectrum occupancy prediction module; The embedding module uses 3D convolutional kernels to segment the historical time-frequency map and convert it into image patches for subsequent feature extraction. The encoding module includes two sub-modules: text encoding and time-frequency map encoding. The text encoding uses the pre-trained model BERT-base-uncased to extract features from local and global text descriptions. The time-frequency map encoder uses a stack of 3D SwingTransformer blocks and Patch Merging layers to extract multi-scale features from the time-frequency map. The gated attention fusion module fuses local text descriptions and time-frequency plots in the early stages of feature extraction, guiding the model to focus on salient features in each spectrogram. The multi-head self-attention fusion module fuses global text features after spectrogram feature extraction, enabling the model to understand the global regularity of the frequency-time-frequency graph at a more abstract feature level. The spectrum occupancy prediction module uses stacked three-dimensional convolutional layers to form a lightweight model. Based on the high-dimensional fusion features obtained by the encoding and hierarchical fusion modules, it makes full use of the spatial, temporal and frequency information contained therein to predict future spectrum occupancy.
7. A text-enhanced multimodal spectrum occupancy prediction system according to claim 6, characterized in that, The system combines textual knowledge with time-frequency graphs. By utilizing the rich semantic information about spectrum occupancy in the textual knowledge, it assists multimodal models in feature extraction, time-series modeling, and spectrum occupancy prediction. Compared with other spectrum occupancy prediction frameworks, the introduction of textual knowledge in the time-frequency graph feature extraction process enables the prediction system to better understand the spectrum usage patterns of a certain frequency band within a certain time period. Furthermore, by designing a lightweight prediction model, it balances both spectrum occupancy prediction accuracy and computational efficiency.