An intelligent communication spectrum planning system and method based on online deep reinforcement learning

Abstract

The application discloses an intelligent communication spectrum planning system and method based on online deep reinforcement learning, and relates to the technical field of artificial intelligence.The system comprises a perception module, an intelligent communication module and a sending module.The perception module is used for collecting electromagnetic signals in an environment as spectrum data and providing the spectrum data to the intelligent communication module.The intelligent communication module is used for intelligently analyzing and deciding electromagnetic spectrum situations, dynamically adjusting signal parameters and optimizing communication quality according to electromagnetic spectrum information and communication quality requirements in a real-time changing environment.The sending module is used for performing air transmission according to a spectrum planning result of the intelligent communication module.The application realizes efficient and real-time operation of the whole system on an edge device, and realizes real-time training and reasoning of a deep reinforcement learning model on the edge device with limited computing power through multi-link performance improvement design.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to an online deep reinforcement learning-based intelligent communication spectrum planning system and method. Background Technology

[0002] With the rapid development of communication technologies such as 5G, IoT, and cognitive radio, the use of spectrum resources in communication networks is becoming increasingly complex, and communication interference and anti-interference scenarios are becoming more and more common. This makes intelligent communication systems an important component of future communication networks. Intelligent communication not only requires efficient and reliable communication services, but also needs to address complex spectrum resource management, dynamic network environments, and diverse quality of service requirements, supporting efficient spectrum resource utilization and communication performance optimization under various complex communication interference scenarios. However, traditional communication spectrum planning methods rely on fixed preset rules or static allocation strategies, making it difficult to cope with rapidly changing electromagnetic communication environments and the anti-interference requirements under sudden, multi-type interference signals.

[0003] In recent years, deep reinforcement learning (DRL) models, such as deep Q-value networks (DQN), which combine deep neural networks (DNN) and reinforcement learning (RL), have shown great potential in communication network spectrum planning due to their ability to learn optimal strategies through continuous interaction with the environment.

[0004] However, existing AI-powered intelligent frequency selection algorithms generally suffer from three shortcomings, specifically:

[0005] First, most of them use offline training methods, and the data from offline training may not be able to fully cover all possible network states.

[0006] Secondly, due to the dynamic nature of the network environment, models trained offline may perform poorly in actual operation and have insufficient generalization ability.

[0007] Third, the ability to integrate real-time perceived spectrum status information with historical data to optimize spectrum allocation strategies is still insufficient. In particular, when multiple interference patterns exist or suddenly appear in the environment, it is difficult to adapt and adjust in a short period of time and to infer and calculate the optimal communication frequency selection strategy. Summary of the Invention

[0008] To address the above problems, this invention provides an online deep reinforcement learning-based intelligent communication spectrum planning system and method that can sense and adapt to the ever-changing electromagnetic spectrum state in real time, optimize spectrum resource allocation through continuous learning and strategy adjustment, and improve spectrum utilization and communication quality.

[0009] The technical solution of this invention is: an online deep reinforcement learning-based intelligent communication spectrum planning system, comprising:

[0010] The sensing module is responsible for collecting electromagnetic signals in the environment as spectrum data and providing them to the smart communication module.

[0011] The intelligent communication module is responsible for intelligent analysis and decision-making of the electromagnetic spectrum situation. It dynamically adjusts signal parameters and optimizes communication quality based on the real-time changing electromagnetic spectrum information and communication quality requirements in the environment.

[0012] The transmitting module is responsible for transmitting data over the air based on the spectrum planning results from the intelligent communication module.

[0013] The intelligent communication module includes establishing a dual-depth Q-value network model, which continuously interacts with the environment in the spectrum environment through online-policy or offline-policy, mapping the input spectrum signal data to the correct frequency point selection, and learning the optimal spectrum allocation strategy.

[0014] The dual-depth Q-value network model includes:

[0015] Define the spectrum environment state space:

[0016]

[0017] In the formula, Indicates frequency point, Indicates frequency point The field strength at a given point, where N represents the number of frequency points, n∈N, and t represents a certain time. Suppose there are M time points within one period, and each time point has a spectral environment state. ;

[0018] Define the action:

[0019] Action space A and specific actions for:

[0020]

[0021]

[0022] Define the reward mechanism and objective function:

[0023] If the agent predicts at time t that a frequency band will be unoccupied or have low interference at time t+1, it will be given a positive reward. Choosing a frequency band that is already occupied, causing interference or performance degradation, will result in a negative reward. Specifically:

[0024]

[0025] In the formula, The environmental reward is obtained at time t+1 after the model infers and predicts the action at time t.

[0026] The objective function of the policy network model is defined as maximizing the cumulative reward, expressed as:

[0027]

[0028] In the formula, These are the parameters of the policy network model. It is a strategy distribution.

[0029] The dual-depth Q-value network model also includes:

[0030] Policy network model output:

[0031] Given the current spectrum environment state Output motion space Each action Value, and maximum The value represents the selection action Represented as:

[0032]

[0033]

[0034] Target network model output:

[0035] The maximum value used to estimate future states The weights of the target network model are... ;

[0036] Target value The calculation is as follows:

[0037] (1)

[0038] In the formula, For instant rewards, As a discount factor, To determine the target network model for the spectral state at time t+1 The largest of all motion spaces value;

[0039] Loss function:

[0040] The loss function measures the difference between the current Q value and the target value, and uses the mean squared error as the loss metric.

[0041] (2)

[0042] In the formula, For the target value, For the policy network model, the state and actions of Value estimation.

[0043] The transmitting module includes a communication controller and a communication radio. The communication controller is connected to the communication radio and is responsible for interacting with the smart communication module, receiving spectrum planning results, and feeding back channel communication quality to the smart communication module.

[0044] A smart communication spectrum planning method based on online deep reinforcement learning includes:

[0045] Step 1: The system inputs and reads pre-collected offline spectrum status data, or collects online spectrum status data in real time through a spectrum analyzer, thus obtaining the spectrum environment status. ;

[0046] Step 2: Design a sliding window LRU and perform statistical analysis of the probability of frequency point usage in the spectrum;

[0047] The sliding window LRU covers the entire spectrum action space, and each bucket stores the statistical number of times the corresponding frequency point has not been used or interfered with;

[0048] When the spectrum state is received at time t At that time, based on the interference threshold, The frequency points used in the one-hot encoded array are first cached, and then the corresponding data in the bucket are incremented using the one-hot encoded frequency point data to achieve real-time incremental updates.

[0049] If the amount of cached data exceeds the window_size threshold, remove the earliest one-hot encoded data from the cache.

[0050] Step 3: The agent passes through Explore strategies for frequency selection decisions. The exploration strategy uses an initial probability epsilon to perform random branching. If the random probability is less than epsilon, a priority probability frequency decision is made based on a sliding window LRU. Otherwise, a policy network model is applied to make an inference frequency decision to obtain the action frequency.

[0051] Step 4: Application Explore the action frequency points of strategic decision-making, conduct relevant business process communication, and generate empirical data samples for state decision-making;

[0052] Step 5: After the communication decision action is executed, at time t+1, based on the communication quality feedback from the communication radio and the status data collected by the spectrum analyzer... The system detects whether the frequency selection action is being interfered with; if not, it receives positive reinforcement. Otherwise, one will receive a negative incentive. ;

[0053] Then, according to formula (1), the target value is calculated using the target network model in the agent, and then the inference Q value obtained by the policy network model in step 3 is used to calculate the loss function using formula (2). This loss is approximately represented as TD-error.

[0054] Finally, at time t+1, we can obtain empirical data samples of the decision made at time t. action, reward, loss );

[0055] Spectrum decision-making experience sample data ( action, reward, loss Added to the system's TD-error ring-priority empirical replay buffer pool;

[0056] The interference avoidance rate of the agent's frequency selection decision within the episode is evaluated. If it exceeds the interference avoidance rate threshold, it means that the agent can make frequency selection decisions in response to the current spectrum situation. Otherwise, the agent learns the spectrum environment, optimizes the network model parameters, and adapts to the current spectrum situation.

[0057] Step 6: When the interference avoidance rate of the agent's frequency selection decision is lower than the threshold during the episode, the agent samples data from the TD-error ring priority experience replay buffer based on weights each time during the learning process, and obtains batch_size pieces of experience data each time.

[0058] Step 7: Based on the acquired batch_size empirical data, the agent calculates the actual inference value through the policy network model, calculates the expected target value through the target network model, and then calculates the loss through the MSE loss function; the agent fixes the parameters of the target network model, performs gradient calculation and gradient backpropagation on the policy network model, and realizes the parameter update of the policy network model.

[0059] Step 3 includes:

[0060] Step 3.1: Priority probability frequency point decision based on sliding window LRU: Assign corresponding sampling probabilities according to the idle statistics in each bucket, so that the sampling will be randomly selected from the relatively idle buckets first. The bucket that is finally randomly sampled is the frequency point action.

[0061] Step 3.2: Applying the policy network model for inference frequency decision-making: Input to the policy network model The inference Q-value is obtained, and the output is the Q-value of each frequency point action in the action space. The index of the maximum Q-value is calculated by the argmax operator, which is the frequency point action.

[0062] In step 4, the system supports a hybrid online-policy and offline-policy strategy mode. In the online-policy mode, the system directly applies the frequency action output in the previous step. In the offline-policy mode, when applying the frequency, the system selects the frequency action based on the frequency avoidance rate within the episode period between the sliding window LRU and the policy network model.

[0063] In step 5,

[0064] The agent caches empirical data of size `buffer_size` in a ring-priority empirical replay buffer pool, and adjusts the weights of the loss using alpha. This serves as the sampling probability for the empirical data in this article;

[0065] When alpha=1, the sampling probability is the size of the loss.

[0066] When alpha=0, all sample data have equal weights, resulting in perfect average sampling.

[0067] When the amount of experience sample data exceeds the buffer_size, the oldest experience data is covered by the ring index, so that the ring cache only retains and focuses on the state decision experience within the recent time window.

[0068] Step 7 includes:

[0069] Step 7.1: Whenever the number of training steps of the policy network model exceeds the set threshold update_step, synchronize the policy network model parameters to the target network model;

[0070] Step 7.2: When the interference avoidance rate of the agent's frequency selection decision exceeds the avoidance rate threshold in the episode, the agent's learning and training are paused, and only the policy network model is used for inference frequency selection decision.

[0071] The present invention has the following effects:

[0072] 1. By combining anti-interference communication service scenarios, a dual-depth Q-value network model is designed for online deep reinforcement learning and inference decision-making. By designing a sliding window LRU, the problem of low sample quality and performance of exploration strategies in the deep reinforcement learning process is solved.

[0073] The design is based on a TD-error ring-priority experience replay buffer pool, which realizes that the ring cache only retains and focuses on the state decision experience within the recent time window, supports high-performance and high-quality experience sample probability sampling, and improves the model training speed and correct convergence ability.

[0074] 2. Effectively combine real-time perceived spectrum status with historical data to improve the model's generalization ability and adaptability. Through the combination of online-policy and offline-policy, adaptive adjustment of model hyperparameters and other technical capabilities, it can achieve a frequency selection interference avoidance rate of over 98% under different sparsity solution spaces caused by electromagnetic communication spectrum environments with different interference signal densities.

[0075] 3. Achieve efficient and real-time operation of the entire system on edge devices. Through multi-stage performance improvement design, enable real-time training and inference of deep reinforcement learning models on edge devices with limited computing power. Attached Figure Description

[0076] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the accompanying drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. In the drawings, the parts are not necessarily drawn to scale.

[0077] Figure 1 This is a system principle block diagram of the present invention.

[0078] Figure 2 This is a flowchart of the method of the present invention.

[0079] Figure 3 This is a schematic diagram of the principle of the dual-depth Q-value network model in this invention.

[0080] Figure 4 This is a schematic diagram illustrating the learning efficiency of the agent using the Sliding-Window LRU strategy in this invention.

[0081] Figure 5 This is a schematic diagram illustrating the learning efficiency of a conventional stochastic decision-making agent.

[0082] Figure 6 This is a comparative diagram of decision avoidance rates.

[0083] Figure 7 This is a diagram illustrating the training performance of the policy network model when alpha=0.6.

[0084] Figure 8 This is a diagram illustrating the training performance of the policy network model when alpha=1.

[0085] Figure 9This diagram illustrates the impact of batch size on model training performance in a spectral environment where the solution space is not sparse. Detailed Implementation

[0086] like Figure 1-3 As shown, this invention provides an online deep reinforcement learning-based intelligent communication spectrum planning system, comprising:

[0087] The sensing module is responsible for collecting electromagnetic signals in the environment as spectrum data and providing them to the smart communication module.

[0088] The intelligent communication module is responsible for intelligent analysis and decision-making of the electromagnetic spectrum situation. It dynamically adjusts signal parameters and optimizes communication quality based on the real-time changing electromagnetic spectrum information and communication quality requirements in the environment.

[0089] The transmitting module is responsible for transmitting data over the air based on the spectrum planning results from the intelligent communication module.

[0090] Specifically, the perception module is responsible for collecting electromagnetic signals in the environment, supporting real-time signal data acquisition over a wide frequency band, generating signal strength information at each frequency point under a specific resolution configuration, evaluating the signal-to-noise ratio, and providing it as spectrum data input to the intelligent communication module for online training and inference analysis decision-making. The reconnaissance module includes a spectrum analyzer and a reconnaissance receiver.

[0091] The intelligent communication module is responsible for intelligent analysis and decision-making regarding the electromagnetic spectrum situation. Based on real-time changes in the electromagnetic spectrum information and communication quality requirements in the environment, it dynamically adjusts signal parameters to optimize communication quality. Through AI algorithms, it automatically identifies signal types, enabling the perception and understanding of interference signals. It also supports adaptive anti-interference dynamic frequency selection, automatically choosing the optimal communication frequency band. Specifically, this includes intelligent frequency point selection, spectrum detection, interference signal perception, and frequency-domain-time domain conversion. In application, an embedded edge AI chip is used.

[0092] The transmitting module is responsible for converting the processed digital signal into an radio frequency signal for over-the-air transmission, based on the spectrum planning results from the intelligent communication module and considering different communication scenarios, using different modulation methods, and setting the transmission power and frequency. The communication controller is specifically responsible for interacting with the intelligent communication module, receiving the spectrum planning results, and providing feedback on channel communication quality and other information to the intelligent communication module to facilitate subsequent spectrum planning decisions.

[0093] The intelligent communication module includes establishing a dual-depth Q-value network model, which continuously interacts with the environment in the spectrum environment through online-policy or offline-policy, mapping the input spectrum signal data to the correct frequency point selection, and learning the optimal spectrum allocation strategy.

[0094] Specifically, the spectrum data and signal interference model include:

[0095] The original spectrum signal data format is as follows:

[0096]

[0097] In the formula, The above formula represents 1000 frequency points from 0 to 1GHz, assuming the resolution is set to 1MHz and N=1000. Indicates frequency point The electric field strength at that location is expressed in dBm.

[0098] The interference of spectral signals can be expressed by the following formula:

[0099]

[0100]

[0101] In the formula, This indicates whether there is interference at this frequency point; 0 indicates no interference, and 1 indicates interference. The threshold value representing the signal interference field strength can often be set as the background noise level, measured in dBm. If the field strength of a selected frequency is greater than or equal to... If the frequency is not specified, it means that there is interference on that frequency. Using this frequency will reduce the quality of communication or even prevent normal communication.

[0102] In the field of communication interference, common signal interference patterns include single-tone interference, multi-tone interference, frequency sweeping interference, frequency hopping interference, jamming interference, etc. In specific scenarios, there are also multiple types and quantities of mixed interference signal sources, making the spectrum situation more complex and further increasing the necessity and difficulty of implementing intelligent spectrum planning.

[0103] The spectrum planning objective is modeled as follows:

[0104] The goal of spectrum planning is to rationally allocate spectrum resources in wireless communication systems, avoid signal interference, and improve communication quality. A dual-depth Q-value network model is established and continuously interacts with the spectrum environment through online or offline policies. This model maps input spectrum signal data to the correct frequency selection, learning the optimal spectrum allocation strategy. This enables the agent to efficiently and accurately select target frequencies in complex spectrum environments, maximizing the probability of correct frequency selection and thus maximizing the cumulative reward per episode.

[0105] Define the spectrum environment state space:

[0106]

[0107] In the formula, t represents a certain moment. Assume a time period is episode, which contains M moments, and each moment has a certain spectral environment state. ;

[0108] Define the action:

[0109] An agent's action is defined as selecting a specific frequency or adjusting the frequency used to avoid interference or optimize spectrum utilization. The action space is defined as the selection of all possible frequency points. Assuming there are a total of N available frequency points, the action space A and the specific actions... for:

[0110]

[0111]

[0112] The dual-depth Q-value network model needs to be considered in relation to the environmental state at time t. To infer and predict the next action This makes the environment state at time t+1 There will be no interference.

[0113] Define the reward mechanism and objective function:

[0114] Design a reward mechanism to evaluate the quality of the agent's reasoning and prediction actions. At time t, if the agent predicts that a frequency band will not be occupied or has low interference at time t+1, it will receive a positive reward. Choosing a frequency band that is already occupied, causing interference or performance degradation, will result in a negative reward. .

[0115]

[0116] In the formula, The environmental reward is obtained at time t+1 after the model infers and predicts the action at time t.

[0117] The objective function of the policy network model is defined as maximizing the cumulative reward. Assuming the agent completes the task in M ​​time steps, the objective function can be expressed as follows:

[0118]

[0119] In the formula, These are the model parameters of the policy network model. It is a strategy distribution.

[0120] This invention implements a dual-depth Q-value network model (DDQN) with several improvements. The model's input environment state space adapts to the actual state space and action space. The model's loss function is designed to minimize the difference between the Q-value estimate of the current state-action pair and the target value. The target value considers both immediate reward and maximizing future states. The value is calculated, and the policy network model and the target network model are decoupled to enhance the stability of the policy network model during the learning process.

[0121] Policy network model output:

[0122] Given the current spectrum environment state The action space output by the model network Each action Value, and maximum The value represents the selection action Represented as:

[0123]

[0124]

[0125] In the formula, These are the parameters of the policy network model.

[0126] Target network model output:

[0127] The maximum value used to estimate future states The weights of the target network model are... ;

[0128] Target value calculation:

[0129] Target value Defined as:

[0130] (1)

[0131] In the formula, For instant rewards, This serves as a discount factor, balancing the importance of immediate and future rewards. To determine the target network model for the spectral state at time step t+1 The largest of all motion spaces value

[0132] Loss function:

[0133] The loss function measures the difference between the current Q value and the target value, and uses the mean squared error (MSE) as the loss metric.

[0134] (2)

[0135] In the formula, For the target value, For the policy network model, the state and actions of Value estimation. By continuously learning from environmental data, the error is calculated based on the loss function and backpropagated to adjust the policy model parameters. This minimizes the loss function, thus bringing the result closer to the target value. .

[0136] The target network model is used to maintain The stability of value estimation prevents the updating of policy network model parameters. This can cause drastic changes in the target value. Therefore, the parameters of the target network model... Only after each fixed number of training steps is the policy network... By replicating the original, we can eventually achieve a gradual approximation of the real thing. value.

[0137] An online deep reinforcement learning-based intelligent communication spectrum planning method trains the model online using either pre-collected offline data or real-time online data acquired via a spectrum analyzer. If pre-collected offline data is available, a pre-trained model can be built, accelerating adaptive training and adaptation to the local environment during field deployment. Even without pre-collected offline data, training and adaptive frequency selection can be performed using real-time online data.

[0138] The specific implementation steps are as follows:

[0139] Step 1: The system inputs and reads pre-collected offline spectrum status data, or online spectrum status data collected in real time by a spectrum analyzer, thus obtaining the spectrum status. .

[0140] Step 2: The system of this invention constructs and maintains a sliding-window LRU (Least Recently Used) to perform statistical analysis of the probability of spectrum frequency point usage. The LRU buckets cover the entire spectrum action space, and each bucket stores the statistical number of times a corresponding frequency point has not been used or interfered with. When the spectrum status is received at time t... At that time, based on the interference threshold, The frequency usage data is converted into a one-hot encoded array. First, this one-hot encoded data is cached. Then, using this one-hot encoded frequency data, the corresponding bucket's statistical data is incremented to achieve high-performance, real-time incremental updates. If the cached data exceeds the `window_size` threshold, the oldest one-hot encoded data in the cache is removed, thus focusing on the frequency usage statistics within the most recent time window. Using a Sliding-Window LRU (Least Recently Used) bucket can significantly improve the agent's sample learning efficiency and accuracy. Figure 4-5 As shown, the horizontal axis represents the number of statistical rounds (each statistical round represents 100 steps, each episode has 1000 steps, and dividing the x-axis by 10 gives the number of episodes), and the vertical axis represents the avoidance rate within that statistical round (0.0~1.0). The yellow curve represents the decision avoidance rate curve of the policy network model. In the same environment, the Sliding-Window LRU strategy of this invention improves the frequency-selective avoidance rate from 90% to 99% compared to random decision-making, and reduces the number of episodes required to learn a higher frequency-selective avoidance rate from 43 episodes to 10 episodes.

[0141] Step 3: The agent passes through Explore strategies for frequency selection decisions. The exploration strategy uses an initial probability epsilon (less than 1, e.g., 0.9) to randomly branch. If the random probability is less than epsilon, a priority probability frequency decision is made based on a sliding window LRU. Otherwise, a policy network model is applied to make an inference frequency decision to obtain the action frequency. After each run of the exploration strategy, epsilon decays according to epsilon_decay (e.g., 0.995), eventually decaying until it reaches epsilon_min (e.g., 0.001). This allows the agent to maintain a certain level of exploration capability even when trained to a high avoidance rate, thus enhancing the agent's environmental adaptability.

[0142] Step 3.1: Priority probability frequency point decision based on sliding window LRU: Assign a corresponding sampling probability according to the idle statistics in each bucket, so that the sampling is more likely to randomly select a relatively idle bucket, and less idle buckets can also be randomly sampled with a lower probability. The bucket that is finally randomly sampled is the frequency point action, so as to achieve the diversity of decision sample data while ensuring the effectiveness of frequency selection decision.

[0143] Step 3.2: Applying the policy network model for inference frequency decision-making: Input to the policy network model The inference Q-value is obtained, and the output of the model is the Q-value of each frequency point action in the action space. The index of the maximum Q-value is calculated by the argmax operator, which is the frequency point action.

[0144] Step 4: Application of the System of the Invention This invention explores the action frequencies for policy decisions, conducts relevant business process communication, and generates state decision-making experience data samples. The system supports a hybrid online-policy and offline-policy mode. In online-policy mode, the system directly applies the frequency action output from the previous step. In offline-policy mode, the system can only make decisions, simulate applications, and learn asynchronously in the background. When actually applying frequencies, the system can choose the decision frequency action based on whether the Sliding-Window LRU or the policy network model has better performance in frequency selection and avoidance rate within the episode period.

[0145] In the hybrid mode, the agent policy network model of this invention has a lower avoidance rate than Sliding-Window LRU in the early stages of reinforcement learning. At this point, the latter's frequency selection decision can be applied. When the agent policy network model exceeds the avoidance rate level of the latter, the frequency action of the policy network model is adopted. This results in a higher overall avoidance rate. Figure 6 As shown, the horizontal axis represents the number of episodes for model training and testing, and the vertical axis represents the number of successful avoidances in 1000 decisions within each episode. The purple curve represents the Sliding-Window LRU decision avoidance rate (85%), the yellow curve represents the policy network decision avoidance rate (from 66.6% to 98.8%), and the white curve represents the random decision avoidance rate of the control group (12%). It can be seen that in the initial stage (episodes 0-16), before adaptation and complete training, the Sliding-Window LRU decision avoidance rate is higher than the policy network model and significantly higher than the random decision. Therefore, in the hybrid mode, using Sliding-Window LRU decision-making in the initial stage (episodes 0-16) and using the decision network model in the later stages (after episode 17) can achieve a better overall avoidance rate.

[0146] Step 5: After the communication decision action in the previous step is executed, at time t+1, based on the radio communication quality feedback and the status data collected by the spectrum analyzer... It can detect whether the frequency selection action is being interfered with. If it is not interfered with, it receives positive reinforcement. Otherwise, one will receive a negative incentive. Then, according to formula (1), the target value is calculated using the target network model in the agent, and then compared with the inference obtained in step 3.2. The value is calculated using formula (2) to obtain the loss function. This loss can be approximately represented as TD-error (Temporal Difference-Error), which is used to measure the target. The error between the estimated Q value and the actual Q value. Finally, at time t+1, we can obtain the empirical data sample for the decision at time t ( action, reward, loss );

[0147] Spectrum decision-making experience sample data ( action, reward, loss This is added to the PER-ring cache (Prioritized experience replay) of the system based on TD-error ring priority. The agent caches experience data of size buffer_size in the PER-ring cache and adjusts the weight of the loss using alpha. This serves as the sampling probability for this empirical data. When alpha=1, the sampling probability is entirely based on the magnitude of the loss; samples with larger losses have higher sampling probabilities. When alpha=0, all sample data have equal weights, resulting in completely average sampling. The system design of this invention adaptively adjusts alpha based on the sparsity of the solution space, thereby controlling the impact of the loss on the sampling probability. When the empirical sample data exceeds the buffer_size, the oldest empirical data is covered using a ring index, thus achieving high performance in ring caching by retaining and focusing only on state decision-making experience within the recent time window. Figure 7-8 As shown, the horizontal axis represents the number of episodes for model training and testing, and the vertical axis represents the number of successful avoidances in 1000 decisions within each episode. The yellow curve represents the decision avoidance rate of the policy network model, assuming a sparse decision solution space in the communication environment. When alpha = 0.6, the model's optimal avoidance rate is 83.5%. When alpha = 1, the model's optimal avoidance rate is 97.2%. It is evident that when the solution space has different degrees of sparsity, this invention, through adaptive alpha values, can achieve better model training results, increasing the avoidance rate from 83.5% to 97.2% in the same communication environment.

[0148] The interference avoidance rate of the agent's frequency selection decision within the episode is evaluated. If it exceeds a certain interference avoidance rate threshold (such as 95%), it indicates that the current deep reinforcement learning agent can make good frequency selection decisions in the current spectrum situation environment. Otherwise, it indicates that the spectrum environment has changed, and the model still needs to further learn the spectrum environment to optimize the communication network parameters and adapt to the current spectrum situation environment.

[0149] Step 6: When the interference avoidance rate of the agent's frequency selection decision is lower than the threshold within the episode period, further learning and training of the agent is required. During the learning process, the agent samples data from the TD-error ring-priority experience replay buffer (PER-ring cache) based on weights each time, obtaining batch_size pieces of experience data each time. When the batch_size is set smaller, it is easier to find the correct convergence direction but convergence is slower; when the batch_size is set larger, convergence is faster but it is prone to local optima. In order to comprehensively optimize training efficiency and avoid the learning fluctuations caused by sample noise from affecting training efficiency and accuracy, the system of this invention designs and implements an adaptive batch_size based on the sparsity of the solution space. When the Sliding-Window LRU strategy and the conventional random strategy are at a low avoidance rate level, it indicates that there is a lot of signal interference in the electromagnetic communication spectrum environment, that is, the solution space is relatively sparse. In this case, the batch_size will be adjusted to a smaller value (e.g., 1, 2); otherwise, the batch_size will be adjusted to a larger value (e.g., 8, 16). The effect of this strategy is as follows: Figure 9 As shown: The second to last column (buffer_replay_batchsize) in the figure represents the batch_size of different specifications, and the last column (avoid_num_max) represents the number of times the policy network model correctly avoids errors in 1000 decisions. It can be seen that in this communication environment where the solution space is not sparse, an adaptive batch_size (such as when it is 2) can achieve a better avoidance rate of >99% when training the policy network model.

[0150] Step 7: Based on the batch_size empirical data obtained in the previous step, the agent calculates the actual inference value through the policy network model, calculates the expected target value through the target network model, and then calculates the loss through the MSE loss function. The agent fixes the parameters of the target network model and only performs gradient calculation and backpropagation on the policy network model to update the parameters of the policy network model.

[0151] Step 7.1: Whenever the number of training steps of the policy network model exceeds the set threshold update_step, the policy network model parameters are synchronized to the target network model. By decoupling the policy network model and the target network model, the same Q-network is avoided from being used for both action selection (maximization part, optimizing the maximum Q-value of the current state action combination) and loss calculation (minimization part, the target Q-value is oriented towards future actions), which can easily lead to a certain bias and make the target Q-value estimation inaccurate.

[0152] Step 7.2: When the interference avoidance rate of the agent's frequency selection decision exceeds the avoidance rate threshold (e.g., 95%) in the episode period, the agent's learning and training can be paused, and only the policy network model is used for inference frequency selection decision.

[0153] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A smart communication spectrum planning method based on online deep reinforcement learning, characterized in that, include: Step 1: The system inputs and reads pre-collected offline spectrum status data, or collects online spectrum status data in real time through a spectrum analyzer, thus obtaining the spectrum environment status. ; Step 2: Design a sliding window LRU and perform statistical analysis of the probability of frequency point usage in the spectrum; The sliding window LRU covers the entire spectrum action space, and each bucket stores the statistical number of times the corresponding frequency point has not been used or interfered with; When the spectrum state is received at time t At that time, based on the interference threshold, The frequency points used in the one-hot encoded array are first cached, and then the corresponding data in the bucket are incremented using the one-hot encoded frequency point data to achieve real-time incremental updates. If the amount of cached data exceeds the window_size threshold, remove the earliest one-hot encoded data from the cache. Step 3: The agent passes through Explore strategies for frequency selection decisions. The exploration strategy uses an initial probability epsilon to perform random branching. If the random probability is less than epsilon, a priority probability frequency decision is made based on a sliding window LRU. Otherwise, a policy network model is applied to make an inference frequency decision to obtain the action frequency. Step 4: Application Explore the action frequency points of strategic decision-making, conduct relevant business process communication, and generate empirical data samples for state decision-making; Step 5: After the communication decision action is executed, at time t+1, based on the communication quality feedback from the communication radio and the status data collected by the spectrum analyzer... To detect whether the frequency selection action is being interfered with; If there is no interference, positive incentives are obtained. Otherwise, one will receive a negative incentive. ; The target value is calculated using the target network model in the agent, and then the inference Q value obtained by the policy network model in step 3 is used to calculate the loss function, which is represented by TD-error. Finally, at time t+1, we can obtain empirical data samples of the decision made at time t. action, reward, loss ); Spectrum decision-making experience sample data ( action, reward, loss Added to the system's TD-error ring-priority empirical replay buffer pool; The interference avoidance rate of the agent's frequency selection decision within the episode is evaluated. If it exceeds the interference avoidance rate threshold, it means that the agent can make frequency selection decisions in response to the current spectrum situation. Otherwise, the agent learns the spectrum environment, optimizes the network model parameters, and adapts to the current spectrum situation. Step 6: When the interference avoidance rate of the agent's frequency selection decision is lower than the threshold during the episode, the agent samples data from the TD-error ring priority experience replay buffer based on weights each time during the learning process, and obtains batch_size pieces of experience data each time. Step 7: Based on the acquired batch_size empirical data, the agent calculates the actual inference value through the policy network model, calculates the expected target value through the target network model, and then calculates the loss through the MSE loss function; the agent fixes the parameters of the target network model, performs gradient calculation and gradient backpropagation on the policy network model, and realizes the update of the policy network model parameters. Define the spectrum environment state space: In the formula, Indicates frequency point, Indicates frequency point The field strength at a given point, where N represents the number of frequency points, n∈N, and t represents a certain time. Suppose there are M time points within one period, and each time point has a spectral environment state. ; Define the action: Action space A and specific actions for: Define the reward mechanism and objective function: If the agent predicts at time t that a frequency band will be unoccupied or have low interference at time t+1, it will be given a positive reward. Choosing a frequency band that is already occupied, causing interference or performance degradation, will result in a negative reward. Specifically: In the formula, The environmental reward is obtained at time t+1 after the model infers and predicts the action at time t. The objective function of the policy network model is defined as maximizing the cumulative reward, expressed as: In the formula, These are the parameters of the policy network model. It is a strategy distribution; Policy network model output: Given the current spectrum environment state Output motion space Each action Value, and maximum The value represents the selection action Represented as: Target network model output: The maximum value used to estimate future states The weights of the target network model are... ; Target value The calculation is as follows: （1） In the formula, For instant rewards, As a discount factor, To determine the target network model for the spectral state at time t+1 The largest of all motion spaces value; Loss function: The loss function measures the difference between the current Q value and the target value, and uses the mean squared error as the loss metric. （2） In the formula, For the target value, For the policy network model, the state and actions of Value estimation.

2. The online deep reinforcement learning-based intelligent communication spectrum planning method according to claim 1, characterized in that, Step 3 includes: Step 3.1: Priority probability frequency point decision based on sliding window LRU: Assign corresponding sampling probabilities according to the idle statistics in each bucket, so that the sampling will be randomly selected from the relatively idle buckets first. The bucket that is finally randomly sampled is the frequency point action. Step 3.2: Applying the policy network model for inference frequency decision-making: Input to the policy network model The inference Q-value is obtained, and the output is the Q-value of each frequency point action in the action space. Then, the index of the maximum Q-value is calculated by the argmax operator, which is the frequency point action.

3. The online deep reinforcement learning-based intelligent communication spectrum planning method according to claim 1, characterized in that, In step 4, the system supports a hybrid online-policy and offline-policy strategy mode. In the online-policy mode, the system directly applies the frequency action output in the previous step. In the offline-policy mode, when applying the frequency, the system selects the frequency action based on the frequency avoidance rate within the episode period between the sliding window LRU and the policy network model.

4. The online deep reinforcement learning-based intelligent communication spectrum planning method according to claim 1, characterized in that, In step 5, The agent caches empirical data of size `buffer_size` in a ring-priority empirical replay buffer pool, and adjusts the weights of the loss using alpha. This serves as the sampling probability for the empirical data in this article; When alpha=1, the sampling probability is the size of the loss. When alpha=0, all sample data have equal weights, resulting in perfect average sampling. When the amount of experience sample data exceeds the buffer_size, the oldest experience data is covered by the ring index, so that the ring cache only retains and focuses on the state decision experience within the recent time window.

5. The online deep reinforcement learning-based intelligent communication spectrum planning method according to claim 1, characterized in that, Step 7 includes: Step 7.1: Whenever the number of training steps of the policy network model exceeds the set threshold update_step, synchronize the policy network model parameters to the target network model; Step 7.2: When the interference avoidance rate of the agent's frequency selection decision exceeds the avoidance rate threshold in the episode, the agent's learning and training are paused, and only the policy network model is used for inference frequency selection decision.

6. An online deep reinforcement learning-based intelligent communication spectrum planning system, characterized in that, A smart communication spectrum planning method applied to any one of claims 1-5 using online deep reinforcement learning includes: The sensing module is responsible for collecting electromagnetic signals in the environment as spectrum data and providing them to the smart communication module. The intelligent communication module is responsible for intelligent analysis and decision-making of the electromagnetic spectrum situation. It dynamically adjusts signal parameters and optimizes communication quality based on the real-time changing electromagnetic spectrum information and communication quality requirements in the environment. The transmitting module is responsible for transmitting data over the air based on the spectrum planning results from the intelligent communication module.

7. The intelligent communication spectrum planning system based on online deep reinforcement learning according to claim 6, characterized in that, The intelligent communication module includes establishing a dual-depth Q-value network model, which continuously interacts with the environment in the spectrum environment through online-policy or offline-policy, mapping the input spectrum signal data to the correct frequency point selection, and learning the optimal spectrum allocation strategy.

8. The intelligent communication spectrum planning system based on online deep reinforcement learning according to claim 6, characterized in that, The transmitting module includes a communication controller and a communication radio. The communication controller is connected to the communication radio and is responsible for interacting with the smart communication module, receiving spectrum planning results, and feeding back channel communication quality to the smart communication module.

Images