Intelligent resource allocation method based on partially observable active inference for 6G
A partially observable deep multi-agent active reasoning framework addresses the inefficiencies of existing resource allocation methods by optimizing resource allocation through belief-based learning, achieving faster convergence and improved real-time performance in 6G networks.
Patent Information
- Application Number
- CN202311234210.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-09-22
- Publication Date
- 2024-12-31
- Estimated Expiration
- 2043-09-22
AI Technical Summary
Existing resource allocation methods for 6G wireless communication networks face challenges in handling complex dynamics, high computational complexity, and slow convergence due to reliance on full environmental observations, which are impractical and inefficient.
A partially observable deep multi-agent active reasoning framework that utilizes belief-based learning to minimize intrinsic free energy, enabling distributed training and execution, optimizing resource allocation through active inference.
The framework improves resource allocation efficiency, accelerates convergence speed, and enhances real-time performance, making it suitable for dynamic 6G communication scenarios with reduced computational overhead.
Smart Images
Figure CN117241403B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of wireless communication technology, and specifically relates to an intelligent resource allocation method based on partially observable active reasoning for 6G. Background Art
[0002] Resource allocation is an important key technology in wireless communication networks. Efficient resource allocation helps to make full use of precious and limited resources and significantly improve many communication indicators, such as spectrum efficiency and energy efficiency. However, how to design an efficient resource allocation scheme for the future sixth-generation (6G) wireless communication network is a very challenging problem. On the one hand, in 6G wireless communication networks, the joint optimization of multi-domain resources such as energy, spectrum, computing, and cache is the main trend. The high-dimensional decision variables are deeply coupled, which makes the optimization problem difficult to solve. On the other hand, with the prevalence of various smart wearable devices, in order to meet the immersive experience of 6G users, the wireless communication environment and user business needs will change dynamically over time, so a real-time resource allocation method is essential.
[0003] Existing resource allocation methods can be divided into two categories: methods based on traditional optimization theory and intelligent methods based on machine learning represented by deep reinforcement learning (DRL). Methods based on traditional optimization theory mainly use traditional mathematical techniques such as alternating optimization and successive convex approximation. In the article "Resource Allocation and Trajectory Design in UAV-Assisted Jamming Wideband Cognitive Radio Networks" (IEEE Trans. Cogn. Commun. Netw., vol. 7, no. 2, pp. 635-647, 2022) published by Yuhao Wang, Long Chen, Yifan Zhou, Fuhui Zhou, et al., in order to maximize the average confidentiality rate of orthogonal frequency division multiplexing spectrum sharing networks, the authors proposed a resource allocation algorithm based on continuous convex approximation to jointly optimize the transmission power of secondary base stations and the flight trajectory of unmanned aerial vehicles (UAVs). However, the resource allocation problem in future wireless communication networks is generally non-deterministic polynomial difficulty (NP-hard), and it is difficult to obtain the optimal solution using methods based on traditional optimization theory. In addition, the high computational complexity and computational delay cannot meet the needs of future 6G communication networks.
[0004] In recent years, DRL has been widely used in designing intelligent resource allocation schemes due to its outstanding advantages of efficient and fast processing of large-scale complex problems. The patent "Spectrum Resource Allocation Method and System Based on Collaborative Distributed DQN Combined Simulated Annealing Algorithm" with publication number CN113613332A discloses a method based on collaborative distributed deep reinforcement learning. Each agent shares the same network model and the same reward function, and uses the experience previously collected by all agents to train the same deep Q learning network (DQN), which can greatly reduce the memory and computing resources required for training. However, this method only considers a single decision variable, the communication scenario is too ideal, and ignores the heterogeneous complexity of future wireless communication networks. Wei Wu, Fengchun Yang, Fuhui Zhou, et al. considered an intelligent reflector-enhanced orthogonal frequency division multiplexing network in their published article "Intelligent Resource Allocation for IRS-Enhanced OFDM Communication Systems: A Hybrid Deep Reinforcement Learning Approach" (IEEE Trans. Wireless Commun., 2022), and designed a decision-making scheme combining DQN and deep deterministic policy gradient algorithm to achieve an increase in user and transmission rates. However, such a network structure is too complex and brings huge computational overhead. In addition, the DRL-based method relies on environmental feedback to optimize the long-term expected reward. Therefore, the DRL-based method converges slowly, which limits its practical deployment capabilities and makes it difficult to apply it to future 6G wireless communication networks.
[0005] In general, resource allocation methods based on traditional optimization theory are difficult to cope with the complex dynamics of 6G communication environments, and intelligent deep reinforcement learning methods have poor real-time performance and are difficult to deploy in practice. In addition, most existing work assumes that complete environmental observations can be obtained. However, due to the insufficient capabilities of sensing devices and limited communication and computing overhead, this is unrealistic in actual systems. Therefore, it is urgent to develop a new general framework for resource allocation suitable for 6G communication networks. Summary of the invention
[0006] The technical problem to be solved by the present invention is to provide an intelligent resource allocation method based on partially observable active reasoning for 6G in view of the deficiencies of the above-mentioned prior art. Through a new partially observable deep multi-agent active reasoning framework, the resource allocation efficiency is improved and the convergence speed is accelerated. In actual communication scenarios, it has good performance, fast convergence speed, meets real-time requirements, and is independent of the full observation state of the environment.
[0007] To achieve the above technical objectives, the technical solution adopted by the present invention is as follows:
[0008] An intelligent resource allocation method for 6G based on partially observable active inference, comprising:
[0009] Step 1: Calculate the channel power gain from the UAV to the secondary user SU and the channel power gain from the primary base station to the primary user PU;
[0010] Step 2: Calculate the transmission rates of the SU and the PU based on the channel power gains obtained in Step 1;
[0011] Step 3: Construct the input state information variables of the sub-band agent and the input state information variables of the UAV agent;
[0012] Step 4: Based on the transmission rates, construct the reward function of the sub-band agent and the reward function of the UAV agent;
[0013] Step 5: Initialize the training episode ep to 0;
[0014] Step 6: Initialize the time step t in the ep episode to 0;
[0015] Step 7: Obtain the input state according to the input state information variables constructed in Step 3, and respectively obtain the sub-band allocation action and the UAV flight direction action;
[0016] Step 8: Obtain the immediate reward of the network according to the reward function constructed in Step 4, complete a single environmental interaction, switch to the state at the next moment, and store the experience sample in the experience pool;
[0017] Step 9: Determine whether t < T is satisfied, where T is the total number of steps in the ep episode. If so, then t = t + 1, and return to Step 7. If not, then enter Step 10;
[0018] Step 10: Calculate the variational free energy and the value network loss;
[0019] Step 11: Minimize the variational free energy and the value network loss to update the network parameters;
[0020] Step 12: Determine whether ep < EP is satisfied, where EP is the total number of episodes. If so, then ep = ep + 1, and return to Step 6. If not, then the optimization ends, and the optimized algorithm model is obtained for intelligent resource allocation, and the transmission rate of the secondary network at this time is calculated.
[0021] To optimize the above technical solution, the specific measures taken also include:
[0022] The above Step 1 includes:
[0023] Step 1-1: Set the channel power gain β per unit distance ref ;
[0024] Step 1-2: Calculate the distance between the nth UAV and the kth SU The distance between PBS and the jth PU
[0025] Step 1-3: According to β ref , Calculate the channel power gain between UAV and SU According to β ref , Calculate the channel power gain between the primary base station PBS and PU
[0026] The above step 2 includes:
[0027] Step 2-1: Calculate the signal-to-noise ratio of the nth UAV and the kth SU in the mth sub-band according to the following formula:
[0028]
[0029] in, and denote the transmission power of UAV and PBS in the mth sub-band, σ 2 represents the noise power, J represents the total number of PUs, ρ k,n [m] = 1 is a binary variable used to indicate whether the mth sub-band is occupied. k,n When [m] = 1, the mth sub-band is occupied by the communication link between the nth UAV and the kth SU, otherwise ρ k,n [m] = 0;
[0030] is the channel power gain from the nth UAV to the kth secondary user SU, and the channel power gain from the primary base station to the jth primary user PU;
[0031] Step 2-2: Calculate the transmission rate of the kth SU and the jth PU:
[0032]
[0033]
[0034] Wherein, B represents the bandwidth of each sub-band.
[0035] The input state information variable of the mth sub-band agent in step 3 above is:
[0036]
[0037] in represents the frequency band allocation strategy of the mth sub-band agent at the t-1th time step;
[0038] is the channel power gain from the nth UAV to the kth secondary user SU, and the channel power gain from the primary base station to the jth primary user PU; and Represent the UAV set and the secondary user set respectively, Yes and A collection of is the transmission rate A collection of;
[0039] represents the position of the UAV at the t-1th time step, Input for some observable states;
[0040] Construct the input state information variable of the nth UAV agent
[0041] The above step 4 constructs the reward function r of the mth sub-band agent m :
[0042]
[0043] Among them, α1, α2, α3, and α4 are all non-negative constants, is the penalty term when the minimum transmission rate of the PU is not met, and It is the penalty term when the sum of the transmission rates of users with high transmission rate requirements and users with low transmission rate requirements is not met; the specific expression is as follows:
[0044]
[0045]
[0046]
[0047] in, and They represent the minimum transmission rate thresholds for users with high transmission rate requirements and users with low transmission rate requirements respectively.
[0048] Step 4 above constructs the reward function r of the nth UAV agent n :
[0049]
[0050] in and are the set of users with high transmission rate requirements and users with low transmission rate requirements, d k,n is the distance between the kth SU and the nth UAV;
[0051] is a penalty term. When the distance between the UAV and the low-rate demand user is greater than the threshold d thr When the reduction and reward, specifically expressed as:
[0052]
[0053] Step 7 above is based on the input status Get the sub-band allocation action and the UAV flight direction action respectively
[0054] Step 8 above gets the network's instant reward Complete a single environment interaction and switch to the next state and The experience sample {s t+1 ,s t ,s t-1 ,a t+1 ,s t ,r t}Stored in the experience pool.
[0055] The above step 10 includes:
[0056] Step 10-1. Calculate the output A of the active inference conversion model μ (s t-1 ,a t-1 ), A(·) and μ represent the mapping and network parameter matrices of the active inference conversion model, respectively, and are connected with s t Calculate the least squares error to get the network loss in represents the least squares loss function;
[0057] Step 10-2: Active inference target network outputs target expected free energy Q ω (s t+1 ), Q(·) and ω represent the mapping and network parameter matrix of the target network respectively, and the policy network output π(·) and Represent the mapping of the policy network and the network parameter matrix, respectively. The estimated expected free energy is obtained by one-step bootstrapping.
[0058] Step 10-3, the value network passes the state input st Get the output G θ (a t |s t ), G(·) and θ represent the mapping of the value network and the network parameter matrix, respectively. The active inference value network loss is obtained by estimating the difference between the expected free energy and the target free energy.
[0059] Step 10-4: Model the distribution over actions as a precision-weighted Boltzmann distribution σ(-γG θ (a t |s t ), combined with the strategy network output Entropy H t , conversion model loss function l(μ), value network output G θ (a t |s t ), calculate the intrinsic variational free energy of active inference
[0060] The above step 11 updates the network parameters according to the gradient descent algorithm.
[0061]
[0062]
[0063]
[0064] Among them, ∈ t ,∈ p and ∈ v are the learning rates of the conversion model, policy network, and value network, respectively. Represents a gradient computation operation.
[0065] The present invention has the following beneficial effects:
[0066] The present invention uses a new partially observable deep multi-agent active reasoning general framework for 6G resource allocation, uses a belief-based learning method to achieve active reasoning, simulates the behavior of agents in the real world by minimizing free energy, and solves the information missing problem between distributed agents in partially observable systems through distributed training and distributed execution of multi-agents, which can not only improve resource allocation performance but also improve decision-making speed. The present invention enables the spectrum sharing system to obtain the maximum resource allocation efficiency and rate, and accelerates the convergence speed of the network, with stronger real-time performance, and the framework has strong generalization ability and can be expanded to various wireless communication scenarios, which makes the invention better applied in practice.
[0067] Compared with the traditional deep reinforcement learning method that completely relies on external rewards, the deep active reasoning method proposed in the present invention minimizes the intrinsic free energy of the intelligent agent itself and completes the automatic update of the intelligent agent through a belief-based learning method. Compared with deep reinforcement learning, it is more practical and can therefore achieve faster convergence speed.
[0068] The distributed training and distributed execution multi-agent learning framework can overcome the assumption that traditional methods rely on full observation of the environment, and the multi-agent collaboration method based on sub-bands greatly reduces the dimension of the action space, thereby saving computing and storage overhead.
[0069] This invention takes into account the dynamic transmission rate requirements of users in a multi-UAV spectrum sharing scenario for the first time, and verifies the proposed PODMAI framework in this system model. Compared with traditional optimization-based algorithms, it solves the problem that many parameters are difficult to optimize online at the same time. The framework is highly versatile and can be easily extended to various system scenarios. BRIEF DESCRIPTION OF THE DRAWINGS
[0070] Figure 1 is a flow chart of a deep active inference algorithm for intelligent resource allocation proposed in the present invention;
[0071] Figure 2 It is a structural diagram of the deep active reasoning framework for intelligent resource allocation proposed by the present invention;
[0072] Figure 3 It is a comparison chart of the convergence curves of sub-band intelligent agents using the present invention and other existing technologies;
[0073] Figure 4 It is a comparison chart of the convergence curves of the UAV intelligent body using the present invention and other existing technologies;
[0074] Figure 5 This is a comparison chart of the transmission rates of primary and secondary users under different UAV transmission powers;
[0075] Figure 6 This is a comparison chart of the transmission rates of secondary users with different needs under different UAV transmission powers. DETAILED DESCRIPTION
[0076] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.
[0077] Although the steps in the present invention are arranged with numbers, they are not used to limit the order of the steps. Unless the order of the steps is clearly stated or the execution of a certain step requires other steps as a basis, the relative order of the steps can be adjusted. It is understood that the term "and / or" used in this article involves and covers any and all possible combinations of one or more of the associated listed items.
[0078] The present invention provides an intelligent resource allocation method for 6G based on partially observable active reasoning, which proposes a partially observable deep multi-agent active reasoning framework for 6G resource allocation, which mainly solves the problems of slow convergence speed, poor real-time performance, dependence on the fully observed state of the environment, and poor model generalization ability of the existing resource allocation method. The implementation steps are: calculating the air-to-ground and ground-to-ground channel power gains; calculating the transmission rates of the primary user and the secondary user; constructing the state input information variables of the multi-agent; constructing the multi-agent reward function; initializing the training cycle and iteration rounds; calculating the action output according to the initial state; single-step environment interaction; judging whether the iteration is completed within a single cycle; calculating the variational free energy and value network loss; minimizing the variational free energy and value network loss to complete the network parameter update; judging whether the total training cycle is completed; outputting the optimized model, and calculating the system and transmission rate. The present invention designs an intelligent spectrum allocation and trajectory optimization scheme based on active reasoning. Through the belief-based learning method, the deep active reasoning agent is automatically updated, which achieves a faster convergence speed. The distributed training and distributed execution multi-agent learning framework can solve the problem that traditional methods rely on full observation of the environment. Compared with traditional optimization-based algorithms, it solves the problem that many parameters are difficult to optimize online at the same time. The framework is highly versatile and can be easily extended to various system scenarios. Figure 1 The specific steps are as follows:
[0079] Step 1: Calculate the channel power gain from the nth UAV to the kth secondary user (SU) Channel power gain from the primary base station (PBS) to the jth primary user (PU)
[0080] Step 1-1: Set the channel power gain β per unit distance ref =1×10 -3 ;
[0081] Step 1-2: Calculate the distance between the nth UAV and the kth SU The distance between PBS and the jth PU where w b =(x b ,y b ) represents the horizontal position coordinate of PBS, w p,j =(xp,j ,y p,j ) represents the horizontal position coordinate of the jth PU, w s,k =(x s,k ,y s,k ) represents the horizontal position coordinate of the kth SU, q c,n =(x c,n ,y c,n ) represents the horizontal position coordinate of the nth UAV.
[0082] Step 1-3: Calculate the channel power gain between UAV and SU based on the line-of-sight transmission link Calculate the channel power gain between PBS and PU based on small-scale Rayleigh fading channel where ζ j is a random variable with unit mean exponential distribution.
[0083] Step 2: Based on the channel power gain calculated in step 1, calculate the transmission rate of the kth SU and the jth PU;
[0084] Step 2-1: Calculate the signal-to-noise ratio of the nth UAV and the kth SU in the mth sub-band according to the following formula:
[0085]
[0086] in, and denote the transmission power of UAV and PBS in the mth sub-band, σ 2 represents the noise power, J represents the total number of PUs, ρ k,n [m] = 1 is a binary variable used to indicate whether the mth sub-band is occupied. Specifically, when ρ k,n When [m] = 1, the mth sub-band is occupied by the communication link between the nth UAV and the kth SU, otherwise ρ k,n [m]=0.
[0087] Step 2-2: Calculate the transmission rate of the kth SU and the jth PU:
[0088]
[0089]
[0090] Wherein, B represents the bandwidth of each sub-band.
[0091] Step 3: Construct the input state information variable of the mth sub-band agent in represents the frequency band allocation strategy of the mth sub-band agent at the t-1th time step, in represents the time at t-1 value, represents the position of the UAV at the t-1th time step, and Represent the UAV set and the secondary user set respectively, Yes and A collection of is the transmission rate A collection of It is a partially observable state input. Construct the input state information variable of the nth UAV agent
[0092] Step 4: Based on the transmission rate, construct the reward function r of the mth sub-band agent m :
[0093]
[0094] Among them, α1, α2, α3, and α4 are all non-negative constants, is the penalty term when the minimum transmission rate of the PU is not met, and It is the penalty term when the sum of the transmission rates of users with high transmission rate requirements and users with low transmission rate requirements is not met. The specific expression is as follows:
[0095]
[0096]
[0097]
[0098] in, and They represent the minimum transmission rate thresholds for users with high transmission rate requirements and users with low transmission rate requirements, respectively. M and N represent the number of sub-bands and the number of drones, respectively.
[0099] Construct the reward function r of the nth UAV agent n :
[0100]
[0101] in and are the set of users with high transmission rate requirements and users with low transmission rate requirements, d k,n is the distance between the kth SU and the nth UAV. is a penalty term that reduces the reward when the distance between the UAV and the low-rate demand user is greater than the threshold d thr as follows:
[0102]
[0103] Step 5: Initialize the training episode ep to 0;
[0104] Step 6: Initialize the time step t in the ep episode to 0;
[0105] Step 7: Based on the construction in Step 3 obtain the input state According to the input state obtain the sub-band allocation action and the UAV flight direction action respectively
[0106] Step 8: According to the construction in Step 4 obtain the immediate reward of the network obtain the immediate reward of the sub-band allocation network and the immediate reward of the UAV decision-making network Complete a single environment interaction, switch to the state at the next moment and Store the experience sample {s t+1 , s t , s t-1 , a t+1 , s t , r t} in the experience pool;
[0107] Step 9: Determine whether t < T is satisfied, where T is the total number of steps in the ep episode. If so, then t = t + 1, return to Step 7. If not, then proceed to Step 10;
[0108] Step 10: Calculate the variational free energy and the value network loss;
[0109] Step 10-1: Calculate the output A μ (s t-1 , a t-1 ) of the active inference transition model. A(·) and μ represent the mapping and the network parameter matrix of the active inference transition model respectively. Calculate the least squares error with s t to obtain the network loss where represents the least squares loss function;
[0110] Step 10-2: The active inference target network outputs the target expected free energy Q ω (s t+1), Q(·) and ω represent the mapping of the target network and the network parameter matrix respectively, and the policy network outputs π(·) and represent the mapping of the policy network and the network parameter matrix respectively. The estimated expected free energy is obtained through one-step bootstrap
[0111] Step 10-3: The value network takes the state input s t to obtain the output G θ (a t |s t ). G(·) and θ represent the mapping of the value network and the network parameter matrix respectively. The active inference value network loss is obtained by estimating the difference between the expected free energy and the target free energy
[0112] Step 10-4: The distribution over actions can be modeled as a precision-weighted Boltzmann distribution σ(-γG θ (a t |s t )) on the expected free energy estimate (EFE) of the value network. Combining the output of the policy network entropy H t , the transformation model loss function l(μ), and the value network output G θ (a t |s t ), the intrinsic variational free energy of active inference is calculated
[0113] Step 11: Minimize the variational free energy and the value network loss to update the network parameters;
[0114] Update the network parameters according to the gradient descent algorithm,
[0115]
[0116]
[0117]
[0118] where, ∈ t , ∈ p and ∈ v are the learning rates of the transformation model, the policy network, and the value network respectively, denotes the gradient calculation operation.
[0119] Step 12: Determine whether ep < EP is satisfied, where EP is the total number of episodes. If so, then ep = ep + 1, and return to Step 6. If not, then the optimization ends, the optimized model is obtained, and the secondary network transmission rate at this time is calculated.
[0120] The effect of the present invention is further described below in conjunction with simulation experiments.
[0121] Simulation conditions and parameter settings:
[0122] Assume that there are M = 2 sub-bands in the system, the number of primary users J = 2, the number of secondary users K = 2, and the number of drones N = 2. The channel power gain β at a reference distance of 1 meter ref =1×10 -3 The UAV flies at a fixed altitude of 100 meters and the background noise power is a fixed -169dBm.
[0123] Simulation content:
[0124] Figure 3 This is a comparison chart of the convergence curves of the sub-band intelligent agent trained using the present invention and the prior art. Figure 3 The horizontal axis in represents the number of training rounds, and the vertical axis represents the average reward function. It can be seen that the rewards of the scheme proposed in the present invention fluctuate in the first 600 training cycles and tend to stabilize after 1000 training cycles. Compared with the DQN method, the method proposed in the present invention achieves a convergence speed improvement of about 400 training cycles. In addition, the method proposed in the present invention obtains higher rewards than the random spectrum allocation and random UAV trajectory schemes, which verifies the effectiveness of our proposed method.
[0125] Figure 4 The figure below is a comparison of the convergence curves of the UAV agent trained using the present invention and the prior art. It can be seen that the solution proposed by the present invention speeds up the training cycle by about 600 compared with the DQN method. However, the random strategy has never completed convergence within 1500 training rounds. This verifies that the belief-based learning method in the framework proposed by the present invention is more practical than the value-based learning in reinforcement learning. At the same time, more efficient exploration can be obtained in the early stages of training.
[0126] Figure 5 This is a comparison chart of the primary and secondary networks and transmission rates using the present invention and the prior art. The horizontal axis represents the UAV transmission power, and the vertical axis represents the system and transmission rate. It can be seen that the training method of the present invention can obtain a higher secondary network and transmission rate than other baseline methods. When the transmission power exceeds 180mW, the interference to the primary user exceeds the threshold, so the secondary network and transmission rate tend to be stable.
[0127] Figure 6This is a comparison chart of the transmission rates of different users between the present invention and the prior art. The horizontal axis represents the transmission power of the UAV, and the vertical axis represents the sum transmission rates of users with different rate requirements. It can be seen that the method of the present invention significantly improves the sum transmission rates of users with high transmission rate requirements while ensuring the communication of users with low transmission rate requirements, and the sum transmission rate and the UAV transmission power show an increasing trend. When the transmission power exceeds 180mW, the sum interference to the primary user exceeds the threshold, so the sum transmission rates of the two different types of secondary users tend to be stable.
[0128] Based on the above simulation results and analysis, the partially observable deep multi-agent active reasoning framework proposed in the present invention can enable the spectrum sharing system to obtain the maximum sum rate, accelerate the convergence speed of the network, and have stronger real-time performance. The framework has strong generalization ability and can be extended to various wireless communication scenarios, which enables the invention to be better applied in practice.
[0129] It will be apparent to those skilled in the art that the invention is not limited to the details of the exemplary embodiments described above and that the invention can be implemented in other specific forms without departing from the spirit or essential features of the invention. Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description, and it is intended that all variations falling within the meaning and scope of the equivalent elements of the claims be included in the invention. Any reference numeral in a claim should not be considered as limiting the claim to which it relates.
[0130] In addition, it should be understood that although the present specification is described according to implementation modes, not every implementation mode contains only one independent technical solution. This description of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment may also be appropriately combined to form other implementation modes that can be understood by those skilled in the art.
Claims
1. An intelligent resource allocation method based on partially observable active reasoning for 6G, characterized in that: Including: Step 1: Calculate the channel power gain from the UAV to the secondary user (SU) and the channel power gain from the primary base station to the primary user (PU). Step 2: Calculate the transmission rates of the SU and the PU based on the channel power gains obtained in Step 1. Step 3: Construct the input state information variables of the sub-band agent and the input state information variables of the UAV agent. Step 4: Based on the transmission rates, construct the reward function of the sub-band agent and the reward function of the UAV agent. Step 5: Initialize the training episode ep to 0. Step 6: Initialize the time step t in the ep-th episode to 0. Step 7: Obtain the input state according to the input state information variables constructed in Step 3, and respectively obtain the sub-band allocation action and the UAV flight direction action. Step 8: Obtain the immediate reward of the network according to the reward function constructed in Step 4, complete a single environment interaction, switch to the state at the next moment, and store the experience sample in the experience pool. Step 9: Determine whether t < T is satisfied, where T is the total number of steps in the ep-th episode. If so, then t = t + 1, and return to Step 7. If not, then proceed to Step 10. Step 10: Calculate the variational free energy and the value network loss. Step 11: Minimize the variational free energy and the value network loss to update the network parameters. Step 12: Determine whether ep < EP is satisfied, where EP is the total number of episodes. If so, then ep = ep + 1, and return to Step 6. If not, then the optimization ends, and the optimized algorithm model is obtained for intelligent resource allocation, and the transmission rate of the secondary network at this time is calculated. The input state information variables of the m-th sub-band agent are constructed in Step 3: in represents the frequency band allocation strategy of the mth sub-band agent at the t-1th time step; is the channel power gain from the nth UAV to the kth secondary user SU, and the channel power gain from the primary base station to the jth primary user PU, and Represent the UAV set and the secondary user set respectively, Yes and A collection of is the transmission rate A collection of; represents the position of the UAV at the t-1th time step, Input for some observable states; Construct the input state information variable of the nth UAV agent The step 4 constructs the reward function r of the mth sub-band agent m : Among them, α1, α2, α3, and α4 are all non-negative constants, is the penalty term when the minimum transmission rate of the PU is not met, and It is the penalty term when the sum of the transmission rates of users with high transmission rate requirements and users with low transmission rate requirements is not met; the specific expression is as follows: in, and Respectively represent the minimum transmission rate thresholds of users with high transmission rate requirements and users with low transmission rate requirements; Step 4 constructs the reward function r of the nth UAV agent n : in and are respectively a set of users with high transmission rate requirements and a set of users with low transmission rate requirements, d k,n is the distance between the kth SU and the nth UAV; is a penalty term. When the distance between the UAV and the low-rate demand user is greater than the threshold d thr When the reduction and reward, specifically expressed as: Step 7 is based on the input status Get the sub-band allocation action and the UAV flight direction action respectively 2. The intelligent resource allocation method based on partially observable active reasoning for 6G according to claim 1 is characterized in that: Step 1 includes: Step 1-1: Set the channel power gain β per unit distance ref ; Step 1-2: Calculate the distance between the nth UAV and the kth SU The distance between PBS and the jth PU Step 1-3: According to β ref , Calculate the channel power gain between UAV and SU According to β ref , Calculate the channel power gain between the primary base station PBS and PU 3. The intelligent resource allocation method based on partially observable active reasoning for 6G according to claim 1 is characterized in that: Step 2 includes: Step 2-1: Calculate the signal-to-noise ratio of the nth UAV and the kth SU in the mth sub-band according to the following formula: in, and denote the transmission power of UAV and PBS in the mth sub-band, σ 2 represents the noise power, J represents the total number of PUs, ρ k,n [m] = 1 is a binary variable used to indicate whether the mth sub-band is occupied. k,n When [m] = 1, the mth sub-band is occupied by the communication link between the nth UAV and the kth SU, otherwise ρ k,n [m] = 0; is the channel power gain from the nth UAV to the kth secondary user SU, and the channel power gain from the primary base station to the jth primary user PU; Step 2-2: Calculate the transmission rates of the k-th SU and the j-th PU: where B represents the bandwidth of each sub-band.
4. The 6G-oriented intelligent resource allocation method based on partially observable active reasoning according to claim 1 is characterized in that: Step 8 above gets the network’s instant reward Complete a single environment interaction and switch to the next state and The experience sample {s t+1 ,s t ,s t-1 ,a t+1 ,s t ,r t }Stored in the experience pool.
5. The intelligent resource allocation method based on partially observable active reasoning for 6G according to claim 1 is characterized in that: Step 10 includes: Step 10-1. Calculate the output A of the active inference conversion model μ (s t-1 ,a t-1 ), A(·) and μ represent the mapping and network parameter matrices of the active inference conversion model, respectively, and are connected with s t Calculate the least squares error to get the network loss in represents the least squares loss function; Step 10-2: Active inference target network outputs target expected free energy Q ω (s t+1 ), Q(·) and ω represent the mapping and network parameter matrix of the target network respectively, and the policy network outputs π θ (a t+1 |s t+1 ), π(·) and θ represent the mapping of the policy network and the network parameter matrix, respectively. The estimated expected free energy y is obtained by one-step bootstrapping. t =-r t +l(μ)+βπ θ (a t+1 |s t+1 )Q ω (s t +1 ); Step 10-3, the value network passes the state input s t Get the output G θ (a t |s t ), G(·) and θ represent the mapping of the value network and the network parameter matrix, respectively. The active inference value network loss is obtained by estimating the difference between the expected free energy and the target free energy. Step 10-4: Model the distribution over actions as a precision-weighted Boltzmann distribution σ(-γG θ (a t |s t ), combined with the policy network output π θ (a t+1 |s t+1 ), entropy H t , conversion model loss function l(μ), value network output G θ (a t |s t ), calculate the intrinsic variational free energy of active inference 6. The intelligent resource allocation method based on partially observable active reasoning for 6G according to claim 5 is characterized in that: In Step 11, the network parameters are updated according to the gradient descent algorithm. Among them, ∈ t ,∈ p and ∈ v are the learning rates of the conversion model, policy network, and value network, respectively. Represents a gradient computation operation.