CR-NOMA network power distribution optimization method based on SAC reinforcement learning

Abstract

The invention discloses a CR-NOMA network power distribution optimization method based on SAC reinforcement learning, and belongs to the field of wireless communication network optimization. According to the method, the SAC reinforcement learning algorithm is adopted, the dynamic characteristics of the complex network environment can be learned in a self-adaptive mode, an accurate mathematical model does not need to be established in advance, and compared with a traditional method based on mathematical optimization, the method has higher environment adaptability and robustness. By designing a continuous action space and a softmax mapping mechanism, power distribution parameters can be accurately controlled, quantization errors of a discrete action space are avoided, and the precision and flexibility of power distribution are remarkably improved.

Description

Technical Field

[0001] The present invention belongs to the field of wireless communication network optimization, and in particular relates to an intelligent power allocation optimization method for a non-orthogonal multiple access network using a cognitive radio and a soft actor-critic (SAC) reinforcement learning algorithm. Background Art

[0002] With the rapid development of wireless communication technology and the popularization of smart terminal devices, wireless spectrum resources are becoming increasingly scarce. How to improve spectrum efficiency has become a core challenge facing 5G and B5G communication systems. Non-Orthogonal Multiple Access (NOMA) technology, as one of the key technologies of 5G, allows multiple users to communicate on the same time-frequency resources by realizing multi-user multiplexing in the power domain, which can significantly improve spectrum efficiency compared with traditional orthogonal multiple access technology. In the power domain NOMA system, the base station allocates different transmit powers according to the channel quality of different users, and superimposes the signals of multiple users for transmission. The receiving end uses Successive Interference Cancellation (SIC) technology to separate and decode the signals of each user, thereby realizing non-orthogonal transmission of multiple users.

[0003] At the same time, cognitive radio (CR) technology offers another effective approach to addressing spectrum resource shortages by enabling dynamic spectrum access, allowing secondary users to opportunistically utilize licensed spectrum without interfering with primary users. In underlay cognitive radio networks, secondary users can utilize the same spectrum resources simultaneously with primary users, but their transmit power must be strictly controlled to ensure that interference to the primary user remains below a preset threshold. CR-NOMA networks, which combine CR with NOMA, leverage the strengths of both technologies in improving spectrum efficiency and are poised to become a key development direction for future wireless communication systems.

[0004] However, power allocation in CR-NOMA networks faces many technical challenges. First, power allocation must simultaneously satisfy multiple constraints, including the interference temperature constraint of the primary network, the minimum quality of service requirements of secondary users, and the total system power limit. Second, the wireless channel has time-varying fading characteristics, and the uncertainty of channel state information makes power allocation optimization difficult. Third, in the presence of eavesdroppers, physical layer security constraints must be considered to reduce the user's security outage probability (SOP) through a reasonable power allocation strategy. In addition, there are complex interference relationships between multiple users in the CR-NOMA system, and power allocation decisions will affect each other, making the optimization problem highly coupled and non-convex.

[0005] Existing CR-NOMA network power allocation methods are primarily based on traditional optimization theories and algorithms. Mathematical optimization-based methods establish an optimization model for power allocation and employ theoretical tools such as convex optimization, fractional programming, and game theory to solve the problem. While these methods have a solid theoretical foundation, they often require approximations or relaxations of the original non-convex problem, resulting in performance losses. Furthermore, the optimization algorithms are computationally complex, and the solution time increases dramatically, particularly as the number of users increases, making them difficult to meet the demands of real-time communication. Furthermore, most of these methods assume that channel state information is fully known. However, channel estimation errors are inevitable in practical systems and can significantly impact algorithm performance.

[0006] In recent years, machine learning, particularly deep learning, has demonstrated tremendous potential in wireless resource management. Supervised learning methods use large amounts of labeled data to train neural networks, learning the mapping from channel states to power allocation strategies. However, supervised learning methods require prior knowledge of optimal or near-optimal power allocation schemes as training labels, which is often difficult to achieve in complex CR-NOMA scenarios. Furthermore, supervised learning models have limited generalization capabilities, and performance degrades significantly when the actual network environment differs from the training data distribution.

[0007] Reinforcement Learning (RL), as a machine learning paradigm that does not require labeled data, learns optimal strategies by interacting with the environment and is particularly suitable for solving sequential decision-making problems in dynamic environments. In the field of wireless communications, reinforcement learning has been applied to many aspects such as dynamic spectrum access, power control, and user scheduling. However, traditional reinforcement learning algorithms such as Q-learning and SARSA are based on tables or simple function approximations and are difficult to handle the high-dimensional continuous state-action space of CR-NOMA systems. In contrast, deep reinforcement learning is more capable of handling complex problems. The present invention utilizes an intelligent power allocation method based on deep reinforcement learning to improve the real-time and adaptability of power allocation in CR-NOMA networks. Summary of the Invention

[0008] The purpose of this invention is to provide a CR-NOMA network power allocation optimization method based on SAC reinforcement learning, which realizes dynamic optimal power allocation by intelligently learning network environment characteristics, significantly reduces the probability of security interruption, and improves the overall performance of the system.

[0009] The technical solutions of the present invention are as follows:

[0010] A CR-NOMA network power allocation optimization method based on SAC reinforcement learning includes the following steps:

[0011] Step 1: In the cognitive radio combined with non-orthogonal multiple access system, the primary network includes a primary transmitter PT and a primary user PU; the secondary network includes a secondary base station ST, a relay SR, an eavesdropper E, and two legitimate users U1 and U2; all nodes in the system are equipped with a single antenna and operate in half-duplex mode; a primary link is set up between the primary transmitter PT and the primary user PU; due to the long distance between the secondary base station ST and the two legitimate users, there is no direct link, and auxiliary transmission is required through the relay SR, which works in a decoding and forwarding manner; the transmission power of the secondary network is limited by the interference power threshold of the primary network. At the same time, the communication signal in the primary network will also cause interference to the secondary network. There is an interference link between the primary transmitter PT and the relay SR; there is an eavesdropping link between the relay SR and the eavesdropper E; the SAC reinforcement learning power allocation controller controls the secondary base station ST and the relay SR through the control link; in addition, it is assumed that all communication links are independent of each other, have path loss and experience Nakagami-m fading, h ab It is used to represent the channel coefficient between nodes a and b, a,b∈{P,T,R,U1,U2,E}, where P, T, R, U1, U2, E represent the primary transmitter PT, the secondary base station ST, the relay SR, the legitimate user U1, the legitimate user U2, and the eavesdropper E respectively; h abThe attenuation factor and average channel power gain are expressed as m ab and Ω ab =E{|h ab | 2}, and there is β ab =m ab / Ω ab ;K ab Used to represent the distance between nodes a and b, H represents the height of the relay, and the horizontal distance between U1 and U2 and the relay

[0012] Define the state space of SAC reinforcement learning as a three-dimensional vector:

[0013] S=[ρ R ,H,η] (1)

[0014] where ρ R is the ratio of relay transmission power to system noise, η is the interference coefficient of the primary network to the secondary network; the state variable is mapped to the interval [0,1] through normalization processing;

[0015] Step 2: Double-hop transmission model and security interruption probability modeling

[0016] Based on the above network model, the system adopts a dual-hop transmission protocol: in the first time slot, the secondary base station ST sends the superimposed signal to the relay SR; in the second time slot, the relay SR sends the re-encoded signal together with the artificial interference signal to the two legitimate users. The privacy interruption probability of users U1 and U2 is derived as follows.

[0017] The superposition signal emitted by ST is expressed as:

[0018]

[0019] The signal transmitted by SR is expressed as:

[0020]

[0021] Among them, α1 and α2 are the base station power allocation coefficients, P T represents the transmission power of the secondary base station ST; x1 represents the signal of user U1; x2 represents the signal of user U2, b1, b2, b3 are the relay power allocation coefficients, P R Represents the transmission power of the relay SR, x a It is an artificial interference signal;

[0022] Based on SIC decoding technology and physical layer security theory, the security interruption probability expression of legitimate user U1 and legitimate U2 is derived:

[0023] First, define the instantaneous channel capacity of the first-hop TR link:

[0024]

[0025] in, represents the instantaneous channel capacity of the first hop of user U1, represents the instantaneous channel capacity of the first hop of user U2, It represents the signal-to-interference-noise ratio of the user U1 signal at the relay. Indicates the signal-to-interference-and-noise ratio of user U2 signal at the relay.

[0026] Instantaneous confidentiality capacity of the second-hop RU link:

[0027]

[0028] Among them, C R1 represents the instantaneous channel capacity of the second hop of user U1, C R2 represents the instantaneous channel capacity of the second hop of user U2, γ R1 represents the signal-to-interference-and-noise ratio of user U1 at the legitimate receiving end, γ R2 represents the signal-to-interference-and-noise ratio of user U2 at the legitimate receiving end, represents the signal-to-interference-noise ratio of the user U1 signal at the eavesdropper, represents the signal-to-interference-and-noise ratio of the user U2 signal at the eavesdropper, {x} + =max(x,0);

[0029] The interruption probability of legitimate user U1 is defined as:

[0030]

[0031] Expand to get:

[0032]

[0033] Among them, C th1 represents the target security rate of legitimate user U1,

[0034] make Through mathematical derivation, we can get:

[0035]

[0036] Similarly, the interruption probability of legitimate user U2 is:

[0037]

[0038] Among them, C th2 represents the target security rate of legitimate user U2,

[0039] make Expanded to:

[0040]

[0041] Considering the Nakagami-m fading characteristics of the channel, the channel power gain |h ab | 2 Following the gamma distribution, its probability density function and cumulative distribution function are:

[0042]

[0043] in m ab is the attenuation factor, Ω ab is the average channel power gain;

[0044] Step 3: Continuous Action Space Design

[0045] Build a deep neural network architecture for the SAC algorithm, including an actor network and a dual-critic network, to optimize the confidentiality interruption probability of users U1 and U2;

[0046] Design a continuous action space for SAC reinforcement learning and define a five-dimensional action vector a = [a1, a2, a3, a4, a5], where the first three dimensions encode the relay power allocation parameters and the last two dimensions encode the secondary base station power allocation parameters;

[0047] The continuous action is mapped to the power distribution coefficient through the softmax function:

[0048]

[0049] Ensure power constraints: b1+b2+b3=1, α1+α2=1, b1, b2, b3≥0, α1, α2≥0;

[0050] To meet the requirements of the NOMA protocol, a conditional check is performed after decoding: if b1>b2, swap; if α1>α2, swap, ensuring that b1<b2 and α1<α2;

[0051] The actor network uses a reparameterization technique to output the mean and standard deviation of the actions:

[0052]

[0053] where μ φ (s) represents the action mean output by the policy network, σ φ (s) represents the standard deviation of the action output by the policy network, ∈ represents the standard normal distribution noise, and φ represents the actor network parameters;

[0054] The dual-reviewer network structure is:

[0055]

[0056] in and represents two independent Q-value functions, used to estimate the value of a state-action pair, and represents the deep neural network function, θ1 and θ2 represent the parameters of the two critic networks, [s;a] represents the concatenated vector of state and action;

[0057] Calculation of target Q value for soft Q learning:

[0058]

[0059] Where r represents the immediate reward, s′ represents the next state, d represents the termination flag, γ represents the discount factor, and π φ represents the current strategy, α represents the temperature parameter, θ i′ represents the target network parameters;

[0060] Design a reward function based on the probability of safe interruption:

[0061]

[0062] Where λ1 and λ2 are user weight coefficients, λ3 is the constraint penalty weight, and penalty is the penalty term when the constraint is violated; the optimization goal is to minimize the weighted security interruption probability;

[0063] Step 5: Smart Power Distribution Deployment

[0064] After training convergence, the trained actor network is deployed to the actual system; network status information is collected in real time, and the optimal power allocation action is generated through the actor network, decoded into the actual power allocation coefficient and applied to the CR-NOMA system to achieve intelligent adaptive power optimization control.

[0065] The present invention utilizes a SAC reinforcement learning algorithm that can adaptively learn the dynamic characteristics of complex network environments without requiring a precise mathematical model. Compared to traditional mathematical optimization-based methods, this approach offers greater adaptability and robustness. By designing a continuous action space and softmax mapping mechanism, it precisely controls power allocation parameters, avoiding quantization errors in discrete action spaces and significantly improving the accuracy and flexibility of power allocation. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] Figure 1 This is a communication diagram of the CR-NOMA system based on drone relay in the present invention;

[0067] Figure 2 This is a training flow chart of the SAC algorithm of the present invention;

[0068] Figure 3 This is a graph showing the changes in the security interruption probability of users U1 and U2 during the SAC algorithm training process; DETAILED DESCRIPTION

[0069] The following describes the implementation of the present invention in detail with reference to the accompanying drawings. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process. However, the protection scope of the present invention is not limited to the following embodiments.

[0070] Step 1: If Figure 1 As shown in the figure, the main network includes a main transmitter PT and a main user PU; the secondary network includes a secondary base station ST, a relay SR, an eavesdropper E, and two legitimate users U1 and U2; all nodes in the system are equipped with a single antenna and work in half-duplex mode; a main link is set between the main transmitter PT and the main user PU; due to the long distance between the secondary base station ST and the two legitimate users, there is no direct link and auxiliary transmission is required through the relay SR, which works in a decoding and forwarding manner; the transmission power of the secondary network is limited by the interference power threshold of the main network. At the same time, the communication signal in the main network will also cause interference to the secondary network. There is an interference link between the main transmitter PT and the relay SR; there is an eavesdropping link between the relay SR and the eavesdropper E; the SAC reinforcement learning power allocation controller controls the secondary base station ST and the relay SR through the control link; in addition, it is assumed that all communication links are independent of each other, have path loss and experience Nakagami-m fading, h ab It is used to represent the channel coefficient between nodes a and b, a,b∈{P,T,R,U1,U2,E}, where P, T, R, U1, U2, E represent the primary transmitter PT, the secondary base station ST, the relay SR, the legitimate user U1, the legitimate user U2, and the eavesdropper E respectively; h ab The attenuation factor and average channel power gain are expressed as m ab and Ω ab =E{|h ab | 2}, and there is β ab =m ab / Ω ab ;K ab Used to represent the distance between nodes a and b, H represents the height of the relay, and U1 and U2 are horizontal distances from the relay.

[0071] Define the state space of SAC reinforcement learning as a three-dimensional vector: s = [ρ R ,H,η]. Where ρ Ris the ratio of relay transmission power to system noise, H is the relay height, and η is the interference coefficient of the primary network to the secondary network. The state variables are mapped to the interval [0,1] through normalization processing;

[0072] Step 2: Calculate the system security interruption probability based on the two-slot transmission protocol;

[0073] In the first time slot, the secondary network base station ST combines the signals of user U1 and user U2 and sends them to the relay SR. The superimposed signal transmitted by ST is expressed as Where α1 and α2 are the power allocation coefficients of signals x1 and x2, respectively, satisfying α1<α2 and α1+α2=1. In the second time slot, the relay sends the re-encoded signal to the two users and adds artificial noise as interference. The transmitted signal is expressed as Where b1, b2 and b3 are the power allocation coefficients of signals x1, x2 and interference signal respectively;

[0074] For user U1, set C th1 For its target security rate, when the instantaneous confidentiality capacity of U1 on the entire link is When , confidential communication will be interrupted, and the interruption probability of U1 is:

[0075]

[0076] in The first-hop probability is calculated by symbolic integration, and the second-hop probability is calculated by Gauss-Chebyshev integration:

[0077]

[0078] in, N is the number of points; is the signal-to-interference-and-noise ratio γ at the legitimate user U1 R1 The cumulative distribution function of is the signal-to-interference-and-noise ratio of user U2 at the eavesdropper The probability density function of

[0079] Similarly, for user U2, set C th2 For its target safety rate, the interruption probability of U2 is:

[0080]

[0081] in The same integration method is used to calculate the probability of each hop;

[0082] Step 3: Design a continuous action space for SAC reinforcement learning and define a five-dimensional action vector a = [a1, a2, a3, a4, a5], where a1, a2, a3 are the encodings of the relay power allocation parameters, and a4, a5 are the encodings of the base station power allocation parameters;

[0083] The continuous action is mapped to the power distribution coefficient through the softmax function:

[0084]

[0085]

[0086] Ensure power constraints: b1+b2+b3=1, α1+α2=1, b1>0, b2>0, b3>0, α1>0, α2>0;

[0087] To meet the requirements of the NOMA protocol, a conditional check is performed after decoding: if b1>b2, swap; if α1>α2, swap, ensuring that b1<b2 and α1<α2;

[0088] Step 4: Build the deep neural network architecture of the SAC algorithm, including the actor network and the dual critic network;

[0089] The actor network uses a reparameterization technique to output the mean and standard deviation of the actions:

[0090]

[0091] where μ φ (s) represents the action mean output by the policy network, σ φ (s) represents the standard deviation of the action output by the policy network, ∈ represents the standard normal distribution noise, and φ represents the actor network parameters; the network structure is:

[0092] Input layer: 3D state vector

[0093] Hidden layer 1: linear layer (3→256) + ReLU activation

[0094] Hidden layer 2: linear layer (256→256) + ReLU activation

[0095] Output layer: output 5-dimensional mean vector and 5-dimensional logarithmic standard deviation vector respectively

[0096] The dual-reviewer network structure is:

[0097]

[0098] in and represents two independent Q-value functions, used to estimate the value of a state-action pair, and represents the deep neural network function, θ1 and θ2 represent the parameters of the two critic networks, [s;a] represents the concatenated vector of state and action;

[0099] The input is an 8-dimensional state-action concatenation vector and the output is a 1-dimensional Q value;

[0100] Step 5: Design a reward function based on the probability of safe interruption:

[0101]

[0102] Where λ1 and λ2 are user weight coefficients, λ3 is the constraint penalty weight, and penalty is the penalty term when the constraint is violated; the optimization goal is to minimize the weighted security interruption probability;

[0103] In this embodiment, λ1=0.5, λ2=0.5, and λ3=10 are set; fairness and power efficiency are considered at the same time:

[0104]

[0105] The second term is used to improve fairness between two users, and the third term is used to reduce artificial noise power and improve power efficiency;

[0106] Step 6: Implement the training process of the SAC algorithm, such as Figure 2 As shown;

[0107] Maintain an experience replay buffer with a capacity of 100,000, storing transfer samples (s, a, r, s′, d); where s is the current state; a is the current action; r is the current reward; s′ is the next state; d is the termination flag;

[0108] The policy update uses the reparameterization technique:

[0109]

[0110] Where α is the temperature parameter; π φ (a|s) is the current strategy; Q θ (s,a) is the state-action value function;

[0111] Calculation of target Q value for soft Q learning:

[0112]

[0113] Where r represents the immediate reward, s′ represents the next state, d represents the termination flag, γ represents the discount factor, and π φ represents the current strategy, α represents the temperature parameter, θ i′ represents the target network parameters;

[0114] Critic loss function:

[0115]

[0116] y is the target Q value

[0117] Actor loss function:

[0118]

[0119] Temperature parameter adaptive update:

[0120]

[0121] in is the target entropy value;

[0122] Set training hyperparameters: learning rate α lr =3×10 -4 , discount factor γ = 0.99, soft update coefficient τ = 0.005, batch size 256, training 2000 episodes;

[0123] Step 7: After training convergence, the trained actor network is deployed to the actual system. Network status information is collected in real time, and the optimal power allocation action is generated through the actor network. This is decoded into the actual power allocation coefficient and applied to the CR-NOMA system to achieve intelligent and adaptive power optimization control.

[0124] This paper introduces the SAC reinforcement learning algorithm to achieve intelligent optimization of CR-NOMA network power allocation. The SAC algorithm balances exploration and utilization through the maximum entropy framework, making the learned strategy have good robustness. Experimental results Figure 3 The results show that the present invention can effectively learn the power allocation strategy of the CR-NOMA system and significantly reduce the system's security interruption probability from nearly 100% to about 10% under different channel conditions, with significant performance improvement.

Claims

1. A CR-NOMA network power allocation optimization method based on SAC reinforcement learning, characterized in that: The following steps are involved: Step 1: Build a cognitive radio combined with non-orthogonal multiple access system; Step 2: Double-hop transmission model and security interruption probability modeling; Step 3: Continuous action space design; Step 4: Deploy intelligent power distribution.

2. A CR-NOMA network power allocation optimization method based on SAC reinforcement learning according to claim 1, characterized in that: The step 1 is specifically as follows: In a cognitive radio combined with non-orthogonal multiple access system, the primary network includes a primary transmitter PT and a primary user PU; the secondary network includes a secondary base station ST, a relay SR, an eavesdropper E, and two legitimate users U1 and U2; all nodes in the system are equipped with a single antenna and operate in half-duplex mode; a primary link is set between the primary transmitter PT and the primary user PU; due to the long distance between the secondary base station ST and the two legitimate users, there is no direct link, and auxiliary transmission is required through the relay SR, which works in a decoding and forwarding manner; the transmission power of the secondary network is limited by the interference power threshold of the primary network. At the same time, the communication signal in the primary network will also cause interference to the secondary network. There is an interference link between the primary transmitter PT and the relay SR; there is an eavesdropping link between the relay SR and the eavesdropper E; the SAC reinforcement learning power allocation controller controls the secondary base station ST and the relay SR through the control link; in addition, it is assumed that all communication links are independent of each other, have path loss and experience Nakagami-m fading, h ab It is used to represent the channel coefficient between nodes a and b, a,b∈{P,T,R,U1,U2,E}, where P, T, R, U1, U2, E represent the primary transmitter PT, the secondary base station ST, the relay SR, the legitimate user U1, the legitimate user U2, and the eavesdropper E respectively; h ab The attenuation factor and average channel power gain are expressed as m ab and Ω ab =E{|h ab | 2 }, and there is β ab =m ab / Ω ab ;K ab Used to represent the distance between nodes a and b, H represents the height of the relay, and U1 and U2 are horizontal distances from the relay. Define the state space of SAC reinforcement learning as a three-dimensional vector: S=[ρ R ,H,η] (1) where ρ R is the ratio of relay transmission power to system noise, η is the interference coefficient of the primary network to the secondary network; the state variable is mapped to the [0,1] interval through normalization processing.

3. The CR-NOMA network power allocation optimization method based on SAC reinforcement learning according to claim 2 is characterized in that: The step 2 is specifically as follows: Based on the above network model, the system adopts a dual-hop transmission protocol: in the first time slot, the secondary base station ST sends the superimposed signal to the relay SR; in the second time slot, the relay SR sends the re-encoded signal together with the artificial interference signal to the two legitimate users. The privacy interruption probability of users U1 and U2 is derived as follows; The superposition signal emitted by ST is expressed as: The signal transmitted by SR is expressed as: Among them, α1 and α2 are the base station power allocation coefficients, P T represents the transmission power of the secondary base station ST; x1 represents the signal of user U1; x2 represents the signal of user U2, b1, b2, b3 are the relay power allocation coefficients, P R Represents the transmit power of the relay SR, x a It is an artificial interference signal; Based on SIC decoding technology and physical layer security theory, the security interruption probability expression of legitimate user U1 and legitimate U2 is derived: First, define the instantaneous channel capacity of the first-hop TR link: in, represents the instantaneous channel capacity of the first hop of user U1, represents the instantaneous channel capacity of the first hop of user U2, It represents the signal-to-interference-noise ratio of the user U1 signal at the relay. Indicates the signal-to-interference-and-noise ratio of the user U2 signal at the relay; Instantaneous confidentiality capacity of the second-hop RU link: Among them, C R1 represents the instantaneous channel capacity of the second hop of user U1, C R2 represents the instantaneous channel capacity of the second hop of user U2, γ R1 represents the signal-to-interference-and-noise ratio of user U1 at the legitimate receiving end, γ R2 represents the signal-to-interference-and-noise ratio of user U2 at the legitimate receiving end, represents the signal-to-interference-noise ratio of the user U1 signal at the eavesdropper, represents the signal-to-interference-and-noise ratio of the user U2 signal at the eavesdropper, {x} + =max(x,0); The interruption probability of legitimate user U1 is defined as: Expand to get: Among them, C th1 represents the target security rate of legitimate user U1, make Through mathematical derivation, we can get: Similarly, the interruption probability of legitimate user U2 is: Among them, C th2 represents the target security rate of legitimate user U2, make Expanded to: Considering the Nakagami-m fading characteristics of the channel, the channel power gain |h ab | 2 Following the gamma distribution, its probability density function and cumulative distribution function are: in m ab is the attenuation factor, Ω ab is the average channel power gain.

4. The CR-NOMA network power allocation optimization method based on SAC reinforcement learning according to claim 3 is characterized in that: The step 3 is as follows: Build a deep neural network architecture for the SAC algorithm, including an actor network and a dual-critic network, to optimize the confidentiality interruption probability of users U1 and U2; Design a continuous action space for SAC reinforcement learning and define a five-dimensional action vector a = [a1, a2, a3, a4, a5], where the first three dimensions encode the relay power allocation parameters and the last two dimensions encode the secondary base station power allocation parameters; The continuous action is mapped to the power distribution coefficient through the softmax function: Ensure power constraints: b1+b2+b3=1, α1+α2=1, b1, b2, b3≥0, α1, α2≥0; To meet the requirements of the NOMA protocol, a conditional check is performed after decoding: if b1>b2, swap; if α1>α2, swap, ensuring that b1<b2 and α1<α2; The actor network uses a reparameterization technique to output the mean and standard deviation of the actions: a=μ φ (s)+∈eσ φ (s), where μ φ (s) represents the action mean output by the policy network, σ φ (s) represents the standard deviation of the action output by the policy network, ∈ represents the standard normal distribution noise, and φ represents the actor network parameters; The dual-reviewer network structure is: in and represents two independent Q-value functions, used to estimate the value of a state-action pair, and represents the deep neural network function, θ1 and θ2 represent the parameters of the two critic networks, [s;a] represents the concatenated vector of state and action; Calculation of target Q value for soft Q learning: Where r represents the immediate reward, s′ represents the next state, d represents the termination flag, γ represents the discount factor, and π φ represents the current strategy, α represents the temperature parameter, θ i′ represents the target network parameters; Design a reward function based on the probability of safe interruption: Where λ1 and λ2 are user weight coefficients, λ3 is the constraint penalty weight, and penalty is the penalty term when the constraint is violated. The optimization goal is to minimize the weighted security interruption probability.

5. A CR-NOMA network power allocation optimization method based on SAC reinforcement learning according to claim 4, characterized in that: The step 4 is specifically as follows: After training convergence, the trained actor network is deployed to the actual system; network status information is collected in real time, and the optimal power allocation action is generated through the actor network, decoded into the actual power allocation coefficient and applied to the CR-NOMA system to achieve intelligent adaptive power optimization control.

Images