An optimization method of an IRS-assisted short packet communication broadcast system based on DDPG

Abstract

The present application relates to a kind of optimization method of IRS assisted short packet communication broadcast system based on DDPG, to improve the performance of broadcast system.The present application first constructs the IRS assisted downlink broadcast system model, including system composition and channel modeling and introducing the actual phase shift model to simulate the hardware limit of IRS;Second, according to the statistical channel state information, the optimization objective of system model is established;Then, the DDPG reinforcement learning model is constructed, and the phase shift of IRS and the beamforming vector at BS are optimized by training;Finally, the optimized parameters are used for minimum common block length analysis;To improve the performance of multi-user broadcast communication system.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of new generation mobile communication technology, and is a new method for optimizing intelligent reflecting surface assisted broadcast communication system by using deep reinforcement learning. BACKGROUND

[0002] Intelligent reflecting surface (IRS) is a programmable reflecting surface composed of a large number of passive and low-cost reflecting units. Its working principle is to intelligently control the propagation direction and characteristics of wireless signals by adjusting the phase shift of the reflecting units, thereby significantly enhancing the performance of the wireless communication system. As one of the key technologies for the development of 6G mobile communication, IRS has become a hot spot in the research of intelligent communication systems due to its low power consumption, low cost and high flexibility.

[0003] In recent years, research on IRS is mainly based on two phase shift models: ideal phase shift model and actual phase shift model. Among them, the ideal phase shift model assumes that each reflecting element can independently and accurately adjust the phase shift without hardware limitations. However, under actual hardware conditions, this assumption is difficult to meet. In order to be closer to the actual situation, the actual phase shift model (S. Abeywickrama, R. Zhang, Q. Wu, and C. Yuen, "Intelligent reflecting surface: Practical phase shift model and beamforming optimization," IEEE Trans. Commun., vol. 68, no. 9, pp. 5849-5863, Sep. 2020.) is proposed, which considers the coupling effect of the phase shift and amplitude of the IRS reflecting element. Specifically, in the actual phase shift model, the amplitude of each reflecting element changes with the change of the phase shift angle, and this coupling effect leads to additional complexity in design and optimization. In the broadcast scenario, the optimization problem of IRS usually involves the cooperative optimization of multi-user channels, and the core goal is to maximize the system performance by adjusting the phase shift and amplitude distribution of IRS. However, due to the nonlinear coupling characteristics of the actual phase shift model, such optimization problems are generally non-convex, with characteristics of high difficulty in solving and high computational complexity. Therefore, traditional gradient-based or convex optimization-based methods may face bottlenecks in solving these problems. To address these challenges, the Deep Deterministic Policy Gradient (DDPG) method can be used. DDPG algorithm can effectively handle problems with high-dimensional state space and continuous action space. Through DDPG, an agent can be designed to dynamically adjust the phase shift and amplitude distribution of IRS, thereby maximizing the system performance in the multi-user broadcast scenario. This method not only overcomes the limitations of traditional optimization methods, but also realizes adaptive optimization strategies in complex actual environments. SUMMARY

[0004] The present application provides an optimization method for an IRS-assisted short packet communication broadcast system based on DDPG, which optimizes the beamforming vector at the base station and the phase shift of the IRS by using the Deep Deterministic Policy Gradient (DDPG) algorithm to improve the performance of the multi-user broadcast communication system.

[0005] The purpose of the present application is achieved by the following technical solutions:

[0006] An optimization method for an IRS-assisted short packet communication broadcast system based on DDPG, the specific implementation steps are as follows:

[0007] Step 1, construct an IRS-assisted downlink broadcast system, including channel modeling between base station Base Station, BS, IRS and multiple user equipment User Equipment, UE, and introduce an actual phase shift model to simulate the hardware limitations of IRS.

[0008] Step 2, according to the system model established in step 1, derive the end-to-end channel distribution; according to the statistical channel state information, establish the system optimization goal, that is, maximize the minimum average signal-to-noise ratio of all users, to ensure that each user can obtain good communication quality. The specific implementation is as follows:

[0009] The received signal at the UEi end is y i = H i x + w, where UEi represents the i-th user equipment, H i = h i Φg0f H , g0, h i represent the channel coefficients between BS and IRS, IRS and UEi, respectively, Φ represents the reflection coefficient matrix of IRS, is the beamforming vector at the BS, f H represents the conjugate transpose of f, x represents a short packet signal transmitted by the BS with power P s , and w is an additive complex Gaussian white noise with mean 0 and variance σ 2 ; according to the received signal, the instantaneous signal-to-noise ratio at the UEi end is: γ i = ρ|h i Φg0f H | 2 , where |·| represents the modulus operation, ρ = P s / σ 2 is the transmit signal-to-noise ratio; in order to enable each user to receive reliable short packet signals, when ρ is given, maximizing the average signal-to-noise ratio E(γ i ) at the UEi end is equivalent to maximizing E(·) represents the mean operation;

[0010] Using the cyclically symmetric complex Gaussian distribution characteristics, the distribution of the channel H i can be derived as follows: where represents the real part operation, represents the imaginary part operation, j represents the imaginary unit, obeys the normal distribution with mean μ i and variance σ i 2

[0011] ​ ξ0and ξ i are the large-scale parameters of channel g0and channel h i , respectively, denotes the array response at the BS, k0and k i are the Rician factors of channel g0and channel h i , respectively, N denotes the number of IRS reflecting elements, β(θ l ) ∈ [0, 1] is the amplitude coefficient of the l-th reflecting element of the IRS, and are the line-of-sight components of channel g0and channel h i , respectively;

[0012] obeys a normal distribution with mean and variance , where

[0013]

[0014] From the above derivation, we have The distribution of E(|H i | 2 ) obtained can be used for the design of the reward function, thus giving the following optimization scheme:

[0015]

[0016] H i = h i Φg0f H , i ∈ {1, 2,..., K}

[0017]

[0018] -π≤θ l <π

[0019] where min{·} denotes the minimum element in the array.

[0020] Step 3, the DDPG algorithm is used to optimize the beamforming vector at the BS and the IRS phase shift. The DDPG model includes an action network, an evaluation network, a target action network, and a target evaluation network. The action network outputs the beamforming vector and the IRS phase shift, and the evaluation network outputs the Q value for evaluating the quality of the action. The target action network outputs the target action, and the target evaluation network outputs the target Q value, which are used to train the action network and the evaluation network, respectively, to reduce the fluctuations in the training process.

[0021] Step 4: By continuously iterating the training of the DDPG model, the beamforming vector at the BS and the IRS phase shift configuration are optimized, and the optimized parameters are saved.

[0022] Step 5: Finally, the minimum common block length analysis is performed using the optimized BS beamforming vectors and IRS phase shifts.

[0023] Further:

[0024] The specific implementation of Step 1 is as follows:

[0025] In an IRS-assisted downlink broadcast system, assuming that there is an obstruction between the BS and the UE in the system and they cannot directly communicate, a BS equipped with M antennas broadcasts a short packet signal to K single-antenna UEs (denoted as UEi, i∈{1, 2,..., K}) under the assistance of an IRS with N reflecting elements. The reflection coefficient matrix of the IRS is defined as: which represents a diagonal matrix with as the diagonal elements, θ l is the phase shift coefficient of the lth reflecting element of the IRS, θ l ∈[-π, π), and the actual phase shift model amplitude-phase coupling relationship is:

[0026]

[0027] where β min ≥ 0, θ ≥ 0, and τ ≥ 0 are constants related to the specific circuit implementation. Assuming that the antennas of the BS and the reflecting elements of the IRS are arranged in a uniform linear array, the array responses at the BS and the IRS are where ζ ∈ {M, N}, d is the spacing between the reflecting elements of the IRS or the spacing between the antennas of the base station, λ is the wavelength, θ is the Angle of Arrival (AOA) or Angle of Departure (AOD). The channel coefficients between the BS and the IRS and between the IRS and UEi are denoted as g0and h i .

[0028] where α represents the path loss at a reference distance of 10 meters, α0and α i are path loss exponents, d0is the distance from the BS to the center of the IRS, d i is the distance from the center of the IRS to UEi, and are the line-of-sight components, denotes the complex domain, the AOD and AOA of the signal from the BS to the IRS are and the AOD from the IRS to UEi is It can be obtained that denotes The conjugate transpose of . and These are non-line-of-sight components, where each element follows a cyclic symmetric complex Gaussian distribution with a mean of 0 and a variance of 1.

[0029] The specific implementation of step 3 is as follows:

[0030] To establish the Markov decision process for DDPG reinforcement learning, the beamforming vector at BS and the IRS phase shift are key parameters. Therefore, the beamforming vector at BS and the IRS phase shift are used as action vectors, represented as follows:

[0031] a t =(θ 1,t ,θ 2,t ,...,θ N,t ,|f t |,arg(f t ))

[0032] Where arg(·) represents the argument operation, and the subscript t represents the step size at time t.

[0033] The state vector mainly consists of the channel state, the beamforming vector at the BS, and the phase shift and amplitude of the IRS, and is expressed as:

[0034]

[0035] in

[0036] In a statistical channel, the reward function can be expressed as:

[0037] r t =min{E(|H1| 2 ),E(|H2| 2 ),...,E(|H i | 2 ),...,E(|H K | 2 )}

[0038] The specific implementation of step 4 is as follows:

[0039] In the DDPG network framework, the Actor network and the target Actor network each consist of a fully connected input layer, two hidden layers, a fully connected output layer, and a Tanh function; the Critic network and the target Critic network each consist of a fully connected input layer, two hidden layers, and a fully connected output layer. All layers are connected using batch normalization layers and linear ReLU activation functions, and the Adam optimizer is used.

[0040] The update methods for each network are as follows:

[0041] Actor network parameters θ a Updating using gradient ascent can be represented as:

[0042]

[0043] Where α a To update the learning rate of the Actor network, B is the sampling batch. Let q(s) represent the policy gradient. t ,a t :θ c ) is a parameter of θ c Critic network inputs t and a t The resulting current action-value function Q,π(s) t :θ a ) Input s for the Actor network t Generate a t The execution strategy. The target Critic network generates the target reward:

[0044]

[0045] Where, λ r ∈(0,1] is the discount factor. The parameter is θ ct Target Critic network inputs t+1 and a t+1 The resulting target Q value, a t+1 The parameter is θ at Target Action Network Inputs t+1 The subsequent action. To make the current Q value and the target Q value close, their mean square Bellman error loss needs to be minimized, expressed as:

[0046]

[0047] And update θ according to gradient descent c , represented as:

[0048]

[0049] Where α c To update the learning rate of the evaluation network, the target network uses soft updates to perform weighted replication of parameters to ensure the directionality and stability of the network training process. The soft updates for the target Actor network and the target Critic network are expressed as follows:

[0050] θ ct ←τ r θ c -(1-τ r )θ ct

[0051] θ at ←τ r θ a -(1-τ r )θ at

[0052] Where τ r ∈(0,1] represents the learning rate for updating the evaluation network.

[0053] After defining the action vector, state vector, and reward function, the specific process of the DDPG optimization algorithm is as follows:

[0054] 2-1. Initialize the Actor network, target Actor network, Critic network, and target Critic network;

[0055] 2-2. Initialize the training round (episode) to 0;

[0056] 2-3. Initialize the channel state, IRS phase shift, and amplitude, etc.;

[0057] 2-4. Initialize the step size to 0;

[0058] 2-5. Based on the current state s t The Actor network outputs action a. t ;

[0059] 2-6. Obtain the amplitude corresponding to each phase shift of the IRS unit based on the amplitude-phase shift coupling relationship;

[0060] 2-7. Calculate the reward for the current step: min{E(|H1|)} 2 ),E(|H2| 2 ),...,E(|H i | 2 ),...,E(|H K | 2 )};

[0061] 2-8. Based on the current state s t and action a t Get the new state s t+1 ;

[0062] 2-9. The obtained experience tuples (s) t ,a t ,r t ,s t+1 Fill into the buffer;

[0063] 2-10. If the buffer is full, update the parameters of the Actor network, the target Actor network, the Critic network, and the target Critic network.

[0064] 2-11. Update the current state s t =s t+1 ;

[0065] 2-12. Determine if the time step is less than the maximum step (max_step represents the maximum step size). If it is, return to step 2-5. If it is not, proceed to step 2-13.

[0066] 2-13. Determine whether the number of training rounds, episode, satisfies episode < max_episode (max_episode represents the maximum number of rounds). If it satisfies, return to step 2-3. If it does not satisfy, the optimization process ends, and the optimized IRS phase shift and amplitude information, beamforming vector at BS, and network weight parameters are obtained.

[0067] The specific implementation of step 5 is as follows: Finally, the minimum common block length analysis is performed using the optimized beamforming vector at BS and the IRS phase shift.

[0068] Given a short packet block length T, a signal-to-noise ratio γ, and a short packet block error rate (BLERε), the maximum achievable rate of short packet communication can be approximately expressed as: Where C(γ)=log2(1+γ) represents the Shannon capacity, and V(γ)=(1-(1+γ)) -2 (log2e) 2 Q represents the channel divergence. -1 (·) represents the inverse of the Gaussian Q-function. The physical layer information rate of the UE is Where F represents the number of information bits, ε can be approximated as... The average BLER in a wireless fading channel can be approximated as: Where f γ (x) is the probability density function of γ. Assume... The optimization algorithm for the least common block length to determine the target average BLER for all UEs is as follows:

[0069] (1) Given system parameters N, M, k0, k1, ..., k K d0, d1, ..., d K α, α1,...,α K , θ1,...,θ N f, F, Initialize i to 1;

[0070] (2) Initialize the block length T, and limit the block length to [M]. min M max ], let T - =M min T + =M max ;

[0071] (3) Calculate the block length T respectively. - and T + The average corresponding to the time

[0072] (4) If the calculated average BLER satisfies make ( (This represents the minimum block length required for reliable communication in UEi, indicating that there is no solution);

[0073] (5) If the calculated average BLER satisfies Then let

[0074] (6) If none of the above conditions are met, then satisfying T + -T - Under the condition that ≠ 1, let Calculate the corresponding average This represents the floor function;

[0075] (7) When satisfied Then let Otherwise

[0076] (8) Update T - or T + Continue with steps (6) to (7) until T. + -T - =1 to stop iteration, output

[0077] (9) Determine whether i satisfies i < K. If it does, set i = i + 1 and proceed to step (2); if it does not, proceed to step (10).

[0078] (10) Traversal The maximum value is selected and set as the minimum common block length T of all UEs. min .

[0079] This invention proposes an optimization method for an IRS-assisted short packet communication broadcast system based on DDPG. Its core lies in the first-time derivation of the end-to-end channel distribution of a multi-user broadcast system under the actual phase shift model of IRS, and the establishment of the DDPG system optimization objective based thereon. The DDPG reinforcement learning framework is designed to jointly optimize the beamforming vector at the BS and the IRS phase shift, thereby improving the communication quality of the system. Attached Figure Description

[0080] Figure 1 This is the system model of the present invention;

[0081] Figure 2 This is a flowchart illustrating the implementation of the present invention. Detailed Implementation

[0082] The present invention will be further described below with reference to the accompanying drawings, both theoretically and in terms of specific embodiments.

[0083] An optimization method for an IRS-assisted short packet communication broadcast system based on DDPG, the specific steps of which are as follows:

[0084] 1. An optimization method for an IRS-assisted short packet communication broadcast system based on DDPG, characterized by the following specific implementation steps:

[0085] Step 1: Construct an intelligent reflecting surface (IRS) assisted downlink broadcast system, including channel modeling between the base station (BS), IRS, and multiple user equipment (UE), and introduce an actual phase shift model to simulate the hardware limitations of the IRS;

[0086] Step 2: Based on the system model established in Step 1, derive the end-to-end channel distribution; establish the system optimization objective based on statistical channel state information, namely, maximizing the minimum average signal-to-noise ratio among all users to ensure that each user can obtain good communication quality; specifically as follows:

[0087] The UEi terminal receives the signal y i =H i x+w, where UEi represents the i-th user equipment, H i =h i Φg0f H g0, h i Let represent the channel coefficients between BS and IRS, and between IRS and UEi, respectively, and Φ represent the reflection coefficient matrix of IRS. f is the beamforming vector at BS. H Let f be the conjugate transpose, and x represent the power P emitted by BS. s The short packet signal, w, has a mean of 0 and a variance of σ.2 Additive complex white Gaussian noise; based on the received signal, the instantaneous signal-to-noise ratio at the UEi end is: γ i =ρ|h i Φg0f H | 2 Where |·| represents the modulo operation, ρ=P s / σ 2 To maximize the transmit signal-to-noise ratio; in order to ensure that each user receives a reliable short packet signal, given ρ, maximize the average signal-to-noise ratio E(γ) at the UEi end. i This is equivalent to maximizing E(·) represents the mean operation;

[0088] The channel H can be derived using the properties of a cyclic symmetric complex Gaussian distribution. i Distribution: in This indicates the operation of taking the real part. This indicates the operation of taking the imaginary part, where j represents the imaginary unit. Follows the mean μ i The variance is σ i 2 The normal distribution, where

[0089]

[0090] ξ i Channel g0 and channel h are respectively i Large-scale parameters, Indicates the array response at BS, k0 and k i These are channel g0 and channel h, respectively. i Rice factor, N represents the number of IRS reflective units, β(θ) l )∈[0,1] represents the amplitude coefficient of the l-th reflecting unit of the IRS. and Channel g0 and channel h are respectively i The line-of-sight component;

[0091] Follow the mean variance is The normal distribution, where

[0092]

[0093] From the above derivation, we can obtain... The obtained E(|H i | 2The distribution of ) can be used to design the reward function, thus the following optimization scheme is given: Utilizing the Max-Min algorithm (C.Li, C.He, L.Jiang, and F.Liu, “Robust Beamforming Design for Max–Min SINR in MIMO Interference Channels,” IEEE Commun. Lett., vol.20, no.4, pp.724–727, Apr.2016.), the beamforming vector at BS and the IRS phase shift are optimized to maximize the minimum average signal-to-noise ratio among all users, ensuring high communication quality for each user. Therefore, the problem is modeled as follows:

[0094]

[0095] H i =h i Φg0f H , i∈{1,2,...,K}

[0096]

[0097] -π≤θ l <π

[0098] Where min{·} means taking the minimum element in the array.

[0099] Step 3: The Deep Deterministic Policy Gradient (DDPG) algorithm is used to optimize the beamforming vector and IRS phase shift at BS. The DDPG model includes an action network, an evaluation network, a target action network, and a target evaluation network. The action network outputs the beamforming vector and IRS phase shift, and the evaluation network outputs the Q value to evaluate the quality of the action. The target action network outputs the target action, and the target evaluation network outputs the target Q value. These outputs are used to train the action network and the evaluation network, respectively, to reduce fluctuations during the training process.

[0100] Step 4: Optimize the beamforming vector and IRS phase shift configuration at BS by iteratively training the DDPG model, and save the optimized parameters;

[0101] Step 5: Finally, perform minimum common block length analysis using the optimized beamforming vector at BS and IRS phase shift.

[0102] Further:

[0103] The specific implementation of step 1 is as follows:

[0104] In an IRS-assisted downlink broadcast system, assuming there is congestion between the BS and UEs preventing direct communication, a BS equipped with M antennas broadcasts a short packet signal to K single-antenna UEs (denoted as UEi, i∈{1,2,...,K}) with the assistance of an IRS having N reflection elements. The reflection coefficient matrix of the IRS is defined as: It indicates that A diagonal matrix with diagonal elements. θ l Let θ be the phase shift coefficient of the l-th reflecting unit of the IRS. l ∈[-π,π), the amplitude-phase shift coupling relationship of the actual phase shift model is:

[0105]

[0106] Where β min ≥0, θ≥0, and τ≥0 are constants related to the specific circuit implementation. Assuming the antenna of BS and the reflector of IRS are arranged in a uniform linear array, the array response at BS and IRS is: Where ζ∈{M,N}, d is the spacing between IRS reflector units or the spacing between base station antennas, λ is the wavelength, and θ is the Angle of Arrival (AOA) or Angle of Departure (AOD). The channel coefficients between BS and IRS, and between IRS and UEi, are represented by g0 and h. i .

[0107] in α represents the path loss at a reference distance of 10 meters, α0 and α i Here, d is the path loss exponent, d0 is the distance from the BS to the IRS center, and d i The distance from the IRS center to UEi. and It is the line-of-sight component. Representing the complex domain, the AOD and AOA of the signal from BS to IRS are respectively and The AOD from IRS to UEi is θ AODi , can be obtained express The conjugate transpose of . and These are non-line-of-sight components, where each element follows a cyclic symmetric complex Gaussian distribution with a mean of 0 and a variance of 1.

[0108] The specific implementation of step 3 is as follows:

[0109] To establish the Markov decision process for DDPG reinforcement learning, the beamforming vector at BS and the IRS phase shift are key parameters. Therefore, the beamforming vector at BS and the IRS phase shift are used as action vectors, represented as follows:

[0110] a t =(θ 1,t ,θ 2,t ,...,θ N,t ,|f t |,arg(f t ))

[0111] Where arg(·) represents the argument operation, and the subscript t represents the step size at time t.

[0112] The state vector mainly consists of the channel state, the beamforming vector at the BS, and the phase shift and amplitude of the IRS, and is expressed as:

[0113]

[0114] in

[0115] In the statistical channel, the reward function is expressed as:

[0116] r t =min{E(|H1| 2 ),E(|H2| 2 ),...,E(|H i | 2 ),...,E(|H K | 2 )}

[0117] The specific implementation of step 4 is as follows:

[0118] In the DDPG network framework, the Actor network and the target Actor network each consist of a fully connected input layer, two hidden layers, a fully connected output layer, and a Tanh function; the Critic network and the target Critic network each consist of a fully connected input layer, two hidden layers, and a fully connected output layer. All layers are connected using batch normalization layers and linear ReLU activation functions, and the Adam optimizer is used.

[0119] The update methods for each network are as follows:

[0120] Actor network parameters θ a Updating using gradient ascent can be represented as:

[0121]

[0122] Where α aTo update the learning rate of the Actor network, B is the sampling batch. Let q(s) represent the policy gradient. t ,a t :θ c ) is a parameter of θ c Critic network inputs t and a t The resulting current action-value function Q,π(s) t :θ a ) Input s for the Actor network t Generate a t The execution strategy. The target Critic network generates the target reward:

[0123]

[0124] Where, λ r ∈(0,1] is the discount factor. The parameter is θ ct Target Critic network inputs t+1 and a t+1 The resulting target Q value, a t+1 The parameter is θ at Target Action Network Inputs t+1 The subsequent action. To make the current Q value and the target Q value close, their mean square Bellman error loss needs to be minimized, expressed as:

[0125]

[0126] And update θ according to gradient descent c , represented as:

[0127] θ c ←θ c -α c ▽ θc L(θ c )

[0128] Where α c To update the learning rate of the evaluation network, the target network uses soft updates to perform weighted replication of parameters to ensure the directionality and stability of the network training process. The soft updates for the target Actor network and the target Critic network are expressed as follows:

[0129] θ ct ←τ r θ c -(1-τ r )θ ct

[0130] θ at ←τ r θa -(1-τ r )θ at

[0131] Where τ r ∈(0,1] represents the learning rate for updating the evaluation network.

[0132] After defining the action vector, state vector, and reward function, the specific process of the DDPG optimization algorithm is as follows:

[0133] 2-1. Initialize the Actor network, target Actor network, Critic network, and target Critic network;

[0134] 2-2. Initialize the training round (episode) to 0;

[0135] 2-3. Initialize the channel state, IRS phase shift, and amplitude, etc.;

[0136] 2-4. Initialize the step size to 0;

[0137] 2-5. Based on the current state s t The Actor network outputs action a. t ;

[0138] 2-6. Obtain the amplitude corresponding to each phase shift of the IRS unit based on the amplitude-phase shift coupling relationship;

[0139] 2-7. Calculate the reward for the current step: min{E(|H1|)} 2 ),E(|H2| 2 ),...,E(|H i | 2 ),...,E(|H K | 2 )};

[0140] 2-8. Based on the current state s t and action a t Get the new state s t+1 ;

[0141] 2-9. The obtained experience tuples (s) t ,a t ,r t ,s t+1 Fill into the buffer;

[0142] 2-10. If the buffer is full, update the parameters of the Actor network, the target Actor network, the Critic network, and the target Critic network.

[0143] 2-11. Update the current state s t=s t+1 ;

[0144] 2-12. Determine if the time step is less than the maximum step (max_step represents the maximum step size). If it is, return to step 2-5. If it is not, proceed to step 2-13.

[0145] 2-13. Determine whether the number of training rounds, episode, satisfies episode < max_episode (max_episode represents the maximum number of rounds). If it satisfies, return to step 2-3. If it does not satisfy, the optimization process ends, and the optimized IRS phase shift and amplitude information, beamforming vector at BS, and network weight parameters are obtained.

[0146] Step 5: Finally, perform minimum common block length analysis using the optimized beamforming vector at BS and IRS phase shift.

[0147] Given a short packet block length T, a signal-to-noise ratio γ, and a short packet block error rate (BLERε), the maximum achievable rate of short packet communication can be approximately expressed as: Where C(γ)=log2(1+γ) represents the Shannon capacity, and V(γ)=(1-(1+γ)) -2 (log2e) 2 Q represents the channel divergence. -1 (·) represents the inverse of the Gaussian Q-function. The physical layer information rate of the UE is Where F represents the number of information bits, ε can be approximated as... The average BLER in a wireless fading channel can be approximated as: Where f γ (x) is the probability density function of γ. Assume... The optimization algorithm for the least common block length to determine the target average BLER for all UEs is as follows:

[0148] (1) Given system parameters N, M, k0, k1, ..., k K d0, d1, ..., d K α, α1,...,α K , θ1,...,θ N f, F, Initialize i to 1;

[0149] (2) Initialize the block length T, and limit the block length to [M]. min M max ], let T - =M min T + =M max ;

[0150] (3) Calculate the block length T respectively. - and T + The average corresponding to the time

[0151] (4) If the calculated average BLER satisfies make ( (This represents the minimum block length required for reliable communication in UEi, indicating that there is no solution);

[0152] (5) If the calculated average BLER satisfies Then let

[0153] (6) If none of the above conditions are met, then satisfying T + -T - Under the condition that ≠ 1, let Calculate the corresponding average This represents the floor function;

[0154] (7) When satisfied Then let Otherwise

[0155] (8) Update T - or T + Continue with steps (6) to (7) until T. + -T - =1 to stop iteration, output

[0156] (9) Determine whether i satisfies i < K. If it does, set i = i + 1 and proceed to step (2); if it does not, proceed to step (10).

[0157] (10) Traversal The maximum value is selected and set as the minimum common block length T of all UEs. min .

[0158] The present invention is further illustrated by the following specific embodiments:

[0159] An optimization method for an IRS-assisted short packet communication broadcast system based on DDPG is described below, with the following specific implementation steps:

[0160] Step 1: In an IRS-assisted downlink broadcast system, assuming there is congestion between the BS and UEs preventing direct communication, a BS equipped with M=4 antennas broadcasts a short packet signal to K=3 single-antenna UEs (denoted as UEi, i∈{1,2,...,K}) with the assistance of an IRS having N=64 reflection elements. The reflection coefficient matrix of the IRS is defined as: It indicates that A diagonal matrix with diagonal elements. β(θ l ) represents the amplitude coefficient of the l-th reflecting unit of the IRS, β(θ) l )∈[0,1],θ l Let θ be the phase shift coefficient of the l-th reflecting unit of the IRS. l ∈[-π,π), the amplitude-phase shift coupling relationship of the actual phase shift model is: β min =0.3, θ=0.3π, τ=1.6.

[0161] Assuming the antenna of BS and the reflector elements of IRS are both arranged in a uniform linear array, the responses at BS and IRS are: Where ζ∈{M,N}, d is the spacing between IRS reflective elements or the spacing between base station antennas, and λ is the wavelength. θ represents the signal's AOA or AOD. The channel coefficients between BS and IRS, and between IRS and UEi, are represented as g0 and h, respectively. i .in ξ0 and ξ i For large-scale parameters, α = 0.01 represents the path loss at a reference distance of 10 meters, α0 and α i The path loss exponent is α0 = α1 = α2 = α3 = 2, where d0 is the distance from the BS to the IRS center, and d i Let d0 = 40 meters, d1 = 30 meters, d2 = 40 meters, d3 = 70 meters, and k0 and k i These are channel g0 and channel h, respectively. i The Rice factors are k0 = 5, k1 = 3, k2 = 5, and k3 = 7. and It is the line-of-sight component. Representing the complex domain, the AOD and AOA of the signal from BS to IRS are respectively IRS to UEi AOD is achievable express The conjugate transpose of . and These are non-line-of-sight components, where each element follows a cyclic symmetric complex Gaussian distribution with a mean of 0 and a variance of 1.

[0162] Step 2: Based on the system model established in Step 1, derive the end-to-end channel distribution; establish the system optimization objective based on the statistical channel state information, that is, maximize the minimum average signal-to-noise ratio among all users to ensure that each user can obtain good communication quality.

[0163] Step 3: Employ the Deep Deterministic Policy Gradient (DDPG) algorithm to optimize the beamforming vector and IRS phase shift at point BS. The DDPG model comprises an action network, an evaluation network, a target action network, and a target evaluation network. The action network outputs the beamforming vector and IRS phase shift, while the evaluation network outputs the Q-value to assess the quality of the action. The target action network outputs the target action, and the target evaluation network outputs the target Q-value. These outputs are used to train the action network and the evaluation network, respectively, to reduce fluctuations during training.

[0164] Step 4: Optimize the beamforming vector and IRS phase shift configuration at BS by iteratively training the DDPG model, and save the optimized parameters.

[0165] Step 5: Finally, perform minimum common block length analysis using the optimized beamforming vector at BS and IRS phase shift.

[0166] Under the ideal phase shift model, for UEi, the optimal phase shift value of the l-th reflecting unit of the IRS is Table 1 shows the optimized IRS phase shift using the method described in this invention. The amplitude coefficients corresponding to the two sets of phase shift values ​​can be obtained through the amplitude-phase shift coupling formula of the actual phase shift model, thus yielding the IRS reflection coefficient matrix under the actual phase shift model. Table 2 shows the optimized beamforming vector at BS using the method described in this invention. Assuming the number of information bits F = 300 and the target average BLER is... Furthermore, using the beamforming vectors at BS given in Table 2, the minimum common block length optimization algorithm can be used to obtain the IRS reflection coefficient matrix obtained using the method described in this invention, where the minimum common block length T is set to... min The minimum transmit signal-to-noise ratio required for UE200 is ρ = 55.7 dB. When using the optimal phase shift values ​​for UE1, UE2, and UE3 under the ideal phase shift model, the minimum common block length T is... min The minimum transmit signal-to-noise ratios required for 200 are ρ = 99.2 dB, ρ = 99.3 dB, and ρ = 99.8 dB, respectively. The comparison shows that the IRS phase shift optimized using the method described in this invention significantly improves the performance of the IRS-assisted short packet communication broadcast system.

[0167] Table 1 shows the IRS phase shift values ​​obtained through optimization in specific implementation examples of the present invention.

[0168]

[0169] Table 2 shows the optimized beamforming vector values ​​at BS obtained in a specific embodiment of the present invention.

[0170]

Claims

1. An optimization method for an IRS-assisted short packet communication broadcast system based on DDPG, characterized in that, The specific implementation steps are as follows: Step 1: Construct an intelligent reflecting surface (IRS) assisted downlink broadcast system, including channel modeling between the base station (BS), IRS, and multiple user equipment (UE), and introduce an actual phase shift model to simulate the hardware limitations of the IRS; Step 2: Based on the system model established in Step 1, derive the end-to-end channel distribution; establish the system optimization objective based on statistical channel state information, namely, maximizing the minimum average signal-to-noise ratio among all users to ensure that each user can obtain good communication quality; the specific implementation is as follows: The received signal is ,in Indicates the first Each user device. , , They represent BS and IRS, IRS and The channel coefficients between them This represents the reflection coefficient matrix of the IRS. Beamforming vector at BS, express The conjugate transpose of . This indicates that the power transmitted by BS is short packet signal, The mean is 0 and the variance is Additive complex white Gaussian noise; based on the received signal, The instantaneous signal-to-noise ratio at the terminal is: ,in This indicates the modulo operation. To improve the transmission signal-to-noise ratio; to ensure that every user receives a reliable short packet signal, when Given, maximize Average signal-to-noise ratio at the end This is equivalent to maximizing , This represents the mean operation; The channel can be derived using the properties of a cyclic symmetric complex Gaussian distribution. Distribution: ,in This indicates the operation of taking the real part. This indicates the operation of taking the imaginary part. Represents the imaginary unit. Follow the mean variance is The normal distribution, where , , , and Channels With channel Large-scale parameters, This indicates the array response at BS. and These are channels With channel Rice factor, Indicates the number of IRS reflective units. For the IRS The amplitude coefficient of each reflecting unit, and Channels With channel The line-of-sight component; Follow the mean variance is The normal distribution, where , ; From the above derivation, we can obtain... , obtained The distribution of the reward function can be used to design the reward function, and the following optimization scheme is given: ； in This means retrieving the minimum element in the array. and These are constants related to the specific circuit implementation. For the IRS Phase shift coefficient of each reflecting unit; Step 3: The Deep Deterministic Policy Gradient (DDPG) algorithm is used to optimize the beamforming vector and IRS phase shift at BS. The DDPG model includes an action network, an evaluation network, a target action network, and a target evaluation network. The action network outputs the beamforming vector and IRS phase shift, and the evaluation network outputs the Q value to evaluate the quality of the action. The target action network outputs the target action, and the target evaluation network outputs the target Q value. These outputs are used to train the action network and the evaluation network, respectively, to reduce fluctuations during the training process. Step 4: Optimize the beamforming vector and IRS phase shift configuration at BS by iteratively training the DDPG model, and save the optimized parameters; Step 5: Finally, perform minimum common block length analysis using the optimized beamforming vector at BS and IRS phase shift.

2. The optimization method for an IRS-assisted short packet communication broadcast system based on DDPG according to claim 1, characterized in that, The specific implementation of step 1 is as follows: In an IRS-assisted downlink broadcast system, assuming there is a blockage between the BS and UE preventing direct communication, a system equipped with... The root antenna's BS has With the assistance of the IRS of each reflector unit, the short packet signal is broadcast to A single-antenna UE (denoted as) , The reflection coefficient matrix of the IRS is defined as follows: Its expression is in order to ( () is a diagonal matrix with diagonal elements. , The actual phase shift model amplitude-phase shift coupling relationship is: ， Assuming the antennas at BS and the reflective elements at IRS are arranged in a uniform linear array, the array responses at BS and IRS are: ,in , This refers to the spacing between IRS reflector units or the spacing between base station antennas. It's the wavelength. Angle of Arrival (AOA) or Angle of Departure (AOD) is the signal angle. BS and IRS, IRS and The channel coefficients between them are expressed as and ;in , , , , This indicates the path loss at a reference distance of 10 meters. and This is the path loss index. The distance from the BS to the center of the IRS. For IRS center to distance, and It is the line-of-sight component. Representing the complex domain, the AOD and AOA of the signal from BS to IRS are respectively and From IRS to AOD is , can be obtained , express The conjugate transpose of . ; and These are non-line-of-sight components, where each element follows a cyclic symmetric complex Gaussian distribution with a mean of 0 and a variance of 1.

3. The optimization method for an IRS-assisted short packet communication broadcast system based on DDPG according to claim 1, characterized in that, The specific implementation of step 3 is as follows: To establish the Markov decision process for DDPG reinforcement learning, the beamforming vector at BS and the IRS phase shift are key parameters. Therefore, the beamforming vector at BS and the IRS phase shift are used as action vectors, represented as follows: ； in Indicates argument operation, subscript Indicates step size time; The state vector mainly consists of the channel state, the beamforming vector at the BS, and the phase shift and amplitude of the IRS, and is expressed as: ； in ; In the statistical channel, the reward function is expressed as: 。 4. An optimization method for an IRS-assisted short packet communication broadcast system based on DDPG according to claim 1 or 3, characterized in that, The specific implementation of step 4 is as follows: In the DDPG network framework, the Actor network and the target Actor network consist of a fully connected input layer, two hidden layers, a fully connected output layer, and a Tanh function; the Critic network and the target Critic network consist of a fully connected input layer, two hidden layers, and a fully connected output layer; all layers of the network are connected by batch normalization layers and linear ReLU activation functions, and the Adam optimizer is used. The update methods for each network are as follows: Actor network parameters Updating using gradient ascent can be represented as: ； in To update the learning rate of the Actor network, For sampling batches, Represents the policy gradient. The parameter is Critic network input and The resulting current action-value function Q, Input to the Actor network produce Execution strategy; The target Critic network generates the target reward: ； in, This is the discount factor. The parameter is Target Critic network input and The resulting target Q value, The parameter is Target Action Network Input The subsequent action; to make the current Q value and the target Q value close, it is necessary to minimize their mean square Bellman error loss, expressed as: ； Update according to gradient descent , represented as: ； in To update the learning rate of the evaluation network; The target network uses soft updates to perform weighted replication of parameters to ensure the directionality and stability of the network training process. The soft updates of the target Actor network and the target Critic network are expressed as follows: ； ； in To update the learning rate of the evaluation network.

5. The optimization method for an IRS-assisted short packet communication broadcast system based on DDPG according to claim 4, characterized in that, After defining the action vector, state vector, and reward function, the specific process of the DDPG optimization algorithm is as follows: 2-1. Initialize the Actor network, target Actor network, Critic network, and target Critic network; 2-2. Initialize the training round (episode) to 0; 2-3. Initialize the channel state, IRS phase shift, and amplitude, etc.; 2-4. Initialize the step size to 0; 2-5. Based on the current status Actor network outputs actions ; 2-6. Obtain the amplitude corresponding to each phase shift of the IRS unit based on the amplitude-phase shift coupling relationship; 2-7. Calculate the reward for the current step. ; 2-8. Based on the current status and actions Get a new state ; 2-9. The obtained experience tuples Fill into the buffer; 2-10. If the buffer is full, update the parameters of the Actor network, the target Actor network, the Critic network, and the target Critic network. 2-11. Update the current status ; 2-12. Determine if the time step < max_step (max_step represents the maximum step size). If it is satisfied, return to step 2-5. If it is not satisfied, proceed to step 2-13. 2-13. Determine whether the number of training rounds, episode, satisfies episode < max_episode (max_episode represents the maximum number of rounds). If it satisfies, return to step 2-3. If it does not satisfy, the optimization process ends, and the optimized IRS phase shift and amplitude information, beamforming vector at BS, and network weight parameters are obtained.

6. The optimization method for an IRS-assisted short packet communication broadcast system based on DDPG according to claim 4, characterized in that, Step 5 is implemented as follows: For a given short data packet length Signal-to-noise ratio Block Error Rate (BLER) Under these conditions, the maximum achievable rate of short packet communication can be approximately expressed as: ,in Indicates Shannon capacity. Represents channel divergence, Gauss The inverse of the function; the physical layer information rate of the UE is ,in Indicates the number of information bits. It can be approximated as The average BLER in a wireless fading channel can be approximated as: ,in yes The probability density function; assuming The optimization algorithm for the least common block length to determine the target average BLER for all UEs is as follows: (1) Given system parameters , , , , , , , , , , , ,initialization =1; (2) Initialize block length Limit the block length to ,make , ; (3) Calculate the block length respectively. and The average BLER corresponding to the time , ; (4) If the calculated average BLER satisfies ,make ( Indicates UE The minimum block length required for reliable communication indicates that there is no solution. (5) If the calculated average BLER satisfies Then let ; (6) If none of the above conditions are met, then if the conditions are met... Under the condition that, let Calculate the corresponding average BLER , This represents the floor function; (7) When satisfied Then let Otherwise ; (8) Update or Continue with steps (6) to (7) until... Stop iteration, output ; (9) Judgment Does it meet the requirements? If satisfied, set If the condition is not met, proceed to step (2); if not, proceed to step (10). (10) Traversal Select the maximum value and set it as the minimum common block length of all UEs. .

Images