An IRS-assisted short packet communication system optimization method based on TD3 in an interference environment

Abstract

This invention relates to an optimization method for a smart reflector-assisted short packet communication system based on deep reinforcement learning in interference environments. The aim is to utilize the system's statistical channel state information to design a joint optimization based on TD3 active and passive beamforming to improve communication system performance. First, this invention constructs a model of the smart reflector-assisted short packet communication system under interference environments, including system composition and channel modeling, and introduces a physically realizable smart reflector actual phase shift model. Second, it establishes the optimization objective of the system model based on statistical channel state information. Next, it constructs a TD3 reinforcement learning model to jointly optimize the active beamforming of the base station and the passive beamforming of the smart reflector. Finally, it uses the optimized active and passive beamforming to perform minimum transmission block length analysis to improve the system's communication quality.

Description

Technical Field

[0001] This invention belongs to the field of next-generation mobile communication technology. It is a new method for optimizing intelligent reflector-assisted short packet communication systems using deep reinforcement learning. Specifically, it refers to an optimization method for TD3-based IRS-assisted short packet communication systems in interference environments. Background Technology

[0002] Intelligent Reflecting Surfaces (IRS) are a disruptive technology in the field of wireless communication in recent years and are considered one of the core candidate technologies for 6G networks. They significantly improve communication system performance by intelligently controlling the propagation environment of electromagnetic waves, while also being low-cost, low-power, and easy to deploy. They consist of a large number of programmable metasurface units, each of which can independently control the phase, amplitude, or polarization state of the incident electromagnetic wave, thereby dynamically reconstructing the wireless channel.

[0003] Current academic research on IRS mainly revolves around two phase shift models: the ideal phase shift model and the practical phase shift model. The ideal phase shift model is based on the theoretical assumption that the reflecting unit can achieve independent and precise phase control, ignoring the physical limitations in hardware implementation. However, in engineering practice, it has been found that this idealized assumption is difficult to hold effectively. In their research (S. Abeywickrama, R. Zhang, Q. Wu, and C. Yuen, “Intelligent reflecting surface: Practical phase shift model and beamforming optimization,” IEEE Trans. Commun., vol. 68, no. 9, pp. 5849–5863, Sep. 2020.), S. Abeywickrama et al. proposed a practical phase shift model that is more in line with engineering practice. This model reveals the key characteristic of the phase shift-amplitude coupling effect in the hardware implementation of IRS reflecting units. The actual reflection amplitude of each reflecting unit changes regularly with the phase shift. This mutual constraint relationship of electromagnetic parameters is significantly different from the assumption of the ideal model, and further leads to the challenge of multi-dimensional parameter joint optimization in system design. The discovery of this coupling characteristic necessitates the introduction of new constraints in the design of IRS beamforming algorithms, which significantly increases the complexity of mathematical modeling and engineering implementation. This has become a core issue that urgently needs to be addressed in the current process of practical application of IRS technology.

[0004] With the development of multi-antenna and dense networking technologies, the cooperative communication scenarios involving multiple antennas and interference pose severe challenges to IRS-assisted short-packet communication systems in practical deployments. To address these challenges, Reinforcement Learning (RL) offers a new approach to the IRS optimization problem. The core idea of ​​RL is to allow agents to interact tentatively in the environment and learn the optimal policy based on feedback. In recent years, Deep Reinforcement Learning (DRL) combines the advantages of deep learning and reinforcement learning, effectively handling problems in high-dimensional state spaces and continuous action spaces, providing an efficient decision-making tool for solving complex tasks. TD3 (TwinDelayed Deep Deterministic Policy Gradient) is an algorithm in reinforcement learning specifically designed to solve problems in continuous action spaces. It stabilizes Q-values ​​by designing a double Q-network and avoids overestimating Q-values ​​by selecting the smaller Q-value through comparison. It also designs a delayed update strategy for the action network, prioritizing the Q-value iteration of the evaluation network over the action network. After the Q-value output stabilizes, the direction of the derivative of the policy gradient is determined, avoiding interference from fluctuation noise on the optimization direction. To address the issue of insufficient exploration capability in the target network, exploration perturbations are introduced into the output of the target action network, sensitive noise is added, and the output of random variables is pruned to ensure the validity of the physical meaning. This approach has great potential in large-scale planning tasks and long-term continuous optimization designs. Summary of the Invention

[0005] This invention provides an optimization method for IRS-assisted short packet communication systems based on TD3 in interference environments. It utilizes the TD3 algorithm to jointly optimize the active beamforming of the base station and the passive beamforming of the IRS to improve the performance of the communication system.

[0006] The objective of this invention is achieved through the following technical solution:

[0007] An optimization method for a TD3-based IRS-assisted short packet communication system in an interference environment includes the following steps:

[0008] Step 1: Construct an Intelligent Reflecting Surface (IRS) assisted short packet communication system. This system includes a Base Station (BS) with M antennas, a User Equipment (UE) with a single antenna, and a system with... The system has an IRS with one reflector unit; due to obstruction, the BS can only communicate with the UE with the assistance of the IRS; the system includes an interference source that affects the received signals of the IRS and the UE; it is assumed that the location, transmit power, and transmission channel characteristics from the interference source to the IRS and the UE are known; and a physical phase shift model of the realizable IRS is introduced to construct the amplitude and phase shift constraint relationship.

[0009] Step 2: Construct a system optimization model using Statistical Channel State Information (CSI) to maximize the average reachability rate of short-packet communication for the UE, ensuring good communication quality even in interference environments. The specific implementation is as follows:

[0010] In interference scenarios, the BS transmits a signal that reaches the UE via a reflection link, and the UE receives the signal as follows: ,in The transmit power at BS, For the power of the interference source, The channel coefficients between the BS and the IRS. The channel coefficients between the IRS and the UE. The channel coefficient between the interference source and the IRS. The channel coefficient between the interference source and the UE. Represents the field of complex numbers. It is the power normalization signal transmitted by the BS to the UE. It is a power-normalized interference signal emitted by the interference source. Follows the pattern with mean 0 and variance of Additive white Gaussian noise, This is a diagonal matrix formed by arranging the elements of the IRS reflection coefficient vector in sequence. The IRS reflection coefficient vector. The normalized beamforming vector of the BS antenna array satisfies , This is the conjugate transpose of the matrix. This represents the vector 2-norm. In interference scenarios, it represents the instantaneous received signal-to-interference-plus-noise ratio (SINR) of the UE. Represented as:

[0011]

[0012] in , , This indicates the modulo operation.

[0013] Short packet communication under a given transport block length L and instantaneous received signal-to-noise ratio Block Error Rate (BLER) The achievable rate of a system in an additive white Gaussian noise channel is expressed as:

[0014]

[0015] in Indicates Shannon capacity. Represents channel divergence at high signal-to-noise ratios. , Gauss The inverse function of a function. Through joint optimization. and An optimization model for an IRS-assisted short packet communication system is constructed with the objective of maximizing the average reachable rate of the UE under interference conditions. The specific optimization problem is expressed as:

[0016]

[0017] in The maximum average achievable rate for UE short packet communication. This represents the mean operation. and ( ) represent the IRS number The amplitude coefficient and phase shift of each reflecting unit, , , These are constants related to the specific circuit implementation. According to existing technology, they can be obtained... for:

[0018]

[0019] Step 3: Employing the Twin-delayed Deep Deterministic Policy Gradient (TD3) algorithm, the BS active beamforming and IRS passive beamforming are jointly optimized. Key design elements of TD3 include the state space, action space, and reward function. The network model comprises an action network, an evaluation network, a target action network, and a target evaluation network. The action network outputs BS active beamforming vectors and IRS passive beamforming vectors, while the evaluation network outputs an action value function to assess the quality of actions performed in the current state. The network is trained by incorporating a dual-target network, a delayed update frequency strategy, a target smoothing strategy, an experience replay buffer, and a soft update method.

[0020] Step 4: By iteratively training the TD3 model, jointly optimize the BS active beamforming vector and the IRS passive beamforming vector, and use the optimized joint beamforming scheme to perform minimum system transport block length analysis.

[0021] Going further:

[0022] The specific implementation method of step 1 is as follows:

[0023] Assuming that BS and IRS are both uniform linear arrays (ULA), then the array responses at BS and IRS are: ,in , It is the distance between the centers of adjacent reflective units. It's the wavelength. The corresponding departure angle or arrival angle of the channel; the channel coefficient between the BS and the IRS. Channel coefficients between IRS and UE Channel coefficient between interference source and IRS and the channel coefficient between the interference source and the UE They are respectively , , and ,in It is the path loss exponent corresponding to the channel. It is a unit reference distance Path loss at the location, The distance from the BS center to the IRS center. The distance from the IRS center to the UE. The distance from the interference source to the IRS center. The distance from the interference source to the UE. , and They are respectively , and The corresponding Rice factor, and the line-of-sight components of each channel are respectively , and ,in The angle at which the BS antenna emits signals. The angle of arrival of the BS transmitted signal to the IRS. The signal origination angle from the IRS to the UE. To interfere with the IRS's angle of arrival, the non-line-of-sight components of each channel , , and The elements in the set are independent and all follow a cyclic symmetric complex Gaussian random variable with mean 0 and variance 1.

[0024] IRS reflection coefficient vector ,definition For the IRS The reflection coefficient of each IRS reflection unit in the actual phase-shift model and The coupling relationship is .

[0025] The specific implementation of step 3 is as follows:

[0026] For the design of key elements of TD3, in order to fully consider the description of environmental information by the state space, it is designed according to CSI as follows:

[0027]

[0028] in , , , , This represents taking the argument of the complex elements in the matrix. The IRS phase shift and BS beamforming vectors are designed as the action space: The target action network adds noise explicitly. And trim the output range, that is Where clip is the clipping function.

[0029] In a statistical channel, the reward function can be expressed as:

[0030]

[0031] The training process of TD3 first initializes the action network. , For network parameters, Given the input state, the first evaluation network , For network parameters, the second evaluation network Target Action Network First target evaluation network and the second objective evaluation network Create an experience replay buffer; initialize the state at the start of training. In time step In the middle, the execution action is given. , and The system calculates the amplitude of the IRS reflection coefficient and the BS beamforming vector, calculates the reward based on the reward function and updates the state, while the experience replay pool collects state transition information. When the experience pool reaches its capacity limit, it begins to randomly sample data. Send it to the training network.

[0032] For the two evaluation networks, the loss function is calculated using the mean squared error:

[0033]

[0034]

[0035] The time difference deviation target is calculated as follows: The target Q-value of the target evaluation network group is calculated as follows:

[0036]

[0037] Update the corresponding evaluation networks using gradient descent.

[0038]

[0039] in and The learning rates are used to evaluate Network 1 and Network 2, respectively. This represents gradient computation. The action network updates using gradient ascent.

[0040]

[0041] in The learning rate is updated for the action network. The objective is to maximize the average Q value, i.e.

[0042]

[0043] The target network is updated via a soft update:

[0044]

[0045] in This is the soft update coefficient.

[0046] TD3 consists of six networks: an action network, a target action network, an evaluation network group consisting of two evaluation networks, and a target evaluation network group consisting of two target evaluation networks. Its networks can be divided into a main network class and a target network class. The main network class consists of the action network and the evaluation network group, while the target network class consists of the target action network and the target evaluation network group. The input layer, first hidden layer, second hidden layer, third hidden layer, and output layer neurons of the action network are as follows: 600, 400, 400 and The hidden layers are activated using the ReLU function, and the output layer uses the Tanh function to limit the range. The target evaluation network group has the same neural network structure, with the following neuron nodes for the input layer, first hidden layer, second hidden layer, third hidden layer, fourth hidden layer, and output layer: The input layer uses 800, 600, 400, 200, and 1 as input parameters, performs layer normalization on the data, and the hidden layer uses the ReLU function for activation; the target network class structure is consistent with the corresponding structure of the main network class.

[0047] The specific implementation of step 4 is as follows:

[0048] In IRS-assisted short packet communication systems, given a transmission block length Number of information bits (The achievable rate at the UE is) Instantaneous received signal-to-noise ratio The BLER under additive complex white Gaussian noise channel can be approximated as: Then the closed-form solution of the average BLER is:

[0049]

[0050] in For instantaneous SINR reception The probability density function, where n is the approximate order of the Gauss-Chebyshev integral. , , , , , , , , , , , , For instantaneous SINR reception The cumulative distribution function. To solve... The cumulative distribution function is first obtained by using the moment matching method. Approximate shape parameters are The scale parameter is The gamma distribution will Approximate shape parameters are The scale parameter is gamma distribution, , , , Finally, we obtained... The cumulative distribution function is expressed as:

[0051]

[0052] in , , , , , The Whittaker W-function, defined by the confluence hypergeometry function, is... , Represents the confluence hypergeometry function. This represents the gamma function.

[0053] Assumption The optimization algorithm for the minimum transport block length of the system, which is the average BLER threshold for UE communication service quality requirements, is as follows:

[0054] (1) Given system parameters , , , , , , , , , , , , and ;

[0055] (2) Initialize block length Let the boundary value be . , ;

[0056] (3) Calculate the block length respectively. and The average BLER corresponding to the time and ;

[0057] (4) If This indicates that when the block length is maximized, the average BLER threshold requirement cannot be met, which means there is no solution. ;

[0058] (5) If This indicates that the minimum block length satisfies the requirement for the average BLER threshold, thus yielding... ;

[0059] (6) If none of the above conditions are met, it means that the minimum transmission block length of the system is within the range of... The minimum transport block length of the system under optimal phase is numerically solved using the binary search method. ,make Calculate the corresponding average BLER , This represents the integer division operation;

[0060] (7) If satisfied Then let Otherwise ;

[0061] (8) Repeat steps (6) to (7) to continuously update. or until Stop iteration and output. .

[0062] This invention proposes an optimization method for an IRS-assisted short packet communication system based on TD3 in an interference environment. Its core lies in the first introduction of TD3 into the IRS-assisted short packet communication system under interference environment and actual phase shift model, jointly optimizing BS active beamforming and IRS passive beamforming, and using the optimized active and passive beamforming to analyze the minimum transmission block length of the system, so as to improve the communication quality of the system. Attached Figure Description

[0063] Figure 1 This is the system model of the present invention;

[0064] Figure 2 This is a schematic diagram of the TD3 algorithm framework of the present invention;

[0065] Figure 3 This is a flowchart illustrating the implementation of the present invention;

[0066] Figure 4 This invention provides a comparison between the minimum transmission block length of the system using the optimization method described in this invention and the minimum transmission block length of the system using the benchmark scheme. , , , dBm, dBm, , , , , , , , , , , , , , , , , , . Detailed Implementation

[0067] The present invention will be further described below with reference to the accompanying drawings, both in terms of theory and specific implementation.

[0068] An optimization method for a TD3-based IRS-assisted short packet communication system in an interference environment, the specific steps of which are as follows:

[0069] Step 1: Construct an Intelligent Reflecting Surface (IRS) assisted short packet communication system. This system includes a Base Station (BS) with M antennas, a User Equipment (UE) with a single antenna, and a system with... The system has an IRS with one reflector unit; due to obstruction, the BS can only communicate with the UE with the assistance of the IRS; the system includes an interference source that affects the received signals of the IRS and the UE; it is assumed that the location, transmit power, and transmission channel characteristics from the interference source to the IRS and the UE are known; and a physical phase shift model of the realizable IRS is introduced to construct the amplitude and phase shift constraint relationship.

[0070] Step 2: Construct a system optimization model using Statistical Channel State Information (CSI) to maximize the average reachability rate of short-packet communication for the UE, ensuring good communication quality even in interference environments. The specific implementation is as follows:

[0071] In interference scenarios, the BS transmits a signal that reaches the UE via a reflection link, and the UE receives the signal as follows: ,in The transmit power at BS, For the power of the interference source, The channel coefficients between the BS and the IRS. The channel coefficients between the IRS and the UE. The channel coefficient between the interference source and the IRS. The channel coefficient between the interference source and the UE. Represents the field of complex numbers. It is the power normalization signal transmitted by the BS to the UE. It is a power-normalized interference signal emitted by the interference source. Follows the pattern with mean 0 and variance of Additive white Gaussian noise, This is a diagonal matrix formed by arranging the elements of the IRS reflection coefficient vector in sequence. The IRS reflection coefficient vector. The normalized beamforming vector of the BS antenna array satisfies , This is the conjugate transpose of the matrix. This represents the vector 2-norm. In interference scenarios, it represents the instantaneous received signal-to-interference-plus-noise ratio (SINR) of the UE. Represented as:

[0072]

[0073] in , , This indicates the modulo operation.

[0074] Short packet communication under a given transport block length L and instantaneous received signal-to-noise ratio Block Error Rate (BLER) The achievable rate of a system in an additive white Gaussian noise channel is expressed as:

[0075]

[0076] in Indicates Shannon capacity. Represents channel divergence at high signal-to-noise ratios. (M. Fang, J. Li, Y. Liu, Z. Chen and XY Zhang, "PerformanceAnalysis of Active Reconfigurable Intelligent Surface-Aided URLLC Systems," in IEEE Trans.Veh.Technol, vol. 73, no. 11, pp. 17842-17847, Nov. 2024), Gauss The inverse function of a function. Through joint optimization. and An optimization model for an IRS-assisted short packet communication system is constructed with the objective of maximizing the average reachable rate of the UE under interference conditions. The specific optimization problem is expressed as:

[0077]

[0078] in The maximum average achievable rate for UE short packet communication. This represents the mean operation. and ( ) represent the IRS number The amplitude coefficient and phase shift of each reflecting unit, , , These are constants related to the specific circuit implementation. According to the literature (Fang M, Li J, Liu Y, et al. Performance analysis of active reconfigurable intelligent surface-aided URLLC systems[J].IEEE Trans.Veh.Technol, 2024, 73(11):17842–17847.), we can obtain... for:

[0079]

[0080] Step 3: Employing the Twin-delayed Deep Deterministic Policy Gradient (TD3) algorithm, the BS active beamforming vector and IRS passive beamforming vector are jointly optimized. Key design elements of TD3 include the state space, action space, and reward function. The network model comprises an action network, an evaluation network, a target action network, and a target evaluation network. The action network outputs the BS active beamforming vector and the IRS passive beamforming vector, while the evaluation network outputs an action value function to assess the quality of actions performed in the current state. The network is trained by incorporating a dual-target network, a delayed update frequency strategy, a target smoothing strategy, an experience replay buffer, and a soft update method.

[0081] Step 4: By iteratively training the TD3 model, jointly optimize the BS active beamforming vector and the IRS passive beamforming vector, and use the optimized joint beamforming scheme to perform minimum system transport block length analysis.

[0082] The specific implementation method of step 1 is as follows:

[0083] Assuming that BS and IRS are both uniform linear arrays (ULA), then the array responses at BS and IRS are: ,in , It is the distance between the centers of adjacent reflective units. It's the wavelength. The corresponding departure angle or arrival angle of the channel; the channel coefficient between the BS and the IRS. Channel coefficients between IRS and UE Channel coefficient between interference source and IRS and the channel coefficient between the interference source and the UE They are respectively , , and ,in It is the path loss exponent corresponding to the channel. It is a unit reference distance Path loss at the location, The distance from the BS center to the IRS center. The distance from the IRS center to the UE. The distance from the interference source to the IRS center. The distance from the interference source to the UE. , and They are respectively , and The corresponding Rice factor, and the line-of-sight components of each channel are respectively , and ,in The angle at which the BS antenna emits signals. The angle of arrival of the BS transmitted signal to the IRS. The signal origination angle from the IRS to the UE. To interfere with the IRS's angle of arrival, the non-line-of-sight components of each channel , , and The elements in the set are independent and all follow a cyclic symmetric complex Gaussian random variable with mean 0 and variance 1.

[0084] IRS reflection coefficient vector ,definition For the IRS The reflection coefficient of each IRS reflection unit in the actual phase-shift model and The coupling relationship is .

[0085] The specific implementation of step 3 is as follows:

[0086] For the design of key elements of TD3, in order to fully consider the description of environmental information by the state space, it is designed according to CSI as follows:

[0087]

[0088] in , , , , This represents taking the argument of the complex elements in the matrix. The IRS phase shift and BS beamforming vectors are designed as the action space: The target action network adds noise explicitly. And trim the output range, that is Where clip is the clipping function.

[0089] In a statistical channel, the reward function can be expressed as:

[0090]

[0091] The training process of TD3 first initializes the action network. , For network parameters, Given the input state, the first evaluation network , For network parameters, the second evaluation network Target Action Network First target evaluation network and the second objective evaluation network Create an experience replay buffer; initialize the state at the start of training. In time step In the middle, the execution action is given. , and The amplitude of the IRS reflection coefficient and the BS active beamforming vector are calculated. The reward is calculated according to the reward function and the state is updated. At the same time, the experience replay pool collects state transition information. When the experience pool reaches its capacity limit, random sampling of sample data begins. Send it to the training network.

[0092] For the two evaluation networks, the loss function is calculated using the mean squared error:

[0093]

[0094]

[0095] The time difference deviation target is calculated as follows: The target Q-value of the target evaluation network group is calculated as follows:

[0096]

[0097] Update the corresponding evaluation networks using gradient descent.

[0098]

[0099] in and The learning rates are used to evaluate Network 1 and Network 2, respectively. This represents gradient computation. The action network updates using gradient ascent.

[0100]

[0101] in The learning rate is updated for the action network. The objective is to maximize the average Q value, i.e.

[0102]

[0103] The target network is updated via a soft update:

[0104]

[0105] in This is the soft update coefficient.

[0106] TD3 consists of six networks: an action network, a target action network, an evaluation network group consisting of two evaluation networks, and a target evaluation network group consisting of two target evaluation networks. Its networks can be divided into a main network class and a target network class. The main network class consists of the action network and the evaluation network group, while the target network class consists of the target action network and the target evaluation network group. The input layer, first hidden layer, second hidden layer, third hidden layer, and output layer neurons of the action network are as follows: 600, 400, 400 and The hidden layers are activated using the ReLU function, and the output layer uses the Tanh function to limit the range. The target evaluation network group has the same neural network structure, with the following neuron nodes for the input layer, first hidden layer, second hidden layer, third hidden layer, fourth hidden layer, and output layer: The input layer uses 800, 600, 400, 200, and 1 as input parameters, performs layer normalization on the data, and the hidden layer uses the ReLU function for activation; the target network class structure is consistent with the corresponding structure of the main network class.

[0107] The specific implementation of step 4 is as follows:

[0108] In IRS-assisted short packet communication systems, given a transmission block length Number of information bits (The achievable rate at the UE is) Instantaneous received signal-to-noise ratio The BLER under additive complex white Gaussian noise channel can be approximated as: Then the closed-form solution of the average BLER is:

[0109]

[0110] in For instantaneous SINR reception The probability density function, where n is the approximate order of the Gauss-Chebyshev integral. , , , , , , , , , , , , For instantaneous SINR reception The cumulative distribution function. To solve... The cumulative distribution function is first obtained by using the moment matching method. Approximate shape parameters are The scale parameter is The gamma distribution will Approximate shape parameters are The scale parameter is gamma distribution, , , , Finally, we obtained... The cumulative distribution function is expressed as:

[0111]

[0112] in , , , , , The Whittaker W-function, defined by the confluence hypergeometry function, is... , Represents the confluence hypergeometry function. This represents the gamma function.

[0113] Assumption The optimization algorithm for the minimum transport block length of the system, which is the average BLER threshold for UE communication service quality requirements, is as follows:

[0114] (1) Given system parameters , , , , , , , , , , , , and ;

[0115] (2) Initialize block length Let the boundary value be . , ;

[0116] (3) Calculate the block length respectively. and The average BLER corresponding to the time and ;

[0117] (4) If This indicates that when the block length is maximized, the average BLER threshold requirement cannot be met, which means there is no solution. ;

[0118] (5) If This indicates that the minimum block length satisfies the requirement for the average BLER threshold, thus yielding... ;

[0119] (6) If none of the above conditions are met, it means that the minimum transmission block length of the system is within the range of... The minimum transport block length of the system under optimal phase is numerically solved using the binary search method. ,make Calculate the corresponding average BLER , This represents the integer division operation;

[0120] (7) If satisfied Then let Otherwise ;

[0121] (8) Repeat steps (6) to (7) to continuously update. or until Stop iteration and output. ;

[0122] The specific implementation process of this invention is as follows:

[0123] An optimization method for a TD3-based IRS-assisted short packet communication system in an interference environment, the specific implementation steps of which are as follows:

[0124] Step 1: Construct an Intelligent Reflecting Surface (IRS) assisted short packet communication system. This system includes a Base Station (BS) with M=4 antennas, a User Equipment (UE) with a single antenna, and a system with... The system has an IRS with one reflector unit; due to obstruction, the BS can only communicate with the UE with the assistance of the IRS; the system includes an interference source that affects the received signals of the IRS and the UE; it is assumed that the location, transmit power, and transmission channel characteristics from the interference source to the IRS and the UE are known; and a physically realizable phase shift model of the IRS is introduced to construct amplitude and phase shift constraint relationships; assuming that both the BS and the IRS are uniform linear arrays (ULA), the array responses at the BS and the IRS are... ,in , It is the distance between the centers of adjacent reflective units. It's the wavelength. , The corresponding departure angle or arrival angle of the channel; the channel coefficient between the BS and the IRS. Channel coefficients between IRS and UE Channel coefficients between the interference source and the IRS and the channel coefficient between the interference source and the UE They are respectively , , and ,in It is the path loss exponent corresponding to the channel. It is a unit reference distance Path loss at a distance of meters. Meters represent the distance from the BS center to the IRS center. The meter represents the distance from the IRS center to the UE. The meter represents the distance from the interference source to the IRS center. The distance from the interference source to the UE. , and They are respectively , and The corresponding Rice factor, The line-of-sight components of each channel are as follows: , and ,in The angle at which the BS antenna emits signals. The angle of arrival of the BS transmitted signal to the IRS. The signal origination angle from the IRS to the UE. To interfere with the IRS's angle of arrival, the non-line-of-sight components of each channel , , and The elements in the set are independent and all follow a cyclic symmetric complex Gaussian random variable with mean 0 and variance 1.

[0125] IRS reflection coefficient vector ,definition For the IRS The reflection coefficient of each IRS reflection unit in the actual phase-shift model and The coupling relationship is , , , .

[0126] Step 2: Utilize statistical CSI to construct a system optimization model aimed at maximizing the maximum average reachable rate of short packet communication UEs, so as to ensure that UEs can still obtain good communication quality in the presence of interference.

[0127] Step 3: Employing the Twin-delayed Deep Deterministic Policy Gradient (TD3) algorithm, the BS active beamforming vector and IRS passive beamforming vector are jointly optimized. Key design elements of TD3 include the state space, action space, and reward function. The network model comprises an action network, an evaluation network, a target action network, and a target evaluation network. The action network outputs the BS active beamforming vector and the IRS passive beamforming vector, while the evaluation network outputs an action value function to assess the quality of actions performed in the current state. The network is trained by incorporating a dual-target network, a delayed update frequency strategy, a target smoothing strategy, an experience replay buffer, and a soft update method.

[0128] In the proposed TD3 framework, the training rounds are set to 300, with 200 training steps per round. The experience backtracking buffer capacity is set to 10,000. The number of neurons in each layer of the action network and evaluation network in TD3 has been given above. When exploring actions, the target network uses normal random noise with a mean of 0.2 and a variance of 1. The noise is pruned in the range of -0.5 to 0.5. The action network is updated every 3 times, and the evaluation network updates the action network once. The optimizer used is Adam.

[0129] Step 4: By iteratively training the TD3 model, jointly optimize the BS active beamforming vector and the IRS passive beamforming vector, and use the optimized joint beamforming scheme to perform the minimum transmission block length analysis of the system.

[0130] Table 1 presents the optimized IRS reflection coefficient matrix. First, the optimal reflection phase shift of the IRS cell is obtained through training. Then, the reflection amplitude is calculated using an actual phase shift model, leading to the optimized IRS reflection coefficient matrix. The reflection coefficient vector of the IRS The elements in the table; Table 2 shows the optimized beamforming vector at BS; Figure 4 This invention compares the minimum transmission block length of the system under the optimization method described in this invention with that under the benchmark scheme. The benchmark scheme's IRS phase shift design follows the optimal continuous phase adjustment strategy under the ideal phase shift model. The beamforming vector at BS is obtained based on the statistical CSI maximum ratio transmission. (Y. Jia, C. Ye and Y. Cui, “Analysis and Optimization of an Intelligent Reflecting Surface-Assisted System With Interference,” IEEE Trans. Wireless Commun, vol. 19, no. 12, pp. 8068-8082, Dec. 2020). Assuming the number of information bits... Interference source power dBm, in the target average BLER Under the given conditions, through the system minimum transmission block length optimization algorithm, it can be seen that the system minimum transmission block length under the optimization method described in this invention is superior to the benchmark scheme at different transmit powers, for example, when the transmit power is... At dBm, the minimum transmission block length of the system using the optimization method described in this invention is When using the benchmark scheme The comparative results show that the BS active beamforming and IRS passive beamforming optimized using the method described in this invention can significantly improve the performance of the IRS-assisted short packet communication system under interference conditions.

[0131] Table 1

[0132]

[0133] Table 2

[0134]

Claims

1. An optimization method for an IRS-assisted short packet communication system based on dual-delay deep deterministic policy gradient TD3 in an interference environment, characterized in that, Includes the following steps: Step 1: Construct a smart reflector IRS-assisted short packet communication system. This system includes a base station (BS) with M antennas, a single-antenna user equipment (UE), and a system with... The system has an IRS with one reflector unit; due to obstruction, the BS can only communicate with the UE with the assistance of the IRS; the system includes an interference source that affects the received signals of the IRS and the UE; it is assumed that the location, transmit power, and transmission channel characteristics from the interference source to the IRS and the UE are known; and a physical phase shift model of the realizable IRS is introduced to construct the amplitude and phase shift constraint relationship. Step 2: Construct a system optimization model using Statistical Channel State Information (CSI) to maximize the average reachability rate of short-packet communication UEs, ensuring that UEs can still achieve good communication quality in interference environments; the specific implementation is as follows: In interference scenarios, the BS transmits a signal that reaches the UE via a reflection link, and the UE receives the signal as follows: ,in The transmit power at BS, For the power of the interference source, The channel coefficients between the BS and the IRS. The channel coefficients between the IRS and the UE. The channel coefficient between the interference source and the IRS. The channel coefficient between the interference source and the UE. Represents the field of complex numbers. It is the power normalization signal transmitted by the BS to the UE. It is a power-normalized interference signal emitted by the interference source. Follows the pattern with mean 0 and variance of Additive white Gaussian noise, This is a diagonal matrix formed by arranging the elements of the IRS reflection coefficient vector in sequence. The IRS reflection coefficient vector. The normalized beamforming vector for the BS antenna array, where, Let A and B be the amplitude coefficients and phase shift coefficients of the BS antenna array factor, respectively, satisfying... , This is the conjugate transpose of the matrix. Represents the vector 2-norm; in interference scenarios, the instantaneous received signal-to-interference-plus-noise ratio (SINR) of the UE. Represented as: in, , , This indicates the modulo operation; Short packet communication under a given transport block length L and instantaneous received signal-to-noise ratio Block error rate BLER The achievable rate of a system in an additive white Gaussian noise channel is expressed as: in Indicates Shannon capacity. Represents channel divergence at high signal-to-noise ratios. , Gauss Inverse function of function; through joint optimization and An optimization model for an IRS-assisted short packet communication system is constructed with the objective of maximizing the average reachable rate of the UE under interference conditions. The specific optimization problem is expressed as: in The maximum average achievable rate for UE short packet communication. This represents the mean operation. and , They represent the IRS number The amplitude coefficient and phase shift of each reflecting unit, , , These are constants related to the specific circuit implementation; according to existing technology, they can be obtained... for: Step 3: The dual-delay deep deterministic strategy gradient TD3 algorithm is used to jointly optimize the BS active beamforming vector and the IRS passive beamforming vector. The key elements of TD3 include the design of the state space, action space, and reward function. The network model includes an action network, an evaluation network, a target action network, and a target evaluation network. The action network outputs the BS active beamforming vector and the IRS passive beamforming vector, and the evaluation network outputs the action value function to evaluate the quality of the action performed in the current state. The network is trained by introducing a dual target network, a delayed update frequency strategy, a target smoothing strategy, an experience replay buffer, and a soft update method. Step 4: By iteratively training the TD3 model, jointly optimize the BS active beamforming vector and the IRS passive beamforming vector, and use the optimized joint beamforming scheme to perform the minimum transmission block length analysis of the system.

2. The optimization method for an IRS-assisted short packet communication system based on TD3 in an interference environment according to claim 1, the specific implementation method of step 1 is as follows: Assuming that BS and IRS are both uniform linear arrays ULA, then the array responses at BS and IRS are: ,in , It is the distance between the centers of adjacent reflective units. It's the wavelength. The corresponding departure angle or arrival angle of the channel; the channel coefficient between the BS and the IRS. Channel coefficients between IRS and UE Channel coefficient between interference source and IRS and the channel coefficient between the interference source and the UE They are respectively , , and ,in It is the path loss exponent corresponding to the channel. It is a unit reference distance Path loss at the location, The distance from the BS center to the IRS center. The distance from the IRS center to the UE. The distance from the interference source to the IRS center. The distance from the interference source to the UE. , and They are respectively , and The corresponding Rice factor, and the line-of-sight components of each channel are respectively , and ,in The angle at which the BS antenna emits signals. The angle of arrival of the BS transmitted signal to the IRS. The signal origination angle from the IRS to the UE. To interfere with the IRS's angle of arrival, the non-line-of-sight components of each channel , , and The elements in the set are independent and all follow a cyclic symmetric complex Gaussian random variable with mean 0 and variance 1; IRS reflection coefficient vector ,definition For the IRS The reflection coefficient of each IRS reflection unit in the actual phase-shift model and The coupling relationship is .

3. The optimization method for an IRS-assisted short packet communication system based on TD3 in an interference environment according to claim 1, wherein step 3 is specifically implemented as follows: For the design of key elements of TD3, in order to fully consider the description of environmental information by the state space, it is designed according to CSI as follows: in , , , , This indicates taking the argument of the complex elements in the matrix; the IRS phase shift and BS beamforming vectors are designed as the action space: ; Target action networks by explicitly adding noise And trim the output range, that is Where clip is the clipping function; In a statistical channel, the reward function can be expressed as: The training process of TD3 first initializes the action network. The training process of TD3 first initializes the action network. , For network parameters, Given the input state, the first evaluation network , The network parameters for the first evaluation network, and the network parameters for the second evaluation network. Target Action Network First target evaluation network and the second objective evaluation network Create an experience replay buffer; Initialize the state at the start of training. In time step In the middle, the execution action is given. , and The system calculates the amplitude of the IRS reflection coefficient and the BS beamforming vector, calculates the reward based on the reward function and updates the state, while the experience replay pool collects state transition information. When the experience pool reaches its capacity limit, it begins to randomly sample data. Send to the training network; For the two evaluation networks, the loss function is calculated using the mean squared error: The time difference deviation target is calculated as follows: The target Q-value of the target evaluation network group is calculated as follows: Update the corresponding evaluation networks using gradient descent. in and The learning rates are used to evaluate Network 1 and Network 2, respectively. This represents gradient operation; The action network is updated using gradient ascent. in The learning rate updated for the action network; The objective is to maximize the average Q value, i.e. The target network is updated via a soft update: in This is the soft update coefficient.

4. The optimization method for an IRS-assisted short packet communication system based on TD3 in an interference environment according to claim 1, characterized in that, TD3 consists of six networks: an action network, a target action network, a set of evaluation networks consisting of two evaluation networks, and a set of target evaluation networks consisting of two target evaluation networks. The networks can be divided into a main network class and a target network class. The main network class consists of the action network and the evaluation network, while the target network class consists of the target action network and the target evaluation network. The input layer, first hidden layer, second hidden layer, third hidden layer, and output layer neurons of the action network are as follows: 600, 400, 400 and The hidden layers are activated using the ReLU function, and the output layer is limited by the Tanh function. The target evaluation network group has the same neural network structure, with the following neuron nodes for the input layer, first hidden layer, second hidden layer, third hidden layer, fourth hidden layer, and output layer: The input layer uses 800, 600, 400, 200, and 1 as input parameters, performs layer normalization on the data, and the hidden layer uses the ReLU function for activation; the target network class structure is consistent with the corresponding structure of the main network class.

5. The optimization method for an IRS-assisted short packet communication system based on TD3 in an interference environment according to claim 1, wherein step 4 is specifically implemented as follows: In IRS-assisted short packet communication systems, given a transmission block length Number of information bits The achievable rate at the UE is Instantaneous received signal-to-noise ratio The BLER under additive complex white Gaussian noise channel can be approximated as: Then the closed-form solution of the average BLER is: in For instantaneous SINR reception The probability density function, where n is the approximate order of the Gauss-Chebyshev integral. , , , , , , , ,in , , , The interval partitioning parameter in the Gauss-Chebyshev integral. For instantaneous SINR reception The cumulative distribution function; in order to solve The cumulative distribution function is first obtained by using the moment matching method. Approximate shape parameters are The scale parameter is The gamma distribution will Approximate shape parameters are The scale parameter is gamma distribution, , , , Finally, we obtained The cumulative distribution function is expressed as: in , , , , , The Whittaker W-function, defined by the confluence hypergeometry function, is... , Represents the confluence hypergeometry function. Represents the gamma function; Assumption The optimization algorithm for the minimum transport block length of the system, which is the average BLER threshold for UE communication service quality requirements, is as follows: (1) Given system parameters , , , , , , , , , , , Path loss index and average BLER threshold ; (2) Initialize block length Let the boundary value be . , ; (3) Calculate the block length respectively. and The average BLER corresponding to the time and ; (4) If This indicates that when the block length is maximized, the average BLER threshold requirement cannot be met, which means there is no solution. ; (5) If This indicates that the minimum block length satisfies the requirement for the average BLER threshold, thus yielding... ; (6) If none of the above conditions are met, it means that the minimum transmission block length of the system is within the range of... The minimum transport block length of the system under optimal phase is numerically solved using the binary search method. ,make Calculate the corresponding average BLER , This represents the integer division operation; (7) If satisfied Then let Otherwise ; (8) Repeat steps (6) to (7) to continuously update. or until Stop iteration and output. .

Images