Resource and phase shift optimization method in RIS-assisted vehicle communication based on SAC

Abstract

The invention discloses a resource and phase shift optimization method in RIS auxiliary vehicle communication based on SAC, and belongs to the technical field of intelligent reflector auxiliary vehicle communication. An RIS-assisted vehicle communication network is provided, a C-V2X communication mode is considered, M cellular vehicle users, V2I communication with a BS and K pairs of device-device vehicle users are provided, and the M cellular vehicle users, the K pairs of device-device vehicle users directly communicate with one another in a V2V mode. In addition, the requirement of vehicle communication for timeliness of data is high, and information age serves as a criterion for measuring information freshness and is very important in the aspect of vehicle communication. Therefore, the optimization problem is constructed into an MDP problem by taking minimization of the information age of the V2I link and preferential transmission of the effective load of the V2V link as targets, the BS serves as an agent, allocates resources and controls phase shift for vehicles, adopts an SAC algorithm, and gradually converges and keeps high stability, so that the system makes the optimal strategy selection and improves the system performance.

Description

Technical Field

[0001] The invention belongs to the technical field of intelligent reflective surface assisted vehicle communication, and in particular relates to a resource and phase shift optimization method in RIS assisted vehicle communication based on SAC. Background Art

[0002] In recent years, driven by the continuous progress of science and technology, the momentum of new energy vehicles has been advancing by leaps and bounds, and vehicles have undergone a major transformation towards intelligence. This has led to a higher demand for vehicle communication technology, prompting many institutions to conduct in-depth research to meet different needs and improve communication quality. In-vehicle communications include V2V, V2P, V2I, and V2N, collectively referred to as V2X (Vehicle-to-Everything), which has become the focus of exploration. The following three main technical paths can be used to achieve V2X communication: millimeter wave (mmwave) communication, dedicated short-range communication (DSRC), and cellular-V2X (C-V2X). Compared with mmwave and DSRC, C-V2X utilizes cellular networks, can be closely integrated with existing mobile communication networks, reduces deployment costs and complexity, has a large coverage area, and can provide higher bandwidth.

[0003] Recently, the 3rd Generation Partnership Project (3GPP) has been committed to the standardization of Long Term Evolution of V2X (LTE-V2X) and 5G New Radio (NR-V2X), and has proposed many communication use cases in different scenarios, such as high-speed communication between vehicles and base stations to realize various entertainment applications and functions, such as road traffic congestion, dynamic map updates, and secondly, the exchange of safety information between vehicles, including the location, speed and acceleration of vehicles, which requires high reliability of vehicle communication. An effective resource allocation scheme can increase the vehicle communication rate and ensure the reliable transmission of vehicle safety information. In addition, the high mobility of vehicles also affects vehicle communication, so it becomes challenging to implement an efficient resource allocation scheme. Fortunately, the continuous development of artificial intelligence has spawned DRL, which integrates deep learning and reinforcement learning. It turns out that DRL is a more promising approach to achieve autonomous decision-making and vehicle control. DRL formulates the resource optimization problem as a Markov decision process (MDP). Deep neural networks (DNNs) can effectively process information such as vehicle channel states and learn the optimal resource allocation strategy through reinforcement learning algorithms. Therefore, more and more studies adopt the MDP framework to simulate resource allocation in the context of vehicles.

[0004] V2X provides a more efficient and safer traffic experience through real-time information exchange and sharing, and supports the realization of intelligent transportation systems using autonomous driving technology. Therefore, real-time information is crucial in V2X communication, which can be measured by the age of information (AoI). AoI refers to the time that has passed since the last received information was generated. Compared with latency, AoI also plays a key role in V2X communication to ensure timely exchange of information. In vehicle networks, AoI is related to the information transmission rate of the vehicle. A reasonable resource allocation scheme can effectively increase the transmission rate of the vehicle, thereby reducing the vehicle's AoI and improving the safety of the vehicle.

[0005] In order to further improve the information transmission rate and information freshness of the vehicle and realize real-time and reliable communication, we will assist vehicle communication through RIS (intelligent metasurface). Generally speaking, RIS is considered to be an adaptive (intelligent) thin composite material similar to wallpaper, which can cover walls, buildings, ceilings and other parts with high flexibility. RIS is a passive array structure that can adjust the phase of each reflective element on the surface almost continuously or discretely, and has low power consumption. RIS is able to modify radio waves and can send the received signal to a designated receiver through external programming and control, thereby enhancing the power of the signal, suppressing interference and expanding the communication range. Summary of the invention

[0006] In view of the problems in the background technology, the present invention develops a resource and phase shift optimization method in RIS-assisted vehicle communication based on SAC. The present invention studies the resource optimization and phase shift matrix design problems of RIS-assisted vehicle communication in a high-mobility vehicle environment based on deep reinforcement learning Soft actor-critic, with the goal of minimizing the sum of AoIs of all V2I links and improving the payload transmission success rate of V2V links, so that the system can make the best strategy selection and improve system performance.

[0007] To achieve the above object, the present invention provides the following technical solutions:

[0008] A method for optimizing resources and phase shift in RIS-assisted vehicle communication based on SAC, specifically comprising:

[0009] Step 1: System model setting; the system model setting includes channel model setting, AoI model setting of CUE, and load transmission probability model setting of DUE;

[0010] Step 2: The optimal channel allocation, power allocation and RIS phase shift are obtained by training the model, and then the achievable capacity, AoI and V2V load transmission success rate of the V2I link are calculated;

[0011] Step 3: Conduct numerical and simulation evaluations.

[0012] Preferably, in step 1, the specific steps of setting the system model are:

[0013] (1) Channel model setting: Assume that the coordinates of the base station BS and the intelligent reflection surface RIS are (x BS ,y BS ,z BS ) and (x RIS ,y RIS ,z RIS ), at time slot n, the coordinates of the transmitter in the kth D2D pair are The coordinates of the receiver are The coordinates of the mth CUE are The set of available channels is defined as L = {1, 2, ···, L}. At time slot n, the channel gain of the kth V2V link occupying the lth subchannel is

[0014]

[0015] where ρ is the path loss constant for the reference distance d0 = 1 m, β is the logarithmic shadow fading random variable, is the geometric distance between the transmitter and the receiver in the kth V2V pair at time slot n, and α is the corresponding path loss exponent; is the frequency-dependent small-scale fading power component, which is assumed to be exponentially distributed with a unit mean. At time slot n, both are communicating on the lth channel, and the channel coefficient from the mth CUE to the BS is Channel coefficients from the mth CUE to the receiver in the kth V2V pair Channel coefficients from the k'th V2V transmitter to the kth V2V receiver and the channel coefficient from the kth V2V transmitter to the BS are similar to the above definitions; in addition, the channel gain from the mth CUE to the RIS is defined as

[0016]

[0017] The meaning of is the geometric distance from the mth CUE to the RIS at time slot n, λ[l] is the carrier length of the lth subchannel, π and j are basic mathematical symbols, and e is a complex function;

[0018] The arrival and departure array responses of the RIS can be defined as

[0019]

[0020] Here d is the spacing between each element in RIS, is the angle of arrival (AoA) of a signal at time slot n. In the same way, we can calculate the channel gain from RIS to BS at time slot n as Channel gain from the transmitter to the RIS in the kth V2V pair and the channel gain from RIS to the receiver in the kth V2V pair express it;

[0021] The received signal-to-interference-plus-noise ratio (SINR) of the mth V2I link and the kth V2V link on the lth subchannel is expressed as

[0022]

[0023] in represents the V2I link transmit power, represents the V2V link transmission power at time slot n, H represents the conjugate transposed matrix, σ 2 is the white noise power, and the diagonal phase shift matrix of RIS is F represents the number of elements in RIS. As mentioned above, f is used to represent the fth element in F. Due to hardware limitations, the phase shift can only be selected from a limited set of discrete values, where and β f ∈[0,1] is used as the relevant transmission coefficient and the amplitude of the phase shift, where Q is a variable that controls the discreteness of the phase shift and

[0024]

[0025] Then, the uplink transmission rate of the mth CUE and the uplink transmission rate of the kth V2V pair are:

[0026]

[0027] (2) CUE AoI model settings:

[0028] V2I communication must exchange information with the BS in a timely manner, and Must be strictly greater than or equal to R th , which will reset the AoI to Δn, where R th represents the minimum rate required for V2I communication. Less than R th The AoI of V2I will gradually increase when . Here, the AoI of the mth V2I link is defined as represents the time since the last successful communication; therefore, the AoI changes through the following equation

[0029]

[0030] (3) Load transmission probability model setting of DUE:

[0031] In order to ensure the reliability of V2V transmission security information, the probability of V2V transmission successfully delivering a data packet of size D within the time budget N is modeled as

[0032]

[0033] Where B is the bandwidth of the spectrum subband, D represents the size of the periodically generated V2V payload, and Δn represents the channel coherence time.

[0034] Preferably, in step 2, the specific steps of optimizing channel allocation, power allocation and RIS phase shift are:

[0035]

[0036] Constraint C2 imposes restrictions on the transmission power range within a V2V pair, constraints C3 and C4 assume that each V2V pair occupies only one channel, constraint C5 limits the range of RIS phase shift, and optimization objective P1 involves vehicle resource allocation and RIS phase change control based on AoI perception;

[0037] BS centrally handles vehicle channel sharing, power allocation and RIS phase shift matrix control. At each time slot n, the agent observes the current state s n , and then take action a according to the current policy n After the action is completed, the agent will receive the relevant reward r from the system environment n , then, the state transitions to the next state s n+1 , marking the completion of the cycle of this dynamic process; the entire sequence is represented by e n =(s n ,a n ,r n ,s n+1 ), which encompasses the important components of the DRL framework for solving optimization problems.

[0038] Furthermore, the specific optimization algorithm of step 2 is: construct a DRL framework including state, action and reward, and then use the SAC algorithm to solve the framework to obtain the optimal strategy; specifically:

[0039] (1) State space: At time slot n, the system state consists of the following components: V2I link local channel V2V link local channel Interference to the receiver from the previous time slot in a V2V link The load left for transmission in the V2V link V2I Link And the phase shift matrix Θ of RIS n ; Therefore, the system state at time slot n is given by the following formula

[0040]

[0041] (2) Action Space: As mentioned above, the BS acts as an agent to determine the channel allocation, power control, and RIS phase shift matrix of the vehicle. represents the channel and power selection of a V2V vehicle user pair, Θ n represents the phase shift selection of RIS, so the action space of time slot n is defined as

[0042]

[0043] (3) Reward function: The objective has two main parts: one is to maintain the connection with the BS and keep the AoI at a minimum level, and the other is to increase the success probability of V2V transmission payload; therefore, the reward for time slot n is set to

[0044]

[0045] Among them, λ1 and λ2 represent the weight coefficients used to balance the rewards, which are composed of two parts. The first part represents the reward obtained by the relevant AoI in V2I transmission, and the second part represents the reward obtained by the remaining payload transmission amount in V2V transmission;

[0046] (4) SAC training phase: Let φ be the parameters of the actor network, θ1 and θ2 be the parameters of the two critic networks, and The parameters of the two target critic networks; at each training step n, the reinforcement learning goal is to maximize both the expected reward and entropy, defined as

[0047]

[0048] Here γ represents the discount factor, H(·) represents the impact on future rewards, and α is a measure of entropy. The importance of the entropy term is controlled and is called the temperature parameter. The goal of RL is to discover the optimal strategy.

[0049] π * = arg max π J(π(a n |s n )). (16)

[0050] During the training process, the experience generated by the training is saved in the buffer [e1,e2,···,e n,···], when the size of the experience in the buffer is greater than I, I tuples are randomly selected from the replay buffer to form a mini-batch of training data, and let (s i ,a i ,r i ,s i+1 ) becomes the i-th tuple in the mini-batch, and then s i Input the actor network and get a i,new and logπ θ (a i,new |s i ), then the loss function gradient of the temperature parameter α can be obtained as

[0051]

[0052] Where H0 = dim(a n ), then s i 、a i Input two critic networks and output action-value function Q θ1 (s i ,a i ) and Q θ2 (s i ,a i ), then s i+1 Input actor network and output a i+1 and logπ θ (a i+1 |s i+1 ); Finally, s i+1 and a i+1 Input two target critic networks and output and Then the target value can be calculated as follows

[0053]

[0054] The gradient of the loss function for parameters θ1 and θ2 is as follows

[0055] ▽ θ1 J i (θ1)=▽ θ1 Q θ1 (a i ,s i )(Q θ1 (s i ,a i )-r i +γQ tar (s i+1 ,a i+1 )), (19)

[0056] ▽ θ2 J i (θ2)=▽ θ2 Q θ2 (a i ,s i )(Q θ2 (s i ,a i )-r i +γQ tar (s i+1 ,a i+1 )). (20)

[0057] The algorithm then calculates the gradient of the φ loss function by first transforming s i and a i,new Input two critic networks and get Q θ1 (s i ,a i,new ) and Q θ2 (s i ,a i,new ), and then take the minimum value of the two to get

[0058] Q(s i ,a i,new )=min{Q θ1 (s i ,a i,new ),Q θ2 (s i ,a i,new )}, (twenty one)

[0059] The gradient calculation formula of the φ loss function is

[0060]

[0061] where ξ is a noise sampled from a multivariate normal distribution, and f φ (ξ i ;s i ) is action a i,new reparameterized function; in addition, the Adam optimizer is used to update α, θ1, θ2, and φ based on the gradient of the above loss function. Finally, the two parameters of the targetcritic network Updated to

[0062]

[0063] Among them, τ1 and τ2 are constants that satisfy the following conditions: τ1<<1 and τ2<<1; after all parameters are updated, the algorithm will continue to the next training. When all training is completed, the algorithm will obtain the optimal strategy.

[0064] Compared with the prior art, the present invention has the following beneficial effects:

[0065] The present invention proposes a RIS-assisted vehicle communication network, considering the C-V2X communication mode, with M cellular vehicle users (CUEs), which use V2I communication with the BS, and K pairs of device-to-device vehicle users (DUEs), which communicate directly with each other through the V2V mode. In addition, vehicle communication has high requirements on the timeliness of data, and information age, as a criterion for measuring the freshness of information, is very important in vehicle communication. Therefore, the present invention aims to minimize the information age of the V2I link and prioritize the transmission of the payload of the V2V link, and constructs this optimization problem into an MDP problem, in which the BS acts as an agent to allocate resources and control phase shifts for vehicles, and adopts the SAC algorithm, which gradually converges and maintains high stability. The simulation results show that the convergence speed, cumulative reward, AoI performance, and payload transmission probability of the method of the present invention are better than PPO, DDPG, TD3, and related random algorithms. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] Figure 1 Comparison of rewards between the solution of the present invention and PPO, DDPG, TD3, and random strategies within the training segment.

[0067] Figure 2 This is a comparison chart of the sum of V2I link information transmission rates between the solution of the present invention and PPO, DDPG, TD3, and random strategies at different V2I powers.

[0068] Figure 3 This is a comparison chart of the sum of AoI of the V2I link between the solution of the present invention and PPO, DDPG, TD3, and random strategies at different V2I powers.

[0069] Figure 4 This is a comparison chart of the average effective load transmission success probability of the V2V link under different V2I powers between the solution of the present invention and PPO, DDPG, TD3, and random strategies.

[0070] Figure 5 This is a comparison chart of the sum of V2I link information transmission rates between the solution of the present invention and PPO, DDPG, TD3, and random strategies under different V2V effective transmission loads.

[0071] Figure 6 This is a comparison chart of the sum of AoI of the V2I link between the solution of the present invention and PPO, DDPG, TD3, and random strategies under different V2V effective transmission loads.

[0072] Figure 7 This is a comparison chart of the sum of V2I link information transmission rates between the solution of the present invention and PPO, DDPG, TD3, and random strategies under different numbers of RIS elements.

[0073] Figure 8 This is a comparison chart of the sum of AoI of the V2I link between the solution of the present invention and PPO, DDPG, TD3, and random strategies under different numbers of RIS elements.

[0074] Fig. 9 This is a comparison chart of the average information transmission rate of a single user V2I link under different numbers of RIS elements and different numbers of users in the solution of the present invention. DETAILED DESCRIPTION

[0075] In order to further explain the technical means and effects adopted by the present invention to achieve the predetermined invention purpose, the specific implementation mode, structure, characteristics and effects of the present invention are described in detail below in combination with the accompanying drawings and preferred embodiments.

[0076] Step (1): The total training time is divided into N equal time slots, each time slot has an interval of Δn, and the N time slots are represented by the set N = {1,···,n,···,N}. Considering the mobility of the vehicle, the channel state is time-varying. It is assumed that the channel state information (CSI), the distance between the vehicle and the BS, the vehicle and the vehicle, and the vehicle and the RIS do not change in a time slot, but will be different in different time slots. In the RIS-assisted vehicle communication network, the present invention considers the C-V2X communication mode, M cellular vehicle users (CUEs), and V2I communication with the BS (considering the uplink SIMO scenario, the vehicle is equipped with an antenna and the BS is equipped with an M-element uniform linear array), K pairs of device-to-device communication vehicle users (DUEs), which communicate directly with each other through V2V links. It is worth noting that each vehicle in the system plays the role of a cellular user and a vehicle user. The sets of CUEs and DUEs are defined as M = {1, 2, ···, M} and K = {1, 2, ···, K}, respectively. Assuming that M V2I links have been pre-allocated with orthogonal spectrum sub-bands with fixed transmission power, each DUE needs to reuse the spectrum of a CUE, using x m,k Indicates the spectrum sharing situation, x m,k =1 means that the kth DUE reuses the spectrum of the mth CUE. Different types of V2X links will interfere with each other. In order to improve the service quality of the communication link, RIS is deployed to enhance the performance of the communication system, where RIS is a uniform rectangular array composed of F passive elements. The goal of the present invention is to minimize the information age of V2I and maximize the transmission success rate of V2V payload.

[0077] Step (2): Assume that the coordinates of BS and RIS are (x BS ,y BS,z BS ) and (x RIS ,y RIS ,z RIS ), at time slot n, the coordinates of the transmitter in the kth D2D pair are D t is the transmitter, and the coordinates of the receiver are D r is the receiver, and the coordinates of the mth CUE are In addition, the set of L available channels is defined as L = {1, 2, ···, L}. At time slot n, the channel gain of the kth V2V link occupying the lth subchannel is

[0078]

[0079] where ρ is the path loss constant for the reference distance d0 = 1 m, β is the logarithmic shadow fading random variable, is the geometric distance between the transmitter and the receiver in the kth V2V pair at time slot n, and α is the corresponding path loss exponent. is the frequency-dependent small-scale fading power component, which is assumed to be exponentially distributed with a unit mean. At time slot n, both communicate on the lth channel, and the channel coefficient from the mth CUE to the BS is Channel coefficients from the mth CUE to the receiver in the kth V2V pair Channel coefficients from the k'th V2V transmitter to the kth V2V receiver and the channel coefficient from the kth V2V transmitter to the BS All have similar definitions to the above. In addition, the channel gain from the mth CUE to the RIS is defined as

[0080]

[0081] in, is the geometric distance from the mth CUE to the RIS at time slot n, λ[l] is the carrier length of the lth subchannel, π and j are basic mathematical symbols, and e is a complex function.

[0082] The arrival and departure array responses of the RIS can be defined as

[0083]

[0084] Here d is the spacing between each element in RIS, is the angle of arrival (AoA) of a signal at time slot n. In the same way, the channel gain from RIS to BS at time slot n can be Channel gain from the transmitter to the RIS in the kth V2V pair and the channel gain from RIS to the receiver in the kth V2V pair Express it.

[0085] Step (3): After obtaining the relevant channel gain, we can get the corresponding information transmission rate of the vehicle. First, the received signal to interference and noise ratio (SINR) of the mth V2I link and the kth V2V link on the lth subchannel is expressed as

[0086]

[0087] in represents the V2I link transmit power, represents the V2V link transmission power at time slot n, H is the conjugate transposed matrix, is the interference signal, σ 2 is the white noise power, and the diagonal phase shift matrix of RIS is F represents the number of elements in RIS, and f is used to represent the fth element among F elements. Due to hardware limitations, the phase shift can only be selected from a limited set of discrete values, where and β f ∈[0,1] is used as the relevant transmission coefficient and the amplitude of the phase shift, where Q is a variable that controls the discreteness of the phase shift and the interference signal

[0088]

[0089] Then, the uplink transmission rate of the mth CUE and the uplink transmission rate of the kth V2V pair are:

[0090]

[0091] Step (4): Next, establish the system's AoI model and payload transmission requirements. V2I communication must exchange information with the BS in a timely manner, and Must be strictly greater than or equal to R th , which will reset the AoI to Δn, where R th represents the minimum rate required for V2I communication. Less than R th The AoI of V2I will gradually increase when . Here, we define the AoI of the mth V2I link as represents the time since the last successful communication. Therefore, AoI changes through the following equation

[0092]

[0093] At the same time, in order to ensure the reliability of V2V transmission security information, we model the probability of V2V transmission successfully delivering a data packet of size D within the time budget N as

[0094]

[0095] Where B is the bandwidth of the spectrum sub-band, D represents the size of the periodically generated V2V payload, and Δn represents the channel coherence time.

[0096] Step (5): In order to improve the relevant performance of the communication system, the present invention studies a framework for optimizing channel resource allocation, power control and RIS phase shift matrix based on the above model, and expresses it as an optimization problem, whose goal is to minimize the AoI of the V2I link and maximize the transmission success rate of the V2V payload. Therefore, this problem can be regarded as

[0097]

[0098] is the AoI in V2I. Constraint C2 imposes a limit on the transmission power range within the V2V pair, and the maximum power is Constraints C3 and C4 assume that each V2V pair occupies only one channel. In addition, constraint C5 also limits the range of the RIS phase shift, which is designed to be a discrete phase shift due to hardware limitations. The objective function is a non-convex function, which is a challenge for traditional optimization algorithms because they have difficulty finding a solution quickly. Therefore, the next step will explore the application of reinforcement learning methods to handle and solve this non-convex problem.

[0099] Step (6): The optimization target P1 involves vehicle resource allocation and RIS phase shift control based on AoI perception. To solve this problem, the present invention expresses it as a DRL process. In this method, the BS centrally handles vehicle channel sharing, power allocation, and RIS phase shift matrix control. At each time slot n, the agent observes the current state s n , and then take action a according to the current policy n After the action is completed, the agent will receive the relevant reward r from the system environment n , then, the state transitions to the next state s n+1 , marking the completion of the cycle of this dynamic process. The entire sequence can be expressed as e n =(s n ,a n ,r n ,s n+1 ), which encompasses the important components of the DRL framework for solving optimization problems.

[0100] Now, we first build a DRL framework that includes states, actions, and rewards, and then use the SAC algorithm to solve the framework to obtain the optimal strategy. The relevant details are as follows:

[0101] State space: At time slot n, we can get the state of the system consisting of the following parts: V2I link local channel V2V link local channel Interference to the receiver from the previous time slot in a V2V link The load left for transmission in the V2V link V2I Link And the phase shift matrix Θ of RIS n Therefore, the system state at time slot n is given by

[0102]

[0103] Action Space: As mentioned above, the BS acts as an agent to decide the channel allocation, power control and RIS phase shift matrix of the vehicle. represents the channel and power selection of a V2V vehicle user pair, Θ n represents the phase shift selection of RIS, so the action space of time slot n is defined as

[0104]

[0105] Reward function: In DRL, the design of rewards is flexible, and a good reward can improve the performance of the system. In the resource optimization problem described above, our goal has two main parts: one is to maintain the connection with the BS and keep the AoI at a minimum level, and the other is to increase the success probability of V2V transmission payload. Therefore, the reward for time slot n is set to

[0106]

[0107] Among them, λ1 and λ2 represent the weight coefficients used to balance the rewards, which consist of two parts. The first part represents the reward obtained by the relevant AoI in V2I transmission, and the second part represents the reward obtained by the remaining payload transmission amount in V2V transmission.

[0108] Step (7): Use the SAC algorithm to solve this DRL framework and finally obtain the optimal strategy. In the field of RIS-assisted vehicle communication networks, the challenge is how to adapt to the high-speed mobility of vehicles and the unpredictability of certain environmental information (such as real-time vehicle position and channel status). In addition, in the DRL framework, actions and states are expressed as high-dimensional continuous variables, and their complexity makes it difficult for traditional DRL algorithms to cope with. In order to meet this dynamic decision-making challenge, the present invention introduces an AoI-aware joint scheme for vehicle resource allocation and RIS phase offset control. The scheme is based on the SAC algorithm, which performs well in processing neural networks to approximate high-dimensional states and actions. In addition, the SAC algorithm can also use policy gradients to effectively solve problems characterized by continuous states and actions.

[0109] Let φ be the parameters of the actor network, θ1 and θ2 be the parameters of the two critic networks, and The parameters of the two targetcritic networks. At each training step n, the reinforcement learning goal is to maximize the expected return and entropy simultaneously, which can be defined as

[0110]

[0111] Here γ represents the discount factor, and H(·) represents the impact on future rewards. α is a measure of entropy, and the importance of the entropy term is controlled, called the temperature parameter. E is a mathematical symbol that represents expectation. The goal of RL is to discover the optimal strategy.

[0112] π * = arg max π J(π(a n |s n )). (16)

[0113] During the training process, we will save the experience generated by the training in the buffer [e1,e2,L,e n ,L], when the size of the experience in the buffer is greater than I, we will randomly select I tuples from the replay buffer to form a mini-batch of training data, and let (s i ,a i ,r i ,s i+1 ) becomes the i-th tuple in the mini-batch. Then s i Input the actor network and get action a i,new and the logarithm of the strategy logπ φ (a i,new |s i ), then we can get the loss function gradient of the temperature parameter α as

[0114] ▽ α J i (α)=▽ α E[-αlogπ φ (a i,new |s i )-αH0], (17)

[0115] Where H0 = dim(a n ), then s i 、a i Input two critic networks and output action-value function Q θ1 (s i ,a i ) and Q θ2 (si ,a i ), then s i+1 Input actor network and output a i+1 and logπ φ (a i+1 |s i+1 ). Finally, s i+1 and a i+1 Input two target critic networks and output the target value and Then we can calculate the target value as follows

[0116]

[0117] The gradient of the loss function for parameters θ1 and θ2 is as follows

[0118] ▽ θ1 J i (θ1)=▽ θ1 Q θ1 (a i ,s i )(Q θ1 (s i ,a i )-r i +γQ tar (s i+1 ,a i+1 )), (19)

[0119] ▽ θ2 J i (θ2)=▽ θ2 Q θ2 (a i ,s i )(Q θ2 (s i ,a i )-r i +γQ tar (s i+1 ,a i+1 )). (20)

[0120] The algorithm then calculates the gradient of the φ loss function by first transforming s i and a i,new Input two critic networks and get Q θ1 (s i ,a i,new ) and Q θ2 (s i ,a i,new ), and then take the minimum value of the two to get

[0121] Q(si ,a i,new )=min{Q θ1 (s i ,a i,new ),Q θ2 (s i ,a i,new )}, (twenty one)

[0122] The gradient calculation formula of the φ loss function is

[0123]

[0124] where ξ is a noise sampled from a multivariate normal distribution, and f φ (ξ i ;s i ) is action a i,new In addition, the Adam optimizer is used to update α, θ1, θ2, and φ based on the gradient of the above loss function. Finally, the two parameters of the targetcritic network Updated to

[0125]

[0126] Where τ1 and τ2 are constants that satisfy the following conditions: τ1<<1 and τ2<<1. After all parameters are updated, the algorithm will proceed to the next training. When all training is completed, the algorithm will obtain the optimal strategy.

[0127] The relevant experimental data are as follows:

[0128] Figure 1 The training performance of the proposed SAC algorithm is demonstrated against several benchmarks, with rewards as the performance metric. The intuitive display shows that the SAC algorithm has a higher reward value and converges faster and more stably than other benchmarks. It is worth noting that both DDPG and TD3 face challenges in the early stages of training when faced with high-dimensional state spaces. The unique advantages of the SAC algorithm are obvious, as it can effectively solve the complex problems brought about by high-dimensional state spaces and ensure efficient and stable convergence throughout the training process.

[0129] exist Figure 2 , Figure 3 and Figure 4 In this paper, we present a comparative analysis of various training methods and evaluate their performance in terms of V2I AoI, V2I information transmission rate, and V2V payload transmission success probability during V2I power testing. Figure 2 and Figure 3The effect of increasing the V2I transmission power on the total rate of the V2I link and the corresponding AoI is illustrated. As the V2I transmission power increases, the SINR of the BS also increases, which leads to an increase in the information transmission rate, so that the information transmission rate exceeds the threshold in more time slots, thereby reducing the AoI. Figure 4 The relationship between V2I transmission power and the probability of successful V2V payload transmission is revealed. The observed decrease in success probability is attributed to the increased interference caused by higher V2I transmission power on the V2V link. This interference has an adverse effect on the information rate of the V2V link, thereby preventing the V2V link from completing information transmission within the specified time, resulting in a decrease in success rate.

[0130] Figure 5 and Figure 6 The effect of V2V link payload size on the information transmission rate and AoI of the V2I link is demonstrated. Figure 5 It shows that as the V2V payload size increases, the information transmission rate of all schemes decreases. Correspondingly, Figure 6 It is shown that the AoI gradually increases with the increase of V2V payload size. The decrease in information transmission rate is due to the increase in V2V payload, which leads to the extension of V2V transmission time. This extension may require the increase of transmission power of V2V link, thus aggravating the interference to V2I link. In addition, the extension of V2V transmission time makes V2V link continuously share the spectrum resources of V2I link. This continuous interference of V2V link to V2I link reduces the information transmission rate and leads to the gradual increase of information age (AoI). The observed dynamic changes highlight that V2V payload size is a key role in affecting the performance trade-offs of vehicle-to-vehicle communication networks.

[0131] in addition, Figure 7 , 8 The effect of the number of RIS on the V2I information transmission rate and AoI is shown. As can be seen from the figure, the performance of V2I in terms of information transmission rate and AoI will improve with the increase in the number of RIS, which to a certain extent verifies the help that RIS brings to system performance. The increase in the number of RIS will give the signal more opportunities to be reflected to the receiver through RIS, but as the number of RIS continues to increase, the interference between users will also continue to increase, and the complexity brought by RIS will also increase. Therefore, the deployment of RIS needs to be weighed through model training.

[0132] In order to verify the impact of the increase in the number of users on system performance, we choose to observe the change of the average information transmission rate of each user as the number of RIS increases when the number of users is 4, 8, and 12. Fig. 9It shows that the average information transmission rate of each user will continue to increase, and in the scheme with more users, the average information transmission rate of each user is lower. This is because the increase in the number of users will increase the interference between users, thereby reducing SINR and ultimately leading to a decrease in information rate.

[0133] The above description is only a preferred embodiment of the present invention and does not limit the present invention in any form. Although the present invention has been disclosed as a preferred embodiment as above, it is not used to limit the present invention. Any technical personnel in this field can make some changes or modify the technical contents disclosed above into equivalent embodiments without departing from the scope of the technical solution of the present invention. However, any brief modifications, equivalent changes and modifications made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solution of the present invention are still within the scope of the technical solution of the present invention.

Claims

1. A method for optimizing resources and phase shift in RIS-assisted vehicle communication based on SAC, characterized in that: Specifically include: Step 1: System model setting; the system model setting includes channel model setting, AoI model setting of CUE, and load transmission probability model setting of DUE; Step 2: The optimal channel allocation, power allocation and RIS phase shift are obtained by training the model, and then the achievable capacity, AoI and V2V load transmission success rate of the V2I link are calculated; Step 3: Conduct numerical and simulation evaluations.

2. The method for optimizing resources and phase shift in RIS-assisted vehicle communication based on SAC as claimed in claim 1, characterized in that: In step 1, the specific steps of setting the system model are: (1) Channel model setting: Assume that the coordinates of the base station BS and the intelligent reflection surface RIS are (x BS ,y BS ,z BS ) and (x RIS ,y RIS ,z RIS ), at time slot n, the coordinates of the transmitter in the kth D2D pair are The coordinates of the receiver are The coordinates of the mth CUE are The set of available channels is defined as L = {1, 2, ···, L}. At time slot n, the channel gain of the kth V2V link occupying the lth subchannel is where ρ is the path loss constant for the reference distance d0 = 1 m, β is the logarithmic shadow fading random variable, is the geometric distance between the transmitter and the receiver in the kth V2V pair at time slot n, and α is the corresponding path loss exponent; is the frequency-dependent small-scale fading power component, which is assumed to be exponentially distributed with a unit mean. At time slot n, both are communicating on the lth channel, and the channel coefficient from the mth CUE to the BS is Channel coefficients from the mth CUE to the receiver in the kth V2V pair Channel coefficients from the k'th V2V transmitter to the kth V2V receiver and the channel coefficient from the kth V2V transmitter to the BS are similar to the above definitions; in addition, the channel gain from the mth CUE to the RIS is defined as The meaning of is the geometric distance from the mth CUE to the RIS at time slot n, λ[l] is the carrier length of the lth subchannel, π and j are basic mathematical symbols, and e is a complex function; The arrival and departure array responses of the RIS can be defined as Here d is the spacing between each element in RIS, is the angle of arrival (AoA) of a signal at time slot n. In the same way, the channel gain from RIS to BS at time slot n can be Channel gain from the transmitter to the RIS in the kth V2V pair and the channel gain from RIS to the receiver in the kth V2V pair express it; The received signal-to-interference-plus-noise ratio (SINR) of the mth V2I link and the kth V2V link on the lth subchannel is expressed as in represents the V2I link transmit power, represents the V2V link transmission power at time slot n, H represents the conjugate transposed matrix, σ 2 is the white noise power, and the diagonal phase shift matrix of RIS is F represents the number of elements in RIS. As mentioned above, f is used to represent the fth element in F. Due to hardware limitations, the phase shift can only be selected from a limited set of discrete values, where and β f ∈[0,1] is used as the relevant transmission coefficient and the amplitude of the phase shift, where Q is a variable that controls the discreteness of the phase shift and Then, the uplink transmission rate of the mth CUE and the uplink transmission rate of the kth V2V pair are: (2) CUE AoI model settings: V2I communication must exchange information with the BS in a timely manner, and Must be strictly greater than or equal to R th , which will reset the AoI to Δn, where R th represents the minimum rate required for V2I communication. Less than R th The AoI of V2I will gradually increase when . Here, the AoI of the mth V2I link is defined as represents the time since the last successful communication; therefore, the AoI changes through the following equation (3) Load transmission probability model setting of DUE: In order to ensure the reliability of V2V transmission security information, the probability of V2V transmission successfully delivering a data packet of size D within the time budget N is modeled as Where B is the bandwidth of the spectrum subband, D represents the size of the periodically generated V2V payload, and Δn represents the channel coherence time.

3. The method for optimizing resources and phase shift in RIS-assisted vehicle communication based on SAC as claimed in claim 2, characterized in that: In step 2, the specific steps of optimizing channel allocation, power allocation and RIS phase shift are as follows: Constraint C2 imposes restrictions on the transmission power range within a V2V pair, constraints C3 and C4 assume that each V2V pair occupies only one channel, constraint C5 limits the range of RIS phase shift, and optimization objective P1 involves vehicle resource allocation and RIS phase change control based on AoI perception; BS centrally handles vehicle channel sharing, power allocation and RIS phase shift matrix control. At each time slot n, the agent observes the current state s n , and then take action a according to the current policy n After the action is completed, the agent will receive the relevant reward r from the system environment n , then, the state transitions to the next state s n+1 , marking the completion of the cycle of this dynamic process; the entire sequence is represented by e n =(s n ,a n ,r n ,s n+1 ), which encompasses the important components of the DRL framework for solving optimization problems.

4. The method for optimizing resources and phase shift in RIS-assisted vehicle communication based on SAC as claimed in claim 3, characterized in that: The specific optimization algorithm of step 2 is: construct a DRL framework including state, action and reward, and then use the SAC algorithm to solve the framework to obtain the optimal strategy; specifically: 1) State space: At time slot n, the state of the system is composed of the following parts Composition: V2I link local channel V2V link local channel Interference of the previous time slot to the receiver in V2V link The transmission load of the current time slot in the V2V link V2I Link And the phase shift matrix Θ of RIS n ; Therefore, the system state at time slot n is given by the following formula 2) Action Space: As mentioned above, the BS acts as an agent to decide the channel allocation, power control and RIS phase shift matrix of the vehicle. represents the channel and power selection of a V2V vehicle user pair, Θ n represents the phase shift selection of RIS, so the action space of time slot n is defined as 3) Reward function: The objective has two main parts: one is to maintain the connection with the BS and keep the AoI at a minimum level, and the other is to increase the success probability of V2V transmission payload; therefore, the reward for time slot n is set to Among them, λ1 and λ2 represent the weight coefficients used to balance the rewards, which are composed of two parts. The first part represents the reward obtained by the relevant AoI in V2I transmission, and the second part represents the reward obtained by the remaining payload transmission amount in V2V transmission; 4) Training phase of SAC: Let φ be the parameters of the actor network, θ1 and θ2 be the parameters of the two critic networks, and The parameters of the two target critic networks; at each training step n, the reinforcement learning goal is to maximize both the expected reward and entropy, defined as Here γ represents the discount factor, H(·) represents the impact on future rewards, and α is a measure of entropy. The importance of the entropy term is controlled and is called the temperature parameter. The goal of RL is to discover the optimal strategy. p * =argmax π J(π(a n |s n )). (16) During the training process, the experience generated by the training is saved in the buffer [e1,e2,···,e n ,···], when the size of the experience in the buffer is greater than I, I tuples are randomly selected from the replay buffer to form a mini-batch of training data, and let (s i ,a i ,r i ,s i+1 ) becomes the i-th tuple in the mini-batch, and then s i Input the actor network and get a i,new and logπ θ (a i,new |s i ), then the loss function gradient of the temperature parameter α can be obtained as Where H0 = dim(a n ), then s i 、a i Input two critic networks and output action-value function Q θ1 (s i ,a i ) and Q θ2 (s i ,a i ), then s i+1 Input actor network and output a i+1 and logπ θ (a i+1 |s i+1 ); Finally, s i+1 and a i+1 Input two target critic networks and output and Then the target value can be calculated as follows The gradient of the loss function for parameters θ1 and θ2 is as follows The algorithm then calculates the gradient of the φ loss function by first transforming s i and a i,new Input two critic networks and get Q θ1 (s i ,a i,new ) and Q θ2 (s i ,a i,new ), and then take the minimum value of the two to get Q(s i ,a i,new )=min{Q θ1 (s i ,a i,new ),Q θ2 (s i ,a i,new )}, (21) The gradient calculation formula of the φ loss function is where ξ is a noise sampled from a multivariate normal distribution, and f φ (ξ i ;s i ) is action a i,new reparameterized function; in addition, the Adam optimizer is used to update α, θ1, θ2, and φ based on the gradient of the above loss function. Finally, the two parameters of the targetcritic network Updated to Among them, τ1 and τ2 are constants that satisfy the following conditions: τ1<<1 and τ2<<1; after all parameters are updated, the algorithm will continue to the next training. When all training is completed, the algorithm will obtain the optimal strategy.

Images