RIS auxiliary vehicle edge calculation method based on improved DRL

Abstract

The invention relates to an improved DRL-based RIS-assisted vehicle edge calculation method, and the method comprises the steps: introducing an intelligent reflection surface (RIS) to assist the communication of a vehicle, and providing an additional communication path; an improved multi-agent depth deterministic strategy gradient (MADDPG) algorithm is adopted, and power distribution of vehicle users (VU) is optimized; by optimizing unloading power and local execution power, the system can realize optimal configuration of power distribution under different task arrival rates and channel conditions; by using a deep reinforcement learning (DRL) framework, a dynamic environment and a complex channel condition can be better adapted; the problem of communication interruption caused by the fact that the vehicle encounters obstacles such as buildings in the running process is effectively solved, the stability and reliability of communication are improved, and the overall performance of the system is improved.

Description

Technical Field

[0001] The present invention relates to the field of vehicle networking edge computing reinforcement learning technology, and in particular to a RIS-assisted vehicle edge computing method based on improved DRL. Background Art

[0002] Vehicle Edge Computing (VEC) is an emerging technology with great potential in the field of Internet of Vehicles (Iov). It enables vehicles to perform intensive computing tasks locally or offload tasks to nearby edge devices. This method effectively reduces the computing burden of on-board devices and improves task processing efficiency. Existing VEC systems rely on stable wireless communication connections to maintain connections with edge servers while the vehicle is moving. However, due to the presence of obstacles such as buildings, the quality of the VEC system communication link may be severely degraded, thereby hindering the real-time and stability of the task offloading process. In order to address this problem, the intelligent reflecting surface (RIS) technology was introduced.

[0003] The intelligent reflecting surface (RIS) technology provides an alternative concentric path to assist vehicle communication. By adjusting the phase shift of the reflecting surface unit to control the propagation path of the wireless signal, it can enhance the quality of the communication link without adding additional power, greatly improving the performance of the VEC system.

[0004] The combination of VEC technology and RIS technology shows great potential in improving vehicle communication performance. However, in the face of vehicle mobility and complex environments, the optimization of RIS technology phase shift is difficult to achieve the best effect and cannot fully utilize the potential of RIS. In addition, the existing centralized deep reinforcement learning (DRL) algorithm (such as DDPG) is difficult to handle multi-user resource competition and dynamic environmental changes in multi-vehicle user scenarios, resulting in unreasonable power allocation. At the same time, due to hardware limitations, the number of RIS technology phase shift values ​​is limited, which increases the difficulty of system optimization and cannot achieve optimal phase shift control, affecting the performance of the overall system. Summary of the invention

[0005] In view of this, the present invention provides a RIS-assisted vehicle edge computing method based on improved DRL, which effectively solves the problem of unstable communication of vehicle edge computing systems in the prior art and provides strong support for real-time task processing of vehicles in complex environments.

[0006] To achieve the above object, a RIS-assisted vehicle edge computing method based on improved DRL is characterized by comprising the following steps:

[0007] S1. Establish a smart reflective surface RIS-assisted vehicle-mounted edge computing VEC model under a multiple-input multiple-output MIMO system;

[0008] S2, the vehicle user VU divides the task into offload processing and local execution;

[0009] S3, obtaining the optimal phase shift coefficient of the intelligent reflector RIS;

[0010] S301, fixedly optimizing the unloading power of the vehicle user VU;

[0011] S302, use deep deterministic policy gradient DDPG to train the phase shift of the intelligent reflector RIS, obtain the optimal phase shift coefficient of RIS, and find the optimal strategy μ * ;

[0012] S4, obtain the optimal power allocation scheme through the improved MADDPG algorithm;

[0013] The strategy of each agent is π = {π1, π2, ... π K}, where K is the number of agents;

[0014] S401, the local critic receives the state space and action space of the corresponding agent and evaluates the reward of the agent;

[0015] S402, the state space and action space of all agents are used as the input of the global critic of the MADDPG algorithm;

[0016] S403, use the twin global critic with double-delay deterministic policy gradient to replace the global critic. After correction, the policy gradient expression of each agent is:

[0017]

[0018] Among them, E() represents the expected function, D represents the buffer size, a k =π k (s k ) indicates that the kth agent follows its own strategy π k Select the action, represents the local critic Q function, s=(s1,s2,…s K ) represents the state vector of all agents, a=(a1,a2,…a K ) represents the action vectors of all agents, represents the twin global critic Q function, where j = 1, 2, Ψ j represents the parameters Ψ1, Ψ2, g in the twin global critic Q function j represents the two functions contained in the twin global critic Q function;

[0019] S404, Update twin global critic Q function

[0020]

[0021] Among them, j=1,2, π′={π′1,π′2,…π′ K} represents the target strategy, θ′={θ′1,θ′2,…θ′ K} represents the target strategy π′={π′1,π′2,…π′ K} corresponding to the parameter, γ represents the discount factor;

[0022] S405: Update local critic Q function

[0023]

[0024] in,

[0025] Preferably, in the VEC model, the base station BS has M antennas associated with a VEC server, the vehicle user VU has a single antenna, and the offloading process is to offload the task to a VEC edge service processor associated with the base station BS.

[0026] Preferably, obtaining the optimal phase shift coefficient of RIS comprises the following steps:

[0027] Initialize the actor network μ(s;θ μ ) and the critic network Q(s,a;θ Q ) parameter θ μ and θ Q ;

[0028] Set the target actor network μ′(s;θ μ′ ) and the target critic network Q′(s,a;θ Q′ ) parameter θ μ′ and θ Q′ ;

[0029] In each time slot t, the base station BS obtains the current state s t , select action a based on the current strategy μ and exploration noise Δn t ;

[0030] Perform the action a t Transition to the next state s t+1 , and obtain the action a t Rewards received t ;

[0031] Store the experience of each step in a buffer for random sampling training;

[0032] At the end of each time slot t, the target actor network and target critic network parameters θ are soft-updated μ′ and θ Q′ , and gradually approaches the network parameter θ μ and θ Q ;

[0033] Repeat the above steps until the predetermined number of training steps is reached or satisfied.

[0034] Preferably, the parameter θ μ′ and θ Q′ The update process is as follows:

[0035] θ μ′ ←τθ μ +(1-τ)θ μ′

[0036] θ Q′ ←τθ Q +(1-τ)θ Q′

[0037] Among them, τ is a soft update coefficient.

[0038] Preferably, the state space of the kth vehicle user VU at time slot t can be expressed as:

[0039] s=[q k (t),q k,o (t),q k,l (t),q k,o (t)+q k,l (t)-q k (t),γ k (t-1)]

[0040] Among them, q k (t) represents the buffer queue length, q k,o (t) represents the size of the task to be executed, q k,l (t) represents the size of the task executed locally, q k,o (y)+q k,l (t)-q k (t) represents the part of offloaded and local processing tasks overflow, γ k (t-1) represents the signal-to-noise ratio of the kth vehicle user at time slot t.

[0041] Preferably, the action space of the kth vehicle user VU at time slot t can be expressed as:

[0042] a k(t) = [p k,o (t),p k,l (t)]

[0043] Among them, p k,o (t) and p k,l (t) represent the offloading task and local execution power of the kth vehicle user VU at time slot t, respectively.

[0044] Preferably, the local and overall reward expressions of the kth vehicle user VU are:

[0045] r k,l =-[w1(p k,o (t)+p k,l (t))+w2q k (t)]

[0046] The long-term reward expression is:

[0047]

[0048] Among them, w1 and w2 are weight factors used for reward balance.

[0049] Compared with the prior art, the present invention has the following beneficial effects:

[0050] 1. The present invention introduces intelligent reflecting surface (RIS) to assist vehicle communication and provide an additional communication path, which effectively solves the problem of communication interruption caused by obstacles such as buildings encountered by vehicles during driving, and improves the stability and reliability of communication;

[0051] 2. The control scheme provided by the present invention optimizes the unloading power and the local execution power, so that the system can achieve the optimal configuration of power allocation under different task arrival rates and channel conditions, thereby improving resource utilization efficiency; and by adopting the improved multi-agent deep deterministic policy gradient (MADDPG) algorithm, the power allocation of vehicle users (VU) is optimized, which can better cope with resource competition among multiple users and improve the overall performance of the system;

[0052] 3. The present invention uses the DDPG algorithm to optimize the RIS phase shift, which can achieve a better phase shift control effect under limited hardware conditions, solve the non-convex optimization problem, and improve the calculation efficiency and accuracy of the system;

[0053] 4. The deep reinforcement learning (DRL) framework proposed in the present invention can better adapt to dynamic environments and complex channel conditions, and improve the adaptability and stability of the system in different environments. The simulation results show that the scheme proposed in the present invention is significantly superior to the traditional centralized DDPG, TD3 method and random scheme in terms of communication stability, power allocation efficiency and system performance, which verifies the superiority of the present invention in practical applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 This is the operation framework diagram of the present invention;

[0055] Figure 2 A RIS-assisted VEC model diagram for a multi-user MIMO system established by the present invention;

[0056] Figure 3 A comparison diagram of rewards for different numbers of RIS elements under DDPG training in the present invention;

[0057] Figure 4 A diagram showing the relationship between the rewards of the present invention and the centralized DDPG and TD3 algorithms;

[0058] Figure 5 is a relationship diagram of the power consumption of all users under different task arrival rates of the present invention;

[0059] Figure 6 is a relationship diagram of the buffer length of all users under different task arrival rates of the present invention;

[0060] Figure 7 A diagram showing the relationship between power consumption of different VUs in the solution of the present invention;

[0061] Figure 8 A diagram showing the relationship between buffer lengths of different VUs in the solution of the present invention;

[0062] Fig. 9 Pseudo code for the training algorithm of the present invention. DETAILED DESCRIPTION

[0063] In order to further explain the technical means and effects adopted by the present invention to achieve the predetermined invention purpose, the specific implementation mode, structure, characteristics and effects of the present invention are described in detail below in combination with the accompanying drawings and preferred embodiments.

[0064] In order to effectively improve the real-time performance and stability of the vehicle edge computing system communication and provide strong support for the real-time task processing of the vehicle in a complex environment, the present invention provides a RIS-assisted vehicle edge computing method based on an improved DRL, which specifically includes the following steps:

[0065] Step (1): Figure 1As shown, a smart reflective surface RIS-assisted vehicle-mounted edge computing VEC model under a multi-user MIMO system is established;

[0066] In the VEC model, the base station BS has M antennas and is associated with a VEC server, the vehicle user VU has a single antenna, and the offloading process is to offload the task to the VEC edge service processor associated with the base station BS;

[0067] Since the computing power of VU is limited, it can offload part of the tasks to the VEC server for processing; when VU is blocked by obstacles, VU cannot communicate directly and effectively with BS, so RIS with N reflection elements is deployed to assist VU to offload tasks to edge devices (assuming that there are two links at the same time, that is, VU communicates with BS through RIS and VU communicates with BS directly);

[0068] In a certain urban intersection scenario, vehicle users VU are randomly distributed at the intersection. Assuming there are K vehicle users, represented by the set K = {1,2,…,K}, VU can flexibly offload tasks and execute them locally, dividing the task processing process into T discrete times of equally spaced time slots Δt, represented by the set T = {1,2,…,T}. In each time slot t, the vehicle randomly generates a task, where the arrival of the task follows a Poisson distribution with an arrival rate of η; consider a quasi-static scenario, in which the channel conditions remain unchanged within a time slot, but vary between different time slots.

[0069] Step (2): For the kth VU, assume that its task arrival rate at time slot t is η k (t), follows the Poisson distribution, so the arrival amount a of the task of the kth VU during time slot t can be calculated k (t) is:

[0070] a k (t) = η k (t)×Δt

[0071] Wherein, Δt represents the time interval between time slot t and time slot t+1;

[0072] After the task arrives, it will first be saved in the queue buffer and processed at time slot t+1. The queue length of the kth VU at time slot t+1 can be obtained as:

[0073] q k (t+1)=[q k (t)-q k,o (t)-q k,l (t)] + +a k (t)

[0074] Among them, qk,o (t) and q k,l (t) represent the amount of tasks processed by the kth VU at time slot t through offloading and local execution, respectively.

[0075] Step (3): In the VEC model, assume that the coordinates of BS and RIS are (BS x ,BS y ,BS z ) and (RIS x ,RIS y ,RIS z ), therefore, at time slot t, the channel gain between the kth VU and BS is:

[0076]

[0077] Where ρ represents the path loss at the reference distance d0 = 1m, represents the geometric distance between BSs of the kth VU at time slot t, α k,b represents the corresponding path loss exponent, represents the frequency-dependent small-scale attenuation power component, which is assumed to be an exponential distribution with unit mean;

[0078] There is a primary line-of-sight (LoS) between the BS and RIS, and between the RIS and the vehicle. Therefore, these communication links experience small-scale fading, which is modeled as Rician fading of pure LoS components. Since both the RIS and the BS are deployed at fixed locations, the RIS-BS link will remain static, and the channel gain h between the RIS and the BS can be obtained. r,b for:

[0079]

[0080] Among them, d r,b represents the geometric distance from RIS to BS, α r,b represents the path loss exponent of the RIS-BS link, R represents the Rician coefficient associated with small-scale fading, represents the LoS component, and the expression is:

[0081]

[0082] Where λ represents the carrier length, d RIS represents the spacing between RIS elements, θ AoD Indicates the departure angle of the signal from RIS to BS;

[0083] Similarly, the channel gain expression from the kth vehicle to the RIS is:

[0084]

[0085] in, represents the geometric distance between the kth vehicle and the RIS at time slot t, α k,r is the path loss exponent between the vehicle and the RIS, represents the LOS component between the kth vehicle and the RIS at time slot t, and is expressed as follows:

[0086]

[0087] in, represents the arrival angle of the signal from the kth vehicle to the RIS at time slot t;

[0088] Considering that there are two links at the same time, namely, VU communicates with BS through RIS and VU communicates directly with BS, it can be obtained that the signal-to-noise ratio (SNR) between the kth vehicle and BS at time slot t is:

[0089]

[0090] in, represents the unloading power of the kth VU at time slot t, σ 2 Represents the thermal noise power BS can detect the SNR of the th VU at time slot t, that is, γ k (t), and transmits it to the VU in the next time slot; in this case, the kth VU receives γ at time slot t. k (t-1), which can reflect the channel status of the kth VU at time slot t. represents the unloading power of the kth VU at time slot, Θ t represents the diagonal phase shift matrix of RIS, which is expressed as:

[0091]

[0092] Due to hardware limitations, the phase shift can only be selected from a limited set of discrete values. and β n ∈[0,1] represents the relevant transmission coefficient and the amplitude of the phase shift, Q=2 b , b represents the control bit of the discrete degree of phase shift, so it can be calculated

[0093]

[0094] When the vehicle user VU chooses to offload the task to the VEC edge server associated with the BS, the data transmission rate of the kth VU at time slot t can be obtained as:

[0095] C k(t) = Wlog2(1+γ k (t))

[0096] Wherein, W represents the bandwidth of data transmission;

[0097] The number of tasks q processed by the kth VU at time slot t by offloading can be obtained k,o (t):

[0098] q k,o (t) = Δt × C k (t)

[0099] Step (4): When the kth VU chooses local computing, the task size that can be processed by local execution at time slot t is:

[0100] q k,l (t) = Δtf k (t) / L

[0101] Where L represents the CPU frequency required to process a 1-bit task, and f k (t) represents the CPU frequency scheduled by adjusting the chip voltage using the dynamic voltage frequency scaling (DVFS) technology, and the expression is:

[0102]

[0103] Among them, p k,l (t) represents the local execution power of the kth VU at time slot t, and c represents the effective selected capacitance.

[0104] Step (5): In this embodiment, the kth VU aims to minimize the power consumed by offloading and local execution and the queue length of the buffer. Therefore, the multi-objective optimization problem of each VU can be expressed as:

[0105]

[0106]

[0107] Where C1 and C2 represent the power range limits of VU when offloading tasks and executing tasks locally, respectively. Due to the limitation of RIS hardware devices, the phase shift size can only be selected within a limited range, and constraint C3 limits the value of RIS phase shift. Since the objective function is a non-convex function, the optimization problem is difficult to solve, so a new framework is proposed to combine the improved MADDPG algorithm with the centralized DDPG algorithm.

[0108] Since the RIS phase shift controller and VU are not the same type of agents, their actions and rewards are also different. Therefore, the DDPG algorithm is first used to obtain the optimal phase shift coefficient, and then the improved MADDPG algorithm is used to obtain the optimal power allocation scheme.

[0109] Step (6): obtaining the optimal phase shift coefficient of the smart reflector RIS;

[0110] S301, fixedly optimizing the unloading power of the vehicle user VU;

[0111] S302, with the goal of maximizing the information transmission rate of the VU communication link, using deep deterministic policy gradient DDPG to train the phase shift of the smart reflector RIS to obtain the optimal phase shift coefficient of RIS;

[0112] The phase shift coefficient optimization problem of RIS can be expressed as:

[0113]

[0114] Step (7): BS acts as an agent to interact with the RIS-assisted vehicle network and adjusts the phase shift coefficient of RIS according to its strategy, aiming to solve the optimization problem P2. At time slot t, BS obtains the current state s t , and then obtain the corresponding action a based on the current strategy t , by taking an action to transition to the next state s t+1 , and obtain the reward r for making this action t This process can be expressed as t =(s t ,a t ,r t ,s t+1 ), in this model, the relevant state space, action space and reward are expressed as follows:

[0115] State space: The state obtained by BS at time slot t consists of the following parts: RIS phase shift coefficient Plane coordinate position of the vehicle The signal-to-noise ratio of VUk is γ k (t). Therefore, the state space of BS is:

[0116]

[0117] Action space: BS acts as an agent, and its action space is defined as is the phase shift of the nth RIS element at time slot t. The BS maximizes the information transmission rate of VU by adjusting the phase shift of RIS;

[0118] Reward function: The reward function represents the goal to be optimized. In this section, the information transmission rate of VU is optimized by adjusting the RIS phase shift coefficient and a weight factor w is added to balance the reward. Therefore, the reward function is set to:

[0119] r t =w log2(1+γ k (t))

[0120] For high-dimensional state space, DDPG can learn the complex features and patterns of the state through deep neural networks, so as to predict the appropriate action given the current state. Therefore, for the high-dimensional state and action space in this model, DDPG is used for training. After obtaining continuous action values, the appropriate RIS phase shift coefficient is obtained by amplifying and rounding, thus achieving good training results.

[0121] In the DDPG algorithm, the state space and action space are usually represented by a deep neural network, which includes two networks: the actor network is used to learn strategies (i.e. action selection), and its goal is to maximize the critic network's estimate of the action value. The critic network is used to evaluate the value of the strategy, and through supervised learning, it learns to evaluate the quality of the actions output by the actor network. In addition, in order to ensure the stability of the algorithm, the DDPG algorithm uses a target actor network and a target critic network. The network structures of these two are the same as those of the actor and critic networks, respectively; the relevant pseudo code is shown in Algorithm 1;

[0122] In this example, the goal is to find the optimal strategy μ * To maximize the agent's expected long-term discounted reward, the expression is as follows:

[0123]

[0124] The process of the DDPG algorithm can be summarized as follows:

[0125] Initialize the actor network μ(s;θ μ ) and the critic network Q(s,a;θ Q ) parameter θ μ and θ Q ;

[0126] Set the target actor network μ′(s;θ μ′ ) and the target critic network Q′(s,a;θ Q′ ) parameter θ μ′ and θ Q′ ;

[0127] In each time slot t, the base station BS obtains the current state s t , which is the state space of BS, selects action a based on the current strategy μ and exploration noise Δn t ;

[0128] Perform the action a t Transition to the next state s t+1 , and obtain the action a t Rewards received t ;

[0129] Store the experience of each step in a buffer for random sampling training;

[0130] At each time slot t, when the buffer length is greater than I, a batch of data is sampled from the buffer D and the parameters of the critic network are updated to minimize the loss. The loss function is as follows:

[0131]

[0132] y i =r i +γ Q′ (s i+1 ,μ′(s i+1 θ μ′ );θ Q′ )

[0133] At the same time, the actor network parameters are updated according to the gradient of the critic network

[0134]

[0135] Where (J) is the objective function of the actor;

[0136] At the end of each time slot t, the target actor network and target critic network parameters θ are soft-updated μ′ and θ Q′ , and gradually approaches the network parameter θ μ and θ Q ;

[0137] Parameter θ μ′ and θ Q′ The update process is as follows:

[0138] θ μ′ ←τθ μ +(1-τ)θ μ′

[0139] θ Q′ ←τθ Q +(1-τ)θ Q′

[0140] Among them, τ is a soft update coefficient;

[0141] Repeat the above steps until the predetermined number of training steps is reached or satisfied.

[0142] Step (8): Use the MARL algorithm to solve the VU power allocation problem. Each vehicle interacts with the environment as an agent and makes power allocation decisions through corresponding strategies. In this model, the relevant state space, action space, and reward are expressed as follows:

[0143] State space: buffer queue length q k (t), the size of the task q to be offloaded k,o (t), the size of the task executed locally q k,l (t), the portion of the overflow from offloading and local processing tasks q k,o (t)+q k,l (t)-q k (t), in addition, the signal-to-noise ratio SNR of the kth VU at time slot t is γ k (t-1), depends on h r,b and γ k (t-1) can reflect the uncertain channel condition of VU at time slot t. Therefore, the state space of the kth VU at time slot can be expressed as:

[0144] s=[q k (t),q k,o (t),q k,l (t),q k,o (t)+q k,l (t)-q k (t),γ k (t-1)]

[0145] Action space: As described above, each VU allocates offloading power and local execution power according to the corresponding strategy, so the action space of the kth VU at time slot t is defined as:

[0146] a k (t) = [p k,o (t),p k,l (t)]

[0147] Since the deterministic strategy is adopted, the agent can go from 0 to P max Choose any power;

[0148] Reward function: In the proposed scheme, the reward for each agent has two aspects, one is the global reward that reflects the cooperation between agents, and the other is the local reward that helps each agent explore the optimal power allocation scheme; in DRL, the reward can be set flexibly, and a good reward can improve the performance of the system; the goal of this embodiment is to optimize the offloading and local power level of the kth agent, as well as the corresponding buffer length; therefore, at time slot t, the local and global rewards of the kth VU are:

[0149] r k,l =-[w1(p k,o (t)+p k,l (t))+w2q k (t)]

[0150] The long-term rewards are:

[0151]

[0152] Among them, w1 and w2 are weight factors used for reward balance;

[0153] Step (9): Use a multi-agent deep reinforcement learning algorithm to simultaneously learn how to maximize global rewards and local rewards. It mainly includes two critics: a global critic, which is shared among all agents and inputs the states and actions of all agents to maximize the global reward; and a local critic, which receives the states and actions of each agent and evaluates its own reward. Due to the problem of overestimation and suboptimal strategy of the q function caused by function approximation error in the DDPG algorithm, improvements are made on the basis of the MADDPG algorithm, using double-delayed deterministic policy gradient to replace the global critic, which can better improve the performance of the system.

[0154] In a vehicle environment, there are K agents VU, and the strategy of each agent is π = {π1, π2, ... π K}; Agent's strategy π k , Q function and the twin global critic Q function The parameters are θ k , Φ k and Ψ1, Ψ2; for each agent, the revised policy gradient can be expressed as:

[0155]

[0156] Among them, E() represents the expected function, D represents the buffer size, a k =π k (s k) indicates that the kth agent follows its own strategy π k Select the action, represents the local critic Q function, s=(s1,s2,…s K ) represents the state vector of all agents, a=(a1,a2,…a K ) represents the action vectors of all agents, represents the twin global critic Q function, where j = 1, 2, Ψ j represents the parameters Ψ1, Ψ2, g in the twin global critic Q function j represents the two functions contained in the twin global critic Q function;

[0157] Update the twin global critic Q function

[0158]

[0159]

[0160] Among them, j=1,2, π′={π′1,π′2,…π′ K} represents the target strategy, θ′={θ′1,θ′2,…θ′ K} represents the target strategy π′={π′1,π′2,…π′ K} corresponding to the parameter, γ represents the discount factor;

[0161] Update the local critic Q function

[0162]

[0163] in,

[0164] The above description is only a preferred embodiment of the present invention and does not limit the present invention in any form. Although the present invention has been disclosed as a preferred embodiment as above, it is not used to limit the present invention. Any technical personnel in this field can make some changes or modify the technical contents disclosed above into equivalent embodiments without departing from the scope of the technical solution of the present invention. However, any brief modifications, equivalent changes and modifications made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solution of the present invention are still within the scope of the technical solution of the present invention.

Claims

1. A RIS-assisted vehicle edge computing method based on improved DRL, characterized in that: The following steps are involved: S1. Establish a smart reflective surface RIS-assisted vehicle-mounted edge computing VEC model under a multiple-input multiple-output MIMO system; S2, the vehicle user VU divides the task into offload processing and local execution; S3, obtaining the optimal phase shift coefficient of the intelligent reflector RIS; S301, fixedly optimizing the unloading power of the vehicle user VU; S302, use deep deterministic policy gradient DDPG to train the phase shift of the intelligent reflector RIS, obtain the optimal phase shift coefficient of RIS, and find the optimal strategy μ * ; S4, obtain the optimal power allocation scheme through the improved MADDPG algorithm; The strategy of each agent is π = {π1, π2, ... π K }, where K is the number of agents; S401, the local critic receives the state space and action space of the corresponding agent and evaluates the reward of the agent; S402, the state space and action space of all agents are used as the input of the global critic of the MADDPG algorithm; S403, use the twin global critic with double-delay deterministic policy gradient to replace the global critic. After correction, the policy gradient expression of each agent is: Among them, E() represents the expected function, D represents the buffer size, a k =π k (s k ) indicates that the kth agent follows its own strategy π k Select the action, represents the local critic Q function, s=(s1,s2,…s K ) represents the state vector of all agents, a=(a1,a2,…a K ) represents the action vectors of all agents, represents the twin global critic Q function, where j = 1, 2, Ψ j represents the parameters Ψ1, Ψ2, g in the twin global critic Q function j represents the two functions contained in the twin global critic Q function; S404, Update twin global critic Q function Among them, j=1,2, π′={π′1,π′2,…π′ K } represents the target strategy, θ ′ ={θ′1,θ′2,…θ′ K } represents the target strategy π′={π′1,π′2,…π′ K } corresponding to the parameter, γ represents the discount factor; S405: Update local critic Q function in, 2. According to claim 1, a RIS-assisted vehicle edge computing method based on improved DRL is characterized in that: In the VEC model, the base station BS has M antennas associated with a VEC server, the vehicle user VU has a single antenna, and the offloading process is to offload the task to the VEC edge service processor associated with the base station BS.

3. According to the RIS-assisted vehicle edge computing method based on improved DRL in claim 1, it is characterized in that: Obtaining the optimal phase shift coefficient of RIS includes the following steps: Initialize the actor network μ(s;θ μ ) and the critic network Q(s,a;θ Q ) parameter θ μ and θ Q ; Set the target actor network μ′(s;θ μ′ ) and the target critic network Q′(s,a;θ Q′ ) parameter θ μ′ and θ Q′ ; In each time slot t, the base station BS obtains the current state s t , select action a based on the current strategy μ and exploration noise Δn t ; Perform the action a t Transition to the next state s t+1 , and obtain the action a t Rewards received t ; Store the experience of each step in a buffer for random sampling training; At the end of each time slot t, the target actor network and target critic network parameters θ are soft-updated μ′ and θ Q′ , and gradually approaches the network parameter θ μ and θ Q ; Repeat the above steps until the predetermined number of training steps is reached.

4. The RIS-assisted vehicle edge computing method based on improved DRL according to claim 3 is characterized in that: Parameter θ μ′ and θ Q′ The update process is as follows: i μ′ ←tth μ +(1-τ)θ μ′ i Q′ ←tth Q +(1-τ)θ Q′ Among them, τ is a soft update coefficient.

5. The RIS-assisted vehicle edge computing method based on improved DRL according to claim 1 is characterized in that: The state space of the kth vehicle user VU at time slot t can be expressed as: s=[q k (t),q k,o (t),q k,l (t),q k,o (t)+q k,l (t)-q k (t),γ k (t-1)] Among them, q k (t) represents the buffer queue length, q k,o (t) represents the size of the task to be offloaded, q k,l (t) represents the size of the task executed locally, q k,o (t)+q k,l (t)-q k (t) represents the part of offloaded and local processing tasks overflow, γ k (t-1) represents the signal-to-noise ratio of the kth vehicle user at time slot t.

6. The RIS-assisted vehicle edge computing method based on improved DRL according to claim 5 is characterized in that: The action space of the kth vehicle user VU at time slot t can be expressed as: a k (t)=[p k,o (t),p k,l (t)] Among them, p k,o (t) and p k,l (t) represent the offloading task and local execution power of the kth vehicle user VU at time slot t, respectively.

7. The RIS-assisted vehicle edge computing method based on improved DRL according to claim 6 is characterized in that: The local and total reward expressions of the kth vehicle user VU are: r k,l =-[w1(p k,o (t)+p k,l (t))+w2q t (t)] The long-term reward expression is: Among them, w1 and w2 are weight factors used for reward balance.

Images