Reconfigurable intelligent surface auxiliary vehicle-mounted edge calculation method and system based on reinforcement learning
By using multi-agent deep reinforcement learning model and block coordinate descent algorithm in the on-board edge computing system, RIS phase shift and power distribution are optimized, the problems of vehicle communication interruption and resource competition are solved, the stability and efficiency of the system are improved, and the adaptability of the system is improved. Complex environment.
Patent Information
- Application Number
- CN202411031126.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-30
- Publication Date
- 2024-11-22
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
The existing on-board edge computing system has problems such as communication interruption caused by obstacles during vehicle movement, complex RIS phase shift optimization, unreasonable power allocation, and hardware restrictions, resulting in a decline in communication link quality and low resource competition efficiency.
Using a multi-agent deep reinforcement learning model based on reinforcement learning, combined with the block coordinate descent algorithm, RIS phase shift and power distribution are optimized, and through iterative training of multi-agent deep reinforcement learning model, the vehicle's task offloading and local execution in a dynamic environment is realized. Optimal configuration.
It improves the stability and resource utilization efficiency of vehicle communication, enhances the adaptability and performance of the system in multi-users and complex environments, solves the problem of phase shift control under limited hardware conditions, and is significantly better than traditional methods.
Smart Images

Figure CN119012284A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of vehicle networking edge computing technology, and in particular to a reinforcement learning-based reconfigurable intelligent surface-assisted vehicle-mounted edge computing method and system. Background Art
[0002] VEC technology allows vehicles to handle high-intensity computing tasks locally or by offloading to nearby edge devices. This approach can effectively reduce the computing burden of on-board equipment and improve task processing efficiency. Existing VEC systems rely on stable wireless communication connections to maintain connections with edge servers while the vehicle is moving. However, while driving, vehicles often encounter obstacles such as buildings, which hinder communication, causing communication interruptions and affecting the efficiency of task offloading. Intelligent Reflecting Surface (RIS) Technology: RIS technology controls the propagation path of wireless signals by adjusting the phase shift of the reflecting surface unit, which can enhance the quality of the communication link without adding additional power. At present, algorithms for RIS phase shift optimization include block coordinate descent (BCD), alternating optimization, and semi-definite relaxation, which perform well in fixed or simple environments.
[0003] The problems with the above technologies are as follows:
[0004] 1. Communication interruption and instability: Communication interruption caused by obstacles encountered by the vehicle during driving affects the real-time and stability of task offloading;
[0005] 2. Complex RIS phase shift optimization: Existing methods are difficult to achieve optimal results in the face of vehicle mobility and complex environments, and cannot fully utilize the potential of RIS;
[0006] 3. Unreasonable power allocation: Existing centralized deep reinforcement learning (DRL) algorithms (such as DDPG) have difficulty handling multi-user resource competition and dynamic environmental changes in multi-vehicle user scenarios, resulting in unreasonable power allocation;
[0007] 4. Hardware limitation: The number of phase shift values of RIS is limited. Due to hardware conditions, it increases the difficulty of system optimization and affects the overall performance.
[0008] The limitations of the existing technology in solving the above problems are as follows:
[0009] 1. Poor adaptability to dynamic environments: The existing RIS phase shift optimization algorithm is difficult to quickly adapt to changes in dynamic environments and when vehicles have high mobility, resulting in a decrease in the quality of the communication link;
[0010] 2. Limitations of centralized algorithms: When dealing with multi-vehicle user scenarios, centralized DRL algorithms have high computational complexity, low efficiency, difficulty in real-time response, and are prone to single point failures;
[0011] 3. Resource competition in multi-user scenarios: Existing technologies fail to effectively solve the resource competition problem in multi-user scenarios, resulting in low efficiency in power allocation and task offloading;
[0012] 4. Impact of hardware limitations: Due to the limited number of RIS phase shift values, existing algorithms are difficult to achieve optimal phase shift control under limited hardware conditions, which limits the application effect of RIS. Summary of the invention
[0013] The technical problem to be solved by the present invention is to overcome the shortcomings of the prior art and provide a reconfigurable intelligent surface-assisted vehicle-mounted edge computing method and system based on reinforcement learning.
[0014] In order to solve the above technical problems, the technical solution of the present invention is as follows:
[0015] A reconfigurable intelligent surface-assisted vehicle edge computing method based on reinforcement learning, comprising:
[0016] Calculate the number of tasks arriving at the vehicle end at time t, the number of tasks processed by offloading, and the number of tasks that can be executed locally;
[0017] Calculate the task queue length in the vehicle-side buffer at time t+1;
[0018] Determine the multi-objective optimization parameters on the vehicle side and build a multi-agent deep reinforcement learning model based on the multi-agent reinforcement learning algorithm;
[0019] Based on the amount of tasks processed by the vehicle end at time t by offloading, the amount of tasks that can be executed locally, and the length of the task queue in the buffer, the multi-agent deep reinforcement learning model is iteratively trained until the model converges, and the block coordinate descent algorithm is called in each training process to optimize the optimal phase shift of the RIS;
[0020] The power allocation for the vehicle-side unloading tasks and local execution tasks is determined based on the multi-agent deep reinforcement learning model.
[0021] As a preferred solution of the reconfigurable intelligent surface-assisted vehicle-mounted edge computing method based on reinforcement learning described in the present invention, the calculation of the arrival amount of tasks on the vehicle side at time t includes:
[0022] The number of tasks arriving at the vehicle end at time t is calculated by formula 1, which is: k (t) = η k (t)×Δt, where a k(t) is the number of tasks arriving at the vehicle end at time t, η k (t) is the task arrival rate of the vehicle at time t.
[0023] As a preferred solution of the reinforcement learning-based reconfigurable intelligent surface-assisted vehicle-mounted edge computing method of the present invention, the amount of tasks processed by the computing vehicle end at time t by unloading includes:
[0024] The channel gain between the RIS and the BS is calculated by formula 2, which is: Among them, h r,b is the channel gain between RIS and BS, ρ is the reference distance d 0 = 1m path loss, d r,b is the geometric distance from RIS to BS, α r,b is the path loss exponent of the RIS-BS link, R is the Rician coefficient associated with small-scale fading, and the LoS component Defined as:
[0025]
[0026] , where λ is the carrier length, d RIS is the spacing between RIS elements, θ AoD is the departure angle of the signal from RIS to BS, k is the vehicle terminal number;
[0027] The channel gain between the vehicle end and the RIS is calculated by formula 3, which is:
[0028] in, is the channel gain between the vehicle end and RIS, is the geometric distance between the kth vehicle and the RIS at time t, α r,b is the path loss index between the vehicle and RIS, LoS component Defined as:
[0029]
[0030] in, is the arrival angle of the signal from the kth vehicle to the RIS at time t;
[0031] The signal-to-noise ratio between the vehicle end through the RIS and the BS is calculated by formula 4, which is: Among them, γ k (t) is the signal-to-noise ratio of the kth vehicle at time t, is the unloading power of the kth vehicle at time t, σ 2 is the diagonal phase shift matrix of the thermal noise power RIS, specifically: in, and β n ∈[0,1] is the amplitude of the relevant transmission coefficient and phase shift, Q=2 b , b is the control bit of the discrete degree of phase shift, and the calculated for
[0032]
[0033] The data transmission rate of the vehicle at time t is calculated by formula 5, which is: C k (t) = Wlog 2 (1+γ k (t)), where W is the bandwidth of data transmission;
[0034] The amount of tasks handled by the vehicle at time t through unloading is calculated by formula 6, and the formula 6 is: q k,o (t) = Δt × C k (t), where q k,o (t) is the amount of tasks handled by the vehicle end through unloading at time t.
[0035] As a preferred solution of the reinforcement learning-based reconfigurable intelligent surface-assisted vehicle-mounted edge computing method of the present invention, the amount of tasks that can be performed locally at the vehicle end at time t includes:
[0036] The amount of tasks that can be performed locally at the vehicle end at time t is calculated by formula 7, and the formula 7 is: q k,l (t) = Δtf k (t) / L, where q k,l (t) is the amount of tasks processed locally by the k-th vehicle at time t, L is the CPU frequency required to process 1-bit tasks, and f k (t)∈[0,F max ] is to use dynamic voltage and frequency adjustment technology to adjust the CPU frequency scheduled by the chip voltage, calculated as Among them, p k,l (t)∈[0,P max,l ] is the local execution power of the kth vehicle end at time t, and c is the effective selected capacitor.
[0037] As a preferred solution of the reconfigurable intelligent surface assisted vehicle-mounted edge computing method based on reinforcement learning of the present invention, wherein: the task queue length in the buffer zone at the vehicle end at time t+1 is calculated, including:
[0038] The task queue length in the vehicle-side buffer at time t+1 is calculated by formula eight, and the formula eight is: k (t+1)=[q k (t)-qk,o (t)-q k,l (t)]++a k (t).
[0039] As a preferred solution of the reconfigurable intelligent surface-assisted vehicle-mounted edge computing method based on reinforcement learning described in the present invention, wherein: the multi-objective optimization parameters of the vehicle end are determined including:
[0040] The multi-objective optimization parameters of the vehicle are shown in Formula 9:
[0041]
[0042] Among them, C1 and C2 are the power range limits when the vehicle performs unloading tasks and local tasks, respectively, and C3 is the limit value of the RIS phase shift.
[0043] As a preferred solution of the reinforcement learning-based reconfigurable intelligent surface assisted vehicle-mounted edge computing method of the present invention, the iterative training of the multi-agent deep reinforcement learning model until the model converges based on the amount of tasks processed by the vehicle end at time t by unloading, the amount of tasks that can be executed locally, and the length of the task queue in the buffer zone includes:
[0044] The state space of the vehicle at time t is calculated by formula 10: s = [q k (t), q k,o (t), q k,l (t), q k,o (t)+q k,l (t)-q k (t), γ k (t-1)], where γ k (t-1) is the signal-to-noise ratio of the kth vehicle at time t;
[0045] The action space of the vehicle at time t is calculated by formula 11, which is: k (t) = [p k,o (t), p k,l (t)];
[0046] The local reward of the vehicle at time t is calculated by formula 12, which is: k,l =-[w 1 (p k,o (t)+p k,l (t))+w 2 q k (t)]-G 1 -G 2 , where w1 and w2 are weight factors for reward balance, G 1 and G2 is the penalty factor;
[0047] The global reward of the vehicle at time t is calculated by Formula 13, which is:
[0048] The double-delayed deterministic policy gradient is used to replace the global critic in the multi-agent reinforcement learning algorithm, correct the policy gradient of each vehicle, and update the Q function.
[0049] As a preferred solution of the reinforcement learning-based reconfigurable intelligent surface assisted vehicle-mounted edge computing method of the present invention, the dual-delay deterministic policy gradient is used to replace the global critic in the multi-agent reinforcement learning algorithm, the policy gradient of each vehicle end is corrected, and the Q function is updated, including:
[0050] For each vehicle, the corrected policy gradient is:
[0051]
[0052] , where s=(s 1 ,s 2 ,…s K ) and a=(a 1 , a 2 , …a K ) are the total state vector and the total action vector, D is the buffer size, a k =π k (s k ) is the k-th vehicle terminal according to its own strategy π k The action of choice;
[0053] twin global critic Updated to:
[0054] in, is the target strategy, whose parameters are
[0055] The local Updated to:
[0056] in,
[0057]
[0058] As a preferred solution of the reinforcement learning-based reconfigurable intelligent surface assisted vehicle-mounted edge computing method of the present invention, wherein: the optimal phase shift of the RIS is optimized by calling the block coordinate descent algorithm in each training process, including:
[0059] At time t, the block coordinate descent algorithm is used to maximize the sum of the instantaneous modulus values of all vehicle terminals. The specific formula is:
[0060] The present invention also provides a reconfigurable intelligent surface-assisted vehicle-mounted edge computing system based on reinforcement learning, comprising:
[0061] The first calculation module is used to calculate the number of tasks arriving at the vehicle end at time t, the number of tasks processed by unloading, and the number of tasks that can be executed locally;
[0062] The second calculation module is used to calculate the task queue length in the buffer zone of the vehicle end at time t+1;
[0063] The model building module is used to determine the multi-objective optimization parameters on the vehicle side and build a multi-agent deep reinforcement learning model based on the multi-agent reinforcement learning algorithm;
[0064] A model training module, for iteratively training the multi-agent deep reinforcement learning model until the model converges based on the amount of tasks processed by the vehicle end at time t by offloading, the amount of tasks that can be executed locally, and the length of the task queue in the buffer, and calling the block coordinate descent algorithm to optimize the optimal phase shift of the RIS during each training process;
[0065] An execution module is used to determine the power allocation for offloading tasks and local execution tasks on the vehicle side based on the multi-agent deep reinforcement learning model.
[0066] The beneficial effects of the present invention are:
[0067] (1) The present invention introduces a smart reflecting surface (RIS) to assist vehicle communication and provide an additional communication path, thereby effectively solving the problem of communication interruption caused by obstacles such as buildings encountered by vehicles during driving, and improving the stability and reliability of communication.
[0068] (2) The present invention optimizes the unloading power and the local execution power, so that the system can achieve the optimal configuration of power allocation under different task arrival rates and channel conditions, thereby improving the resource utilization efficiency.
[0069] (3) The present invention optimizes the power allocation of vehicle users (VU) by adopting an improved multi-agent deep deterministic policy gradient (MADDPG) algorithm, which can better cope with resource competition among multiple users and improve the overall performance of the system.
[0070] (4) The present invention optimizes the RIS phase shift using a block coordinate descent (BCD) algorithm, which can achieve a better phase shift control effect under limited hardware conditions, solve the non-convex optimization problem, and improve the computational efficiency and accuracy of the system.
[0071] (5) The deep reinforcement learning (DRL) framework proposed in the present invention can better adapt to dynamic environments and complex channel conditions, and improve the adaptability and stability of the system in different environments. The simulation results show that the scheme proposed in the present invention is significantly superior to the traditional centralized deep deterministic policy gradient (DDPG) method and random scheme in terms of communication stability, power allocation efficiency and system performance, which verifies the superiority of the present invention in practical applications. BRIEF DESCRIPTION OF THE DRAWINGS
[0072] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.
[0073] Figure 1 A schematic diagram of the process of the reinforcement learning-based reconfigurable intelligent surface-assisted vehicle-mounted edge computing method provided by the present invention;
[0074] Figure 2 A schematic diagram of a reconfigurable smart surface-assisted vehicle edge computing scenario;
[0075] Figure 3 This is a comparison chart of the reward value convergence during the deep reinforcement learning algorithm training process;
[0076] Figure 4 The power consumption comparison chart of all users under different task arrival rates for various algorithms;
[0077] Figure 5 Comparison chart of buffer lengths of all users under different task arrival rates for various algorithms;
[0078] Figure 6 A performance change diagram of the reinforcement learning-based reconfigurable intelligent surface-assisted vehicle-mounted edge computing method provided by the present invention under different numbers of RIS elements;
[0079] Figure 7 A schematic diagram of the reinforcement learning-based reconfigurable intelligent surface-assisted vehicle-mounted edge computing system provided by the present invention. DETAILED DESCRIPTION
[0080] In order to make the content of the present invention more clearly understood, the present invention is further described in detail below according to specific implementation modes and in combination with the accompanying drawings.
[0081] Figure 1 A flow chart of a method for reconfigurable intelligent surface-assisted vehicle edge computing based on reinforcement learning provided in an embodiment of the present application. The method comprises the following steps:
[0082] Step S101: Calculate the amount of tasks arriving at the vehicle end at time t, the amount of tasks processed by unloading, and the amount of tasks that can be executed locally.
[0083] Specifically, Figure 2 As shown in the figure, a RIS-assisted VEC model under a multi-user MIMO system is constructed. The BS (i.e., base station) has M antennas and is associated with a VEC server. The vehicle user, i.e., the vehicle end, has a single antenna. Since the computing power of the VU (vehicle end) is limited, it can offload part of the task to the VEC server for processing. When the VU is blocked by an obstacle, it cannot communicate directly and effectively with the BS, so a RIS-assisted VU with N reflection elements is deployed to offload the task to the edge device (we only consider the link where the VU communicates with the BS through RIS). Here, a city intersection scenario is constructed, and vehicle users are randomly distributed at the intersection. Assuming that there are K vehicle users, represented by the set K = {1, 2, ..., K}, the VU can flexibly offload the task and execute it locally. In addition, we divide the task processing process into T discrete times of equally spaced time slots Δt, represented by the set T = {1, 2, ..., T}. In each time slot t, the vehicle randomly generates a task, where the arrival of the task follows a Poisson distribution with an arrival rate of η.
[0084] It should be noted that a quasi-static scenario is considered here, where the channel conditions remain unchanged within a time slot but vary between different time slots.
[0085] For the kth VU, assume that its task arrival rate at time slot t is η k (t), which follows the Poisson distribution, so the arrival amount a of the task of the kth VU during time slot t can be calculated k (t) is:
[0086] a k (t) = η k (t)×Δt. (1)
[0087] For the amount of tasks handled by the vehicle end at time t by unloading, in the model proposed in the embodiment, it is assumed that the coordinates of BS and RIS are (BS x , B.S. y , B.S. z ) and (RIS x ,RIS y ,RIS z), BS and RIS, RIS and vehicle have primary line-of-sight (LoS). Therefore, these communication links experience small-scale fading, which is modeled as Rician fading with pure LoS components. Since both RIS and BS are deployed at fixed locations, the RIS-BS link will always remain static, thus, we can obtain the channel gain h between RIS and BS r,b for:
[0088]
[0089] where ρ is the reference distance d 0 = 1m path loss, d r,b is the geometric distance from RIS to BS, α r,b is the path loss exponent of the RIS-BS link, and R is the Rician coefficient associated with small-scale fading. LoS components It is defined as follows:
[0090]
[0091] Where λ is the carrier length, d RIS is the spacing between RIS elements, θ AoD It is the departure angle of the signal from RIS to BS.
[0092] Similarly, the channel gain from the kth vehicle to the RIS is defined as:
[0093]
[0094] in, is the geometric distance between the kth vehicle and the RIS at time t, α k,r is the path loss index between the vehicle and RIS, LoS component Defined as:
[0095]
[0096] in, is the arrival angle of the signal from the kth vehicle to the RIS at time t.
[0097] In this embodiment, the communication link between the vehicle and the BS is completely blocked, and the vehicle can only communicate through the RIS. Therefore, we can get the signal-to-noise ratio (SNR) between the kth vehicle and the BS through the RIS at time slot t:
[0098]
[0099] BS can detect the SNR of the kth VU at time slot t, i.e., γ k(t), and then transmit it to the VU in the next time slot. In this case, the kth VU receives γ at time slot t. k (t-1), so γ k (t-1) can reflect the channel status of the kth VU at time slot t. Here is the unloading power of the kth vehicle at time t, σ 2 is the diagonal phase shift matrix of the thermal noise power RIS, specifically: Due to hardware limitations, the phase shift can only be selected from a limited set of discrete values, where and β n ∈[0,1] is taken as the magnitude of the relevant transmission coefficient and phase shift, where Q = 2 b , b is the control bit of the discrete degree of phase shift. Therefore, it can be calculated for
[0100]
[0101] When the VU chooses to offload the task to the VEC edge server associated with the BS, based on the formula, we can get the data transmission rate of the kth VU at time slot t as:
[0102] C k (t) = Wlog 2 (1+γ k (t)); (8)
[0103] Where W is the bandwidth of data transmission. Therefore, we can get the number of tasks q processed by the kth VU through offloading at time slot t. k,o (t) is:
[0104] q k,o (t) = Δt × C k (t). (9)
[0105] When the kth VU chooses local computing, the task size q that can be processed by local execution at time slot t can be obtained: k,l (t) is:
[0106] q k,l (t) = Δtf k (t) / L; (10)
[0107] Where L is the CPU frequency required to process a 1-bit task, f k (t)∈[0,F max ] is to use the dynamic voltage frequency scaling (DVFS) technology to adjust the CPU frequency scheduled by the chip voltage, calculated as:
[0108]
[0109] Among them, p k,l (t)∈[0,P max,l ] is the local execution power of the kth vehicle end at time t, and c is the effective selected capacitor.
[0110] Step S102: Calculate the task queue length in the vehicle-side buffer at time t+1.
[0111] Specifically, for the kth VU, after the task arrives, it will first be saved in the queue buffer and processed at time slot t+1. Then the queue length of the kth VU at time slot t+1 can be obtained as:
[0112] q k (t+1)=[q k (t)-q k,o (t)-q k,l (t)]++a k (t). (12)
[0113] Step S103: Determine the multi-objective optimization parameters of the vehicle side, and build a multi-agent deep reinforcement learning model based on the multi-agent reinforcement learning algorithm.
[0114] Specifically, in this embodiment, the kth VU aims to minimize the power consumed by offloading and local execution and the queue length of the buffer. Therefore, the multi-objective optimization problem of each VU can be expressed as follows:
[0115]
[0116] Among them, C1 and C2 are the power range limits when the VU performs unloading tasks and local execution tasks respectively. Due to the limitations of the RIS hardware device, the phase shift size can only be selected within a limited range, and constraint C3 limits the value of the RIS phase shift. In addition, the objective function is non-convex, so this optimization problem is difficult to solve. In order to better solve this problem, this embodiment will use a multi-agent reinforcement learning (MARL) algorithm to solve this non-convex problem, but because the RIS controller and each vehicle user are not the same type of agent, their actions and reward goals are not the same. Therefore, the BCD algorithm is first used to obtain the optimal phase shift size, and then the modified MADDPG algorithm is used to train to obtain the optimal power allocation solution.
[0117] Step S104: Based on the amount of tasks processed by the vehicle end through offloading at time t, the amount of tasks that can be executed locally, and the length of the task queue in the buffer, the multi-agent deep reinforcement learning model is iteratively trained until the model converges, and the block coordinate descent algorithm is called in each training process to optimize the optimal phase shift of RIS.
[0118] Specifically, for the multi-objective optimization problem P1, it includes the allocation of VU's unloading power and local execution power and the optimization of the intelligent reflection surface phase shift. In order to solve this problem, this embodiment proposes a multi-agent solution combined with the BCD algorithm. First, based on the relevant location information of VU, BS and RIS, the channel information of VU is calculated, where the RIS phase shift is a variable in the calculation. With the goal of maximizing the modulus of formula (8), the BCD algorithm is used to obtain the optimal phase shift. Secondly, the power allocation problem of VU is solved according to the MARL algorithm.
[0119] Regarding the details of the BCD algorithm to solve the RIS phase shift optimization, for formula (8), it is a complex vector, its modulus value can be calculated, and at time slot t, the BCD algorithm is used to maximize the instantaneous modulus value sum of all vehicles. The relevant formula is expressed as follows:
[0120]
[0121] In each iteration of training, the BCD algorithm is called to optimize the optimal phase shift of RIS. For M RIS elements, we select the best one from the Q values of each element.
[0122] Regarding the use of MARL algorithm to solve the VU power allocation problem, in this embodiment, each vehicle interacts with the environment as an agent and makes power allocation decisions through corresponding strategies. At time slot t, the agent obtains the current state s t , and then obtain the corresponding action a based on the current strategy t , by taking an action to transition to the next state s t+1 , and obtain the reward r for making this action t This process can be expressed as t =(s t ,a t ,r t ,s t+1 ), the relevant state space in this model, at time slot t, the state of the kth VU (agent) consists of the following parts:
[0123] The queue length of the buffer q k (t), the size of the task q to be offloaded k,o (t), the size of the task executed locally q k,l (t), the portion of the overflow from offloading and local processing tasks q k,o (t)+q k,l (t)-q k (t). In addition, according to formula (7), the SNR of the kth VU at time slot t is γ k (t-1), depends on h r,b and Therefore γ k (t-1) can reflect the uncertain channel condition of VU at time slot t. Therefore, the state space of the kth VU at time slot t can be expressed as:
[0124] s=[q k (t),q k,o (t),q k,l (t),q k,o (t)+q k,l (t)-q k (t),γ k (t-1)]. (15)
[0125] As described above, each VU allocates offloading power and local execution power according to the corresponding strategy, so the action space of the kth VU at time slot t is defined as:
[0126] a k (t) = [p k,o (t), p k,l (t)]; (16)
[0127] Since the deterministic strategy is adopted in this embodiment, the agent can go from 0 to P max Select any power.
[0128] In this scheme, each agent has two kinds of rewards, one is the global reward, which is used to coordinate the cooperation among agents; the other is the personal reward, which is used to help each agent explore the optimal power allocation scheme. In DRL, rewards can be set flexibly, and a good reward can improve the performance of the system. The goal is to optimize the VU offloading and local power levels and the corresponding buffer queue lengths. In this embodiment, two penalties are added, one is the penalty for the buffer queue length of the VU being greater than the threshold at time slot t, and the other is that at time slot t, the VU allocates a certain amount of power to offloading execution and local execution, resulting in the total amount of tasks processed being a certain value more than the buffer queue length, thereby generating a penalty. Therefore, the local reward of the kth VU at time slot t is:
[0129] r k,l =-[w 1 (p k,o (t)+p k,l (t))+w 2 q k (t)]-G 1 -G 2 ; (17)
[0130] Among them, w1 and w2 are weight factors for reward balance, G 1 and G 2is the penalty factor;
[0131] The global rewards are:
[0132]
[0133] This embodiment proposes a multi-agent deep reinforcement learning algorithm that can simultaneously learn how to maximize global rewards and local rewards. It mainly includes two critics, one is the global critic, which is shared among all agents and inputs the states and actions of all agents to maximize the global reward; the other is the local critic, which receives the states and actions of each agent and evaluates their respective rewards. In addition, due to the problem of overestimation and suboptimal strategy of the q function caused by function approximation error in the DDPG algorithm, this embodiment makes improvements on the basis of the MADDPG algorithm, and replaces the global critic with a double-delayed deterministic policy gradient, which can better improve the performance of the system.
[0134] Specifically, we consider a vehicle environment with K VUs (agents), each of which has a strategy of π = {π 1 , π 2 ,…,π K}. The agent's strategy π k , Q function and the twin global critic Q function The parameters are θ k ,φ k and ψ 1 , 2 For each agent, the revised policy gradient can be written as:
[0135]
[0136] Where s=(s 1 ,s 2 ,…s K ) and a=(a 1 , a 2 , …a K ) are the total state vector and the total action vector, D is the buffer size, a k =π k (s k ) is the k-th vehicle terminal according to its own strategy π k Select the action.
[0137] Afterwards, twin global critic Updated to:
[0138]
[0139] in,
[0140] is the target strategy, whose parameters are
[0141] Similarly, the local Updated to:
[0142]
[0143] in,
[0144] Step S105: Determine the power allocation for the vehicle-side to perform unloading tasks and local execution tasks based on the multi-agent deep reinforcement learning model.
[0145] Figure 3 The convergence performance of the proposed scheme and the DDPG scheme on rewards is verified. Compared with DDPG, the proposed framework is not only more stable but also has less oscillation. Since DDPG is completely centralized, it must take the observations and actions of all agents as input, which cannot effectively handle individual performance, resulting in suboptimal reward situations. The proposed framework combines multi-agent reinforcement learning (MARL) and block coordinate descent (BCD) algorithms and shows excellent convergence performance. It is able to maximize the local and global rewards of all agents simultaneously, promote cooperation between agents, and achieve optimal performance.
[0146] Figure 4 and Figure 5 It reflects the changes in power consumption and buffer length of all users under different task arrival rates. In addition, we considered different local CPU capacitors and different training methods. From the figure, we can see that the power consumption and buffer length of the vehicle basically increase with the increase of task arrival rate. Among them, the proposed algorithm consumes the least power when c=10e-29, and the buffer length is relatively small. As the capacitance increases, the power consumption and buffer length will also increase. This is because according to formulas (11) and (12), it can be obtained that the larger the capacitance, the fewer tasks are processed locally, which will increase the power consumption and buffer length. Due to the poor performance of the DDPG algorithm, it will always sacrifice a certain amount of power to ensure that there is not too much task accumulation in the buffer. However, when the capacitance and task arrival rate are both large, it cannot obtain a good power allocation strategy, resulting in a large amount of task accumulation.
[0147] Figure 6The impact of the number of RIS elements on network performance is shown. When the number of RIS elements increases, the total power consumption and buffer length will decrease. The performance of our proposed scheme is better than random RIS phase shift, because random RIS phase shift will reduce the data transmission rate during offloading, resulting in more power consumption to ensure data transmission, and will also affect the buffer. Allocating more power will not achieve a high level of information transmission rate, resulting in task accumulation.
[0148] See also Figure 7 The embodiment of the present application also provides a reinforcement learning-based reconfigurable intelligent surface-assisted vehicle-mounted edge computing system. The system includes: a first computing module 201, a second computing module 202, a model building module 203, a model training module 204, and an execution module 205.
[0149] Among them, the first calculation module 201 is used to calculate the number of tasks arriving at the vehicle end at time t, the number of tasks processed by unloading, and the number of tasks that can be executed locally.
[0150] The second calculation module 202 is used to calculate the task queue length in the vehicle-side buffer at time t+1.
[0151] The model building module 203 is used to determine the multi-objective optimization parameters of the vehicle side and build a multi-agent deep reinforcement learning model based on the multi-agent reinforcement learning algorithm.
[0152] The model training module 204 is used to iteratively train the multi-agent deep reinforcement learning model until the model converges based on the amount of tasks processed by offloading at the vehicle end at time t, the amount of tasks that can be executed locally, and the length of the task queue in the buffer, and call the block coordinate descent algorithm to optimize the optimal phase shift of RIS during each training process.
[0153] The execution module 205 is used to determine the power allocation for the vehicle-side to perform unloading tasks and local execution tasks based on the multi-agent deep reinforcement learning model.
[0154] Therefore, the technical solution of the present application introduces intelligent reflective surfaces (RIS) to assist vehicle communication and provide additional communication paths, which effectively solves the problem of communication interruption caused by obstacles such as buildings encountered by vehicles during driving, and improves the stability and reliability of communication. The present invention optimizes the unloading power and local execution power, so that the system can achieve the optimal configuration of power allocation under different task arrival rates and channel conditions, thereby improving resource utilization efficiency. By adopting the improved multi-agent deep deterministic policy gradient (MADDPG) algorithm to optimize the power allocation of vehicle users (VU), it is possible to better cope with resource competition among multiple users and improve the overall performance of the system. The present invention optimizes the RIS phase shift using the block coordinate descent (BCD) algorithm, which can still achieve a better phase shift control effect under limited hardware conditions, solve the non-convex optimization problem, and improve the computational efficiency and accuracy of the system. The deep reinforcement learning (DRL) framework proposed in the present invention can better adapt to dynamic environments and complex channel conditions, and improve the adaptability and stability of the system in different environments. The simulation results show that the scheme proposed by the present invention is significantly superior to the traditional centralized deep deterministic policy gradient (DDPG) method and random scheme in terms of communication stability, power allocation efficiency and system performance, which verifies the superiority of the present invention in practical applications. In addition to the above embodiments, the present invention may also have other implementation methods; all technical solutions formed by equivalent replacement or equivalent transformation fall within the protection scope required by the present invention.
Claims
1. A reconfigurable intelligent surface-assisted vehicle-mounted edge computing method based on reinforcement learning, characterized in that: include: Calculate the number of tasks arriving at the vehicle end at time t, the number of tasks processed by offloading, and the number of tasks that can be executed locally; Calculate the task queue length in the vehicle-side buffer at time t+1; Determine the multi-objective optimization parameters on the vehicle side and build a multi-agent deep reinforcement learning model based on the multi-agent reinforcement learning algorithm; Based on the amount of tasks processed by the vehicle end at time t by offloading, the amount of tasks that can be executed locally, and the length of the task queue in the buffer, the multi-agent deep reinforcement learning model is iteratively trained until the model converges, and the block coordinate descent algorithm is called in each training process to optimize the optimal phase shift of the RIS; The power allocation for the vehicle-side unloading tasks and local execution tasks is determined based on the multi-agent deep reinforcement learning model.
2. The method for reconfigurable intelligent surface-assisted vehicle-mounted edge computing based on reinforcement learning according to claim 1 is characterized in that: The calculation of the arrival amount of tasks on the vehicle side at time t includes: The task arrival amount of the vehicle at time t is calculated by formula 1, which is: ,in, is the number of tasks arriving at the vehicle end at time t, is the arrival rate of tasks on the vehicle side at time t.
3. The method for reconfigurable intelligent surface-assisted vehicle-mounted edge computing based on reinforcement learning according to claim 2 is characterized in that: The amount of tasks processed by the computing vehicle at time t through unloading includes: The channel gain between the RIS and the BS is calculated by formula 2, which is: ,in, is the channel gain between RIS and BS, At the reference distance The path loss when is the geometric distance from RIS to BS, is the path loss exponent of the RIS-BS link, R is the Rician coefficient associated with small-scale fading, and the LoS component Defined as: ,in, is the carrier length, is the spacing between RIS elements, is the departure angle of the signal from RIS to BS, Number the vehicle end; The channel gain between the vehicle end and the RIS is calculated by formula 3, which is: ,in, is the channel gain between the vehicle end and RIS, is the geometric distance between the kth vehicle and the RIS at time t, is the path loss index between the vehicle and RIS, LoS component Defined as: ,in, is the arrival angle of the signal from the kth vehicle to the RIS at time t; The signal-to-noise ratio between the vehicle end through the RIS and the BS is calculated by formula 4, which is: ,in, For the The signal-to-noise ratio of each vehicle at time t is: For the The unloading power of each vehicle at time t is: is the diagonal phase shift matrix of the thermal noise power RIS, specifically: ,in, and are the relevant transmission coefficients and the magnitude of the phase shift, , is the control bit of the discrete degree of phase shift, and is calculated for ; The data transmission rate of the vehicle at time t is calculated by Formula 5, which is: ,in, The bandwidth for data transmission; The amount of tasks handled by the vehicle end at time t through unloading is calculated by formula 6, which is: ,in, is the amount of tasks handled by the vehicle end through unloading at time t.
4. The method for reconfigurable intelligent surface-assisted vehicle-mounted edge computing based on reinforcement learning according to claim 3 is characterized in that: The amount of tasks that can be performed locally at the computing vehicle end at time t includes: The amount of tasks that can be performed locally at the vehicle end at time t is calculated using Formula 7, which is: ,in, For the The amount of tasks that each vehicle performs locally at time t is: The CPU frequency required to process a 1-bit task, To use dynamic voltage and frequency adjustment technology to adjust the CPU frequency scheduled by the chip voltage, it is calculated as ,in, For the The local execution power of each vehicle at time t is: To effectively select capacitors.
5. The method for reconfigurable intelligent surface-assisted vehicle-mounted edge computing based on reinforcement learning according to claim 4 is characterized in that: The calculation of the task queue length in the buffer zone at the vehicle end at time t+1 includes: The task queue length in the vehicle-side buffer at time t+1 is calculated by formula eight, which is: .
6. The method for reconfigurable intelligent surface-assisted vehicle-mounted edge computing based on reinforcement learning according to claim 5 is characterized in that: Determining the multi-objective optimization parameters of the vehicle side includes: The multi-objective optimization parameters of the vehicle are shown in Formula 9: ; Among them, C1 and C2 are the power range limits when the vehicle performs unloading tasks and local execution tasks respectively, and C3 is the limit value of RIS phase shift.
7. The method for reconfigurable intelligent surface-assisted vehicle-mounted edge computing based on reinforcement learning according to claim 6 is characterized in that: The iterative training of the multi-agent deep reinforcement learning model until the model converges based on the amount of tasks processed by the vehicle end at time t by offloading, the amount of tasks that can be executed locally, and the length of the task queue in the buffer includes: The state space of the vehicle at time t is calculated by formula 10, which is: ,in, For the The signal-to-noise ratio of the vehicle end at time t is calculated; the action space of the vehicle end at time t is calculated by formula 11, and the formula 11 is: ; The local reward of the vehicle at time t is calculated by Formula 12, which is: ,in, and are the weight factors used for reward balance, and is the penalty factor; The global reward of the vehicle at time t is calculated by Formula 13, which is: ; The double-delayed deterministic policy gradient is used to replace the global critic in the multi-agent reinforcement learning algorithm, correct the policy gradient of each vehicle, and update the Q function.
8. The method for reconfigurable intelligent surface-assisted vehicle-mounted edge computing based on reinforcement learning according to claim 7 is characterized in that: The dual-delay deterministic policy gradient is used to replace the global critic in the multi-agent reinforcement learning algorithm, to correct the policy gradient of each vehicle, and to update the Q function, including: For each vehicle, the corrected policy gradient is: ,in, and are the total state vector and the total action vector, respectively. is the buffer size, For the Each vehicle end according to its own strategy The action of choice; Updated to: ,in, , is the target strategy, whose parameters are ; No. Local Updated to: ,in, .
9. The method for reconfigurable intelligent surface-assisted vehicle-mounted edge computing based on reinforcement learning according to claim 8 is characterized in that: The method of calling the block coordinate descent algorithm to optimize the optimal phase shift of RIS during each training process includes: At time t, the block coordinate descent algorithm is used to maximize the sum of the instantaneous modulus values of all vehicle terminals. The specific formula is: .
10. A reinforcement learning-based reconfigurable intelligent surface-assisted vehicle-mounted edge computing system according to any one of claims 1 to 9, characterized in that: include: The first calculation module is used to calculate the number of tasks arriving at the vehicle end at time t, the number of tasks processed by unloading, and the number of tasks that can be executed locally; The second calculation module is used to calculate the task queue length in the buffer zone of the vehicle end at time t+1; The model building module is used to determine the multi-objective optimization parameters on the vehicle side and build a multi-agent deep reinforcement learning model based on the multi-agent reinforcement learning algorithm; A model training module, for iteratively training the multi-agent deep reinforcement learning model until the model converges based on the amount of tasks processed by the vehicle end at time t by offloading, the amount of tasks that can be executed locally, and the length of the task queue in the buffer, and calling the block coordinate descent algorithm to optimize the optimal phase shift of the RIS during each training process; An execution module is used to determine the power allocation for offloading tasks and local execution tasks on the vehicle side based on the multi-agent deep reinforcement learning model.
