Reinforcement learning based beamforming method for vehicle network edge computing

Abstract

The application discloses a beamforming method for vehicle network edge computing based on reinforcement learning, which comprises the following steps: step 1, obtaining current environment information, including vehicle position and antenna information, RSU position and antenna information; step 2, performing channel estimation; step 3, performing traffic density evaluation: according to the current regional vehicle position, each vehicle is divided into different traffic density levels; step 4, joint optimization decision stage: according to the traffic density information and the channel condition, the cloud service platform decides the transmitting beam of each vehicle, the receiving beam of each RSU, the selected RSU of each vehicle, the task uploading proportion of each vehicle and the allocation of the computing resources at the RSU. The application dynamically adjusts the vehicle transmitting beam, the RSU receiving beam, the vehicle task sending proportion, the scheduling relationship and the allocation of the edge server computing resources at the RSU, so as to further reduce the perception task processing delay of the vehicle in the high traffic density area and improve the throughput of the whole area.

Description

Technical Field

[0001] This invention belongs to the field of wireless communication technology, and more specifically, relates to a beamforming method for edge computing of vehicle networks based on reinforcement learning. Background Technology

[0002] With the booming development of emerging services such as the Internet of Things (IoT) and mobile internet, the massive increase in the number of wirelessly connected devices has posed unprecedented challenges to communication systems beyond fifth-generation / sixth-generation (5G / 6G). In the 5G IoT scenario, the Internet of Vehicles (IoV), as a typical network architecture, has attracted considerable interest from researchers due to its network structure and resource allocation methods for vehicle computing problems.

[0003] The development of future intelligent transportation systems hinges on the timely and reliable processing of vehicle information. Autonomous vehicles are considered a promising approach, capable of alleviating traffic congestion and improving traffic efficiency. In vehicle-to-everything (V2X) scenarios, autonomous vehicles generate massive amounts of perception data. While vehicles themselves can process this data, their limited computing power necessitates the use of edge-assisted servers to further reduce processing latency. However, the large volume of data from perception tasks requires time to transmit to edge nodes. If the acquired channel rate is too low, the advantages of edge computing cannot be realized.

[0004] To improve channel speeds, the use of high-bandwidth millimeter waves was proposed in the 5G era. However, while millimeter waves offer high bandwidth, they also suffer from significant attenuation. Therefore, the performance of millimeter waves degrades rapidly in the presence of significant obstacles. To overcome this challenge, a common approach is to deploy millimeter waves in conjunction with array antennas, and to utilize beamforming technology to further enhance millimeter wave performance.

[0005] In traditional edge computing scenarios, communication resource considerations typically don't involve the deployment of array antennas; instead, omnidirectional antennas are simply used for information transmission. In this case, the focus is primarily on spectrum allocation. However, in real-world scenarios with a large number of users, allocating a separate spectrum resource to each user would result in wasted communication resources. In contrast, sharing a single millimeter-wave band can significantly reduce spectrum waste. Therefore, deploying array antennas to improve spectrum utilization efficiency is particularly important in such scenarios.

[0006] Several papers have already explored the joint design of millimeter-wave and edge computing. The paper "Mobile Edge ComputingMeets mmWave Communications: Joint Beamforming and Resource Allocation for System Delay Minimization" proposes using millimeter-wave and beamforming technologies to reduce processing latency in edge computing. However, the scenario described in the paper is overly idealistic, failing to consider the actual traffic density in connected vehicles, and the scenario is also too simplistic. The paper "Reconfigurable Intelligent Surface-Aided Mobile Edge Computing: From Optimization-Based to Location-Only Learning-Based Solutions" introduces a Reflecting Intelligent Surface (RIS) to further reduce latency in the aforementioned scenario, but it still suffers from the problems mentioned above. Current research on the joint design of millimeter-wave beamforming and edge computing is limited to this area. Summary of the Invention

[0007] The purpose of this invention is to overcome the shortcomings of the prior art and provide a beamforming method based on reinforcement learning for edge computing of vehicle networks. This method dynamically adjusts the vehicle transmit beam, RSU receive beam, vehicle task transmission ratio, scheduling relationship, and edge server computing resource allocation at the RSU, thereby further reducing the processing latency of vehicle perception tasks in high traffic density areas and improving the throughput of the entire area.

[0008] The objective of this invention is achieved through the following technical solution: a beamforming method for edge computing of vehicle networks based on reinforcement learning, comprising the following steps:

[0009] Step 1: Obtain current environmental information, including vehicle location and antenna information, RSU location and antenna information;

[0010] Step 2: Perform channel estimation: Vehicles and RSUs upload channel data to the cloud service platform. The centralized channel estimation algorithm and resources are used to perform channel estimation between vehicles and RSUs, generate channel information from each vehicle to each RSU, and save this channel information on the corresponding RSUs.

[0011] Step 3: Conduct traffic flow density assessment: Classify each vehicle into different traffic flow density levels based on the current vehicle location in the area;

[0012] Step 4, Joint Optimization Decision Stage: Based on traffic density information and channel conditions, the cloud service platform determines the transmit beam for each vehicle, the receive beam for each RSU, the RSU selected by each vehicle, the task upload ratio for each vehicle, and the allocation of computing resources at the RSU.

[0013] The specific implementation method of step 4 is as follows: all vehicles in the area share a single millimeter-wave frequency band. Considering that the computing power of the vehicles themselves may be insufficient, vehicle i may optionally upload part of the task to the edge server of the RSU after receiving the perception task. The vehicle set is represented as follows. The edge server set is An initial signal s is generated at vehicle i. i The initial signal satisfies E{|s i | 2} = 1, meaning the initial signal power is 1; the initial signal, after passing through the baseband processing matrix and analog processing matrix, is transmitted via the transmitting antenna, as shown below:

[0014] x i =f RF,i *f BB.i *s i

[0015] Where f RF,i f represents the simulation processing matrix of the vehicle. BB.i This represents the digital processing matrix of the vehicle; the edge server distinguishes each signal based on the channel matrix of all vehicles in the area: for vehicle i, all signals except its own are considered interference, so the signal received by the j-th RSU from the i-th vehicle is represented as:

[0016]

[0017] in This represents the digital reception matrix for vehicle i at the j-th edge server. This represents the simulated reception matrix of the j-th edge server; β represents the channel matrix from vehicle i to the j-th edge server. i′,j This represents the scheduling relationship between vehicle i′ and the j-th edge server. If β i′,j =0 indicates that vehicle i′ does not select edge server j for auxiliary task processing, β i′,j =1 indicates that vehicle i′ selects edge server j for auxiliary task processing;

[0018] Let the power of the interference term be expressed as:

[0019]

[0020] Where σ 2Let ||||2| represent the Gaussian white noise power, and ||||2| represent the L2 norm.

[0021] Useful power is expressed as:

[0022]

[0023] The channel rates that vehicle i can currently access are:

[0024]

[0025] Where B is the bandwidth of the frequency band used by the vehicle;

[0026] The final latency consumed by a vehicle to complete a perception task in the edge server is the maximum of the total latency from uploading the task to obtaining the result of the uploaded task and the vehicle's local computation latency; assuming that each vehicle generates the same amount of perception processing tasks at the beginning of each time slot, with a size of τ. i If bits are involved, then the latency for vehicle i to process the perception task is:

[0027]

[0028] The vehicle's local computing power is represented here by C. local To represent, the computing resources allocated by edge server j to vehicle i are represented by C. i,j Represents; θ i ∈[0,1] represents the proportion of tasks that the vehicle chooses to process locally;

[0029] The goal of beamforming strategy is to further reduce the processing latency of vehicle perception tasks within a region while prioritizing vehicles in high-traffic-density areas, which can be expressed by the formula:

[0030]

[0031] pri k This indicates that the vehicle is currently in a priority position, α. k These are the different weights corresponding to the three priorities, n k Then T represents the total number of vehicles under the kth priority level. t Given a latency threshold, the optimization objective is to maximize the sum of the weighted average task processing latency achieved by vehicles with different priorities.

[0032] To solve the beamforming policy objective, the entire environment is treated as a large intelligent agent, and agent modeling and solving are performed on a cloud platform. The state, action, and reward model of the beamforming policy algorithm for edge computing in vehicle networks is as follows:

[0033] (1) State space: The state of the agent at step t is defined as follows:

[0034] st =[t1,t2,...t N ,a t-1 ]

[0035] Where [t1,t2,...,t] N ] represents the normalized value of the latency of the perception task for each vehicle in the current state;

[0036] (2) Action Space: The action of each exploration by the agent is defined as the analog processing matrix FRF, digital baseband processing matrix FBB, vehicle task upload ratio θ, RSU server computing resource allocation ratio C, vehicle and RSU scheduling matrix β, and analog processing matrix WRF and digital baseband processing matrix WBB at the RSU, specifically expressed as:

[0037] a t =[FRF,FBB,θ,C,β,WRF,WBB];

[0038] (3) Reward value: The reward value of the agent system is defined as the objective function of the optimization problem. The agent will continuously learn in the direction of increasing reward, that is, maximize the delay achievement rate under the condition of priority, expressed as:

[0039]

[0040] Where A is a pre-defined weight;

[0041] The specific reinforcement learning algorithm process is as follows:

[0042] Step 1: Randomly initialize the online current actor network parameters, target actor network parameters, current critic network parameters, target critic network parameters, set the decay factor, soft update rate, sample set for batch gradient descent, maximum number of iterations, maximum training step size, random noise function, and memory bank.

[0043] Step 2: Initialize the analog processing matrix, digital baseband matrix, and sensing task upload ratio of all vehicles; initialize the analog processing matrix, digital baseband matrix, and computing resource allocation matrix at the RSU; initialize the scheduling matrix between vehicles and RSUs.

[0044] Step 3: The agent observes the current state s i And by combining the current strategy and random noise, action a is obtained. t =μ(s) t |θ μ )+noise,μ(s t |θ μ ) represents the output obtained by the current actor network based on the current state, θμ The parameters for the current actor network are then used; proceed to Step 4.

[0045] Step 4: Record the vehicles that expect to receive signals on each RSU according to the scheduling matrix between vehicles and RSUs;

[0046] Step 5: Determine whether the vehicle power constraint and time delay requirement constraint are met after the current action is completed. If not, proceed to Step 3; otherwise, proceed to Step 6.

[0047] Step 6: Based on the current environment status... t The reward value r is calculated. t And update to get the state s for the next time step. t+1 At the same time, the empirical sample (s) t ,a t ,r t ,s t+1 Store in the memory bank; if the memory bank is full, proceed to Step 7, otherwise proceed to Step 3;

[0048] Step 7: Randomly select a batch of data from the memory bank and calculate the target value based on this batch of data:

[0049] y t =r t +γQ′(s t+1 ,μ′(μ′(s t+1 |θ μ′ )|θ Q′ ))

[0050] γ is the discount factor, Q′(s t+1 ,μ′(μ′(s t+1 |θ μ′ )|θ Q′ )) represents the Q-value of the target Critic network output, μ′(s t+1 |θ μ′ ) indicates that the target actor network is in state s t+1 Output the data at that time; after obtaining the target value of all data, proceed to Step 8;

[0051] Step 8: Based on minimizing the loss function

[0052]

[0053] To update the parameters of the current Actor network, The gradient operator is then used; proceed to Step 9.

[0054] Step 9, according to Update the parameters of the current Critic network and proceed to Step 10;

[0055] Step 10: Perform a soft update of the target network and the current network, namely:

[0056] θ Q′ =τθ Q +(1-τ)θ Q′ θ μ′ =τθ μ +(1-τ)θ μ′

[0057] τ is the soft update rate; after updating the network parameters, if the maximum training step size has been reached, proceed to Step 2; if the maximum number of iterations has been reached, proceed to Step 11; otherwise, proceed to Step 3.

[0058] Step 11: Output the strategy corresponding to the highest instantaneous reward as the optimal solution to the problem in the current scenario.

[0059] The beneficial effects of this invention are:

[0060] 1. This invention proposes a beamforming strategy for edge computing in vehicular networks based on reinforcement learning. The aim is to further reduce the processing latency of vehicle perception tasks in high-traffic-density areas and improve the overall throughput by dynamically adjusting the vehicle transmit beam, RSU receive beam, vehicle task transmission ratio, scheduling relationships, and edge server computing resource allocation at the RSU. This invention employs a reinforcement learning algorithm that can finely and jointly adjust the above parameters according to the needs under different traffic density conditions. Ultimately, the algorithm can provide an optimal beamforming strategy and computing resource allocation scheme to achieve optimal performance.

[0061] 2. Comprehensive consideration of traffic flow density characteristics and resource allocation: Under the premise that vehicles share a single frequency band for transmission, the direction and characteristics of the beams are controlled by adjusting the beam matrix, and the allocation of computing resources is optimized to tilt resources towards high traffic density areas. This method takes into account traffic flow density characteristics, providing greater mobility and flexibility. It aims to reduce system costs while improving system efficiency and performance, providing an innovative solution for handling high-traffic sensing services in edge computing scenarios. Attached Figure Description

[0062] Figure 1 This is a schematic diagram of a scenario for the present invention;

[0063] Figure 2 This is a flowchart of the beamforming method of the present invention;

[0064] Figure 3 This is a schematic diagram of the DDPG training process. Detailed Implementation

[0065] This invention provides an overall technical solution for communication and perception task processing of autonomous vehicles in narrowband millimeter-wave scenarios. To explore the impact of millimeter-wave beams on communication, a hybrid millimeter-wave beam is modeled as... Figure 1 As shown, the solution of this invention mainly consists of three modules: a traffic flow density estimation module, a channel estimation module, and an optimization problem solving module. In this solution, all autonomous vehicles share a single millimeter-wave communication frequency band and are equipped with a ULA horizontal array antenna. Vehicles periodically perceive the environment using their own cameras and select roadside units as auxiliary processing units for the perception task. The roadside units allocate edge server computing resources to the vehicles based on their location information and return the processing results to the vehicles. In low-traffic-density areas, vehicles can adjust their transmission beams to relinquish a certain proportion of communication resources, thereby increasing the channel rate for vehicles in high-traffic-density areas and reducing task processing latency. The entire process encompasses three key steps: traffic flow density estimation, channel estimation, and optimization problem solving. The optimal decision is output through a deep deterministic reinforcement learning (DDPG) algorithm to optimize task processing latency.

[0066] The technical solution of the present invention will be further described below with reference to the accompanying drawings.

[0067] like Figure 2 As shown, the present invention provides a beamforming method for edge computing of vehicle networks based on reinforcement learning, comprising the following steps:

[0068] Step 1: Obtain current environmental information, including vehicle location and antenna information, RSU location and antenna information;

[0069] Step 2: Perform channel estimation: Vehicles and RSUs upload channel data to the cloud service platform. The centralized channel estimation algorithm and resources are used to perform channel estimation between vehicles and RSUs, generate channel information from each vehicle to each RSU, and save this channel information on the corresponding RSUs.

[0070] Step 3: Conduct traffic flow density assessment: Classify each vehicle into different traffic flow density levels based on the current vehicle location in the area;

[0071] Step 4, Joint Optimization Decision Stage: Based on traffic density information and channel conditions, the cloud service platform determines the transmit beam for each vehicle, the receive beam for each RSU, the RSU selected by each vehicle, the task upload ratio for each vehicle, and the allocation of computing resources at the RSU, and outputs the optimal strategy.

[0072] The specific implementation method of step 4 is as follows: The scenario considered is a narrowband millimeter-wave scenario, where all vehicles in the area share a single millimeter-wave frequency band. Considering the potential inadequacy of the vehicle's own computing power, vehicle i, after receiving the perception task, may optionally upload a portion of the task to the RSU's edge server. The vehicle set is represented as... The edge server set is An initial signal s is generated at vehicle i. i The initial signal satisfies E{|s i | 2} = 1, meaning the initial signal power is 1; the initial signal, after passing through the baseband processing matrix and analog processing matrix, is transmitted via the transmitting antenna, as shown below:

[0073] x i =f RF,i *f BB.i *s i

[0074] Where f RF,i f represents the simulation processing matrix of the vehicle. BB.i This represents the digital processing matrix of the vehicle; the edge server distinguishes each signal based on the channel matrix of all vehicles in the area: for vehicle i, all signals except its own are considered interference, so the signal received by the j-th RSU from the i-th vehicle is represented as:

[0075]

[0076] in This represents the digital reception matrix for vehicle i at the j-th edge server. This represents the simulated reception matrix of the j-th edge server; Let β represent the channel matrix from vehicle i to the j-th edge server; considering that a vehicle typically only selects one edge server for task offloading, let β i′,j This represents the scheduling relationship between vehicle i′ and the j-th edge server. If β i′,j =0 indicates that vehicle i′ does not select edge server j for auxiliary task processing, β i′,j =1 indicates that vehicle i′ selects edge server j for auxiliary task processing;

[0077] Let the power of the interference term be expressed as:

[0078]

[0079] Where σ 2 Let ||||2| represent the Gaussian white noise power, and ||||2| represent the L2 norm.

[0080] Useful power is expressed as:

[0081]

[0082] The channel rates that vehicle i can currently access are:

[0083]

[0084] Where B is the bandwidth of the frequency band used by the vehicle;

[0085] The final latency consumed by a vehicle to complete a perception task in the edge server is the maximum of the total latency from uploading the task to obtaining the result of the uploaded task and the vehicle's local computation latency; assuming that each vehicle generates the same amount of perception processing tasks at the beginning of each time slot, with a size of τ. i If bits are involved, then the latency for vehicle i to process the perception task is:

[0086]

[0087] The vehicle's local computing power is represented here by C. local To represent, the computing resources allocated by edge server j to vehicle i are represented by C. i,j This means that the computing resources allocated to edge computing cannot exceed the sum of its own computing resources; θ i ∈[0,1] represents the proportion of local processing tasks selected by the vehicle. The latency of the vehicle completing one perception task is the maximum of the latency consumed by the local processing task and the latency of uploading to the edge server for auxiliary calculation and obtaining the result. Since the task volume of the edge server sending the result is generally very small, it is ignored here.

[0088] The goal of beamforming strategy is to further reduce the processing latency of vehicle perception tasks within a region while prioritizing vehicles in high-traffic-density areas, which can be expressed by the formula:

[0089]

[0090] pri k This indicates that the vehicle is currently in a priority position, α. k These are the different weights corresponding to the three priorities, n k Then T represents the total number of vehicles under the kth priority level. t Given a latency threshold, the optimization objective is to maximize the sum of the weighted average task processing latency achieved by vehicles with different priorities.

[0091] To solve the beamforming policy objective, the entire environment is treated as a large intelligent agent, and agent modeling and solving are performed on a cloud platform. The state, action, and reward model of the beamforming policy algorithm for edge computing in vehicle networks is as follows:

[0092] (1) State space: The state of the agent at step t is defined as follows:

[0093] s t =[t1,t2,...t N ,a t-1 ]

[0094] Where [t1,t2,...,t] N ] represents the normalized value of the latency of the perception task for each vehicle in the current state;

[0095] (2) Action Space: The action of each exploration by the agent is defined as the analog processing matrix FRF, digital baseband processing matrix FBB, vehicle task upload ratio θ, RSU server computing resource allocation ratio C, vehicle and RSU scheduling matrix β, and analog processing matrix WRF and digital baseband processing matrix WBB at the RSU, specifically expressed as:

[0096] a t =[FRF,FBB,θ,C,β,WRF,WBB];

[0097] (3) Reward value: The reward value of the agent system is defined as the objective function of the optimization problem. The agent will continuously learn in the direction of increasing reward, that is, maximize the delay achievement rate under the condition of priority, expressed as:

[0098]

[0099] Where A is a pre-defined weight;

[0100] like Figure 3 As shown, the specific reinforcement learning algorithm flow is as follows:

[0101] Step 1: Randomly initialize the online current actor network parameters, target actor network parameters, current critic network parameters, target critic network parameters, set the decay factor, soft update rate, sample set for batch gradient descent, maximum number of iterations, maximum training step size, random noise function, and memory bank.

[0102] Step 2: Initialize the analog processing matrix, digital baseband matrix, and perception task upload ratio of all vehicles; initialize the analog processing matrix, digital baseband matrix, and computing resource allocation matrix at the RSU; initialize the scheduling matrix between vehicles and RSUs; obtain the initial state s. t ;

[0103] Step 3: The agent observes the current state s i And by combining the current strategy and random noise, action a is obtained. t =μ(s) t |θ μ)+noise,μ(s t |θ μ ) represents the output obtained by the current actor network based on the current state, θ μ The parameters for the current actor network are then used; proceed to Step 4.

[0104] Step 4: Record the vehicles that expect to receive signals on each RSU according to the scheduling matrix between vehicles and RSUs;

[0105] Step 5: Determine whether the vehicle power constraint and time delay requirement constraint are met after the current action is completed. If not, proceed to Step 3; otherwise, proceed to Step 6.

[0106] Step 6: Based on the current environment status... t The reward value r is calculated. t and perform action a in the environment. t Update to get the next time step state s t+1 At the same time, the empirical sample (s) t ,a t ,r t ,s t+1 Store in the memory bank; if the memory bank is full, proceed to Step 7, otherwise proceed to Step 3;

[0107] Step 7: Randomly select a batch of data from the memory bank and calculate the target value based on this batch of data:

[0108] y t =r t +γQ′(s t+1 ,μ′(μ′(s t+1 |θ μ′ )|θ Q′ ))

[0109] γ is the discount factor, Q′(s t+1 ,μ′(μ′(s t+1 |θ μ′ )|θ Q′ )) represents the Q-value of the target Critic network output, μ′(s t+1 |θ μ′ ) indicates that the target actor network is in state s t+1 Output the data at that time; after obtaining the target value of all data, proceed to Step 8;

[0110] Step 8: Based on minimizing the loss function

[0111]

[0112] To update the parameters of the current Actor network, The gradient operator is then used; proceed to Step 9.

[0113] Step 9, according to Update the parameters of the current Critic network and proceed to Step 10;

[0114] Step 10: Perform a soft update of the target network and the current network, namely:

[0115] θ Q′ =τθ Q +(1-τ)θ Q′ θ μ′ =τθ μ +(1-τ)θ μ′

[0116] τ is the soft update rate; after updating the network parameters, if the maximum training step size has been reached, proceed to Step 2; if the maximum number of iterations has been reached, proceed to Step 11; otherwise, proceed to Step 3.

[0117] Step 11: Output the strategy corresponding to the highest instantaneous reward as the optimal solution to the problem in the current scenario.

[0118] Those skilled in the art will recognize that the embodiments described herein are intended to help the reader understand the principles of the invention, and should be understood that the scope of protection of the invention is not limited to such specific statements and embodiments. Those skilled in the art can make various other specific modifications and combinations based on the technical teachings disclosed in this invention without departing from the spirit of the invention, and these modifications and combinations are still within the scope of protection of this invention.

Claims

1. A beamforming method for vehicle network edge computing based on reinforcement learning, characterized in that, Includes the following steps: Step 1: Obtain current environmental information, including vehicle location and antenna information, RSU location and antenna information; Step 2: Perform channel estimation: Vehicles and RSUs upload channel data to the cloud service platform. The centralized channel estimation algorithm and resources are used to perform channel estimation between vehicles and RSUs, generate channel information from each vehicle to each RSU, and save this channel information on the corresponding RSUs. Step 3: Conduct traffic flow density assessment: Classify each vehicle into different traffic flow density levels based on the current vehicle location in the area; Step 4, Joint Optimization Decision Stage: Based on traffic density information and channel conditions, the cloud service platform determines the transmit beam for each vehicle, the receive beam for each RSU, the RSU selected by each vehicle, the task upload ratio for each vehicle, and the allocation of computing resources at the RSU. Specifically, all vehicles in the area share a single millimeter-wave frequency band. After receiving a sensing task, vehicle i uploads a portion of the task to the edge server of the RSU. The vehicle set is represented as... The edge server set is An initial signal is generated at vehicle i. The initial signal satisfies That is, the initial signal power is 1; the initial signal is transmitted through the transmitting antenna after passing through the baseband processing matrix and the analog processing matrix, and is expressed as: ； where represents the analog processing matrix of the vehicle, represents the digital processing matrix of the vehicle; the edge server distinguishes each signal according to the channel matrix of all vehicles in the area: for vehicle i, except for its own signal, the rest of the signals are considered as interference terms, then the signal received by the jth RSU of the ith vehicle is represented as: ； in This represents the digital reception matrix for vehicle i at the j-th edge server. This represents the simulated reception matrix of the j-th edge server; This represents the channel matrix from vehicle i to the j-th edge server; Indicates vehicle The scheduling relationship with the j-th edge server, if This indicates a vehicle Do not select edge server j for auxiliary task processing. This indicates a vehicle Select edge server j for auxiliary task processing; Let the power of the interference term be expressed as: ； wherein is the Gaussian white noise power, denotes the two-norm; Useful power is expressed as: ； The channel rates that vehicle i can currently access are: ； wherein is the bandwidth of the vehicle usage frequency band; The final latency consumed by a vehicle in the edge server to complete a perception task is the maximum of the total latency of uploading the task and obtaining the result of the uploaded task and the latency of local computation by the vehicle; assuming that each vehicle generates the same amount of perception processing tasks at the beginning of each time slot, the size of which is bits, then the latency of vehicle i to process the perception task is: ； The computing power local to the vehicle is denoted here by and the computing resources allocated to the vehicle i by the edge server j by ; is the proportion of the local processing of the task by the vehicle. The goal of beamforming strategy is to further reduce the processing latency of vehicle perception tasks within a region while prioritizing vehicles in high-traffic-density areas, which can be expressed by the formula: ； This indicates that the vehicle is currently in a priority position. These are the different weights corresponding to the three priorities. This represents the total number of vehicles at the kth priority level. Given a latency threshold, the optimization objective is to maximize the sum of the weighted average task processing latency achieved by vehicles with different priorities. To solve the beamforming policy objective, the entire environment is treated as a large intelligent agent, and agent modeling and solving are performed on a cloud platform. The state, action, and reward model of the beamforming policy algorithm for edge computing in vehicle networks is as follows: (1) State space: The state of the agent at step t is defined as follows: ； in The normalized value representing the latency of the perception task for each vehicle in the current state; (2) Action space: The action of the agent in each exploration is defined as the simulation processing matrix of each vehicle. Digital baseband processing matrix Vehicle task upload ratio RSU server computing resource allocation ratio Vehicle and RSU scheduling matrix and the simulation processing matrix at RSU and digital baseband processing matrix Specifically, it is expressed as: ； (3) Reward value: The reward value of the agent system is defined as the objective function of the optimization problem. The agent will continuously learn in the direction of increasing reward, that is, maximize the delay achievement rate when there is a priority, which is expressed as: ； in These are pre-defined weights; The specific reinforcement learning algorithm process is as follows: Step 1: Randomly initialize the online current actor network parameters, target actor network parameters, current critic network parameters, target critic network parameters, set the decay factor, soft update rate, sample set for batch gradient descent, maximum number of iterations, maximum training step size, random noise function, and memory bank. Step 2: Initialize the simulation processing matrix, digital baseband matrix and perception task upload ratio of all vehicles; Initialize the analog processing matrix, digital baseband matrix, and computing resource allocation matrix at the RSU; initialize the vehicle and RSU scheduling matrix; Step 3: The agent observes the current state. The action is obtained by combining the current strategy and random noise. , This represents the output of the current actor network based on its current state. The parameters for the current actor network are then used; proceed to Step 4. Step 4: Record the vehicles that expect to receive signals on each RSU according to the scheduling matrix between vehicles and RSUs; Step 5: Determine whether the vehicle power constraint and time delay requirement constraint are met after the current action is completed. If not, proceed to Step 3; otherwise, proceed to Step 6. Step 6: Based on the current environment status Calculate the reward value And update to get the state at the next moment. At the same time, the empirical samples Stored in memory; If the memory bank is full, proceed to Step 7; otherwise, proceed to Step 3. Step 7: Randomly select a batch of data from the memory bank and calculate the target value based on this batch of data: ； It is a discount factor. This represents the Q-value output by the target Critic network. Indicates the target actor network in state Output the data at that time; after obtaining the target value of all data, proceed to Step 8; Step 8: Based on minimizing the loss function ； To update the parameters of the current Actor network, The gradient operator is then used; proceed to Step 9. Step 9, according to Update the parameters of the current Critic network and proceed to Step 10; Step 10: Perform a soft update of the target network and the current network, namely: ， ； It is the soft update rate; after updating the network parameters, if the maximum training step size has been reached, proceed to Step 2; if the maximum number of iterations has been reached, proceed to Step 11; otherwise, proceed to Step 3. Step 11: Output the strategy corresponding to the highest instantaneous reward as the optimal solution to the problem in the current scenario.

Images