A method for computing offloading and resource allocation for a car-road cooperative system

Abstract

A computational offloading and resource allocation method for vehicle-road cooperative systems includes: establishing a communication model and a machine learning (ML) task model based on the vehicle-road cooperative system framework; establishing a computational model based on the communication model and the ML task model; modeling the computational offloading and resource allocation problem into a Markov decision model based on the computational model; introducing a multi-action output policy network into the SAC algorithm based on the Markov decision model; allowing the agent to continuously interact with the environment and obtain rewards, ultimately converging to a computational offloading and resource allocation strategy that maximizes the total reward; executing the ML task according to the optimal computational offloading and resource allocation strategy, and transmitting the processing results to an MEC server; constructing a digital twin vehicle on the MEC server; and then using the digital twin vehicle to predict the vehicle's behavior to assist in autonomous driving. This invention maximizes the processing accuracy of the ML task while reducing the total processing latency and energy consumption.

Description

Technical Field

[0001] This invention relates to the field of wireless communication, and in particular to a computational offloading and resource allocation method for vehicle-road cooperative systems. Background Technology

[0002] Driven by the Internet of Vehicles (IoV) and Artificial Intelligence (AI), intelligent transportation systems have significantly improved traffic efficiency in cities, providing comprehensive traffic information services. Among these, vehicle-road cooperative systems are one of the best solutions for realizing intelligent transportation. This system utilizes the intelligent sensing devices of roadside units (RSUs) to provide cooperative intelligent driving services to vehicles. Through the interaction between vehicles and RSUs, vehicle-road cooperative systems can provide more convenient transportation services.

[0003] In vehicle-road cooperative systems, machine learning (ML) methods are needed to fuse video and radar signals from the RSU (Roadside Unit) and vehicle sides. Then, digital twin (DT) technology can be used to build a digital model of the vehicle, enabling optimal driving decisions through a combination of virtual and real-world data to assist in autonomous driving. DT is a virtual transformation technology that uses ML and modeling techniques to process and analyze sensor data, build digital models of physical objects, and predict their behavior and performance. However, processing sensor data using ML methods is a computationally intensive task. Vehicle computing resources cannot meet the computational requirements of these tasks, necessitating the use of other powerful computing platforms to complete the ML tasks.

[0004] Fortunately, cloud computing and mobile edge computing (MEC) platforms can provide computing services for vehicle-to-everything (V2X) systems. For example, data and service requests generated by vehicles can be offloaded to cloud computing centers for processing via the RSU (Roadside Unit). However, cloud computing centers, being far from end users, cannot meet the real-time response requirements of certain vehicle tasks. As an extension of cloud computing, MEC can effectively address the latency issues inherent in cloud computing. MEC servers, deployed near the RSU, can provide computing resources to vehicles, significantly reducing computing service latency. Therefore, sensor data can be offloaded to MEC servers and cloud servers for collaborative processing. After the sensor data processing is complete, the results are sent to the MEC server, thereby building a digital twin vehicle on the MEC server.

[0005] To build a digital twin vehicle on the MEC server to assist autonomous driving, it is necessary to fuse sensor data from both the vehicle and RSU sides using ML algorithms. Specifically, there are two parts of ML tasks to process: raw sensor data from the RSU sensors and data from the vehicle itself. Due to the differences in computing resources among the vehicle, MEC server, and cloud server, only three ML models with increasing complexity can be used. These include a simple ML model deployed on each vehicle, a medium-complexity ML model deployed on the MEC server, and a high-complexity ML model deployed on the cloud server. Therefore, processing the same ML task on the intelligent vehicle, MEC server, or cloud server will yield different inference accuracies. Furthermore, due to the limited resources of the MEC server, it is impossible to process all ML tasks within a specified time, nor can it meet the inference accuracy requirements of all tasks. Therefore, it is crucial to rationally manage the computing resources among the intelligent vehicle, MEC server, and cloud server to improve the system's task processing performance. Summary of the Invention

[0006] The main objective of this invention is to overcome the shortcomings of existing vehicle-to-infrastructure (V2I) systems in terms of processing performance. It proposes a computational offloading and resource allocation method for V2I cooperative systems. Based on the time-varying nature of wireless channels and sensor data, the optimization problem can be formulated as a Markov decision process. Then, a deep reinforcement learning Soft Actor-Critic (SAC) algorithm is employed to jointly optimize the processing accuracy, processing latency, and processing energy consumption of the V2I task.

[0007] The present invention adopts the following technical solution:

[0008] A computational offloading and resource allocation method for vehicle-road cooperative systems, characterized by comprising:

[0009] A communication model and an ML task model are established based on the vehicle-road cooperative system framework, wherein the communication model is used to calculate the vehicle uplink communication rate.

[0010] A computational model is established based on the communication model and the ML task model. This computational model is used to calculate the relevant parameters for the vehicle, RSU, MEC server and cloud server to process ML tasks respectively.

[0011] Based on the computational model, the computational offloading and resource allocation problem is modeled as a Markov decision model, and a multi-action output policy network is introduced into the SAC algorithm according to the Markov decision model.

[0012] The intelligent agent is allowed to continuously interact with the environment and receive rewards, eventually converging to a computational offloading and resource allocation strategy that yields the maximum total reward as the optimal computational offloading and resource allocation strategy.

[0013] The ML task is executed according to the optimal computational offloading and resource allocation strategy, and the processing result is transmitted to the MEC server. A digital twin vehicle is built on the MEC server, and then the digital twin vehicle predicts the behavior of the vehicle to assist in the autonomous driving of the vehicle.

[0014] Preferably, the vehicle-road cooperative framework includes: multiple autonomous vehicles equipped with sensors, RSUs equipped with MEC servers and sensors, and a cloud server connected to the RSUs via a fiber optic network.

[0015] Preferably, the MEC server acts as an agent in the SAC algorithm, controlling computation offloading and resource allocation decisions. When the MEC server receives ML task requests from the vehicle and RSU, it searches for the best action in the current state. After making a decision and processing the ML task, the MEC server constructs a digital twin vehicle to assist in the autonomous driving of the vehicle.

[0016] Preferably, the communication model uses a Markov channel model to calculate the vehicle's signal-to-noise ratio, and the ML task model includes sensor data models from the vehicle side and sensor data models from the RSU side.

[0017] Preferably, in the computational model, when the vehicle processes ML tasks, part of the processing is done locally and the other part is offloaded to the RSU; when the RSU processes ML tasks, part of the processing is also offloaded to the cloud server.

[0018] Preferably, the step of modeling the computational offloading and resource allocation problem as a Markov decision process based on the computational model includes modeling the various parameters and indicators of the computational offloading and resource allocation problem as quadruples corresponding to typical Markov decision processes, wherein the quadruples include the system's state space, action space, reward function, and state transition probabilities.

[0019] Preferably, the action space includes the ratio of tasks offloaded to the MEC server and the cloud server, the vehicle communication bandwidth allocation ratio, and the computing resources allocated by the MEC server to the vehicle.

[0020] Preferably, the SAC algorithm is a stochastic policy algorithm based on the maximum entropy framework, including a policy network, an evaluation network, and a target evaluation network. The policy network outputs continuous actions, the evaluation network is used to guide the update of the policy network, and the target evaluation network is used to ensure that the training and learning process of the evaluation network is more stable.

[0021] Preferably, the digital twin vehicle's assistance in autonomous driving includes the digital twin vehicle making corresponding driving decisions for the autonomous vehicle through augmented reality technology and AI predictive analysis, thereby enhancing the vehicle's intelligence.

[0022] As can be seen from the above description of the present invention, compared with the prior art, the present invention has the following beneficial effects:

[0023] Based on the time-varying nature of wireless channels and sensor data, this invention can formulate the optimization problem as a Markov decision process. Then, a deep reinforcement learning Soft Actor-Critic (SAC) algorithm is employed to jointly optimize the processing accuracy, processing latency, and processing energy consumption of the ML task.

[0024] In order to improve the processing performance of ML tasks and ensure the efficiency of autonomous driving in vehicle-road systems, this invention adopts a computational offloading and resource allocation method based on deep reinforcement learning to maximize the processing accuracy of ML tasks and reduce the total processing latency and energy consumption of the tasks. Attached Figure Description

[0025] Figure 1 This is a model diagram of the vehicle-road cooperative system of the present invention;

[0026] Figure 2 This is a flowchart of the ML task computation unloading and resource allocation method of the present invention.

[0027] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. Detailed Implementation

[0028] To better understand this technical solution, the present invention will be further described below through specific embodiments.

[0029] See Figure 2 A computational offloading and resource allocation method for vehicle-road cooperative systems, including

[0030] 1) Establish a communication model and an ML task model based on the vehicle-road cooperative system framework, where the communication model is used to calculate the vehicle uplink communication rate.

[0031] In this invention, see Figure 1 The vehicle-road cooperative framework includes: multiple autonomous vehicles equipped with sensors, RSUs (Roadside Units) equipped with MEC servers and sensors, and cloud servers connected to the RSUs via a fiber optic network. In the vehicle-road cooperative system, each RSU is equipped with an MEC server and sensors (such as radar and cameras), and can be wired to the cloud server via a fiber optic backbone network. The RSUs cover a certain number of vehicles, which can be represented as V = {v1, v2, ..., v...} N}

[0032] In this step, the communication model uses a Markov channel model to calculate the vehicle's signal-to-noise ratio. The communication model is established based on the vehicle-road cooperative system framework, including the following:

[0033] Considering that the wireless channel between the vehicle and the RSU is a time-varying channel, it is modeled as a first-order finite-state Markov Channel (FSMC). In time slot t∈Γ, the channel gain of the wireless link between vehicle k and the RSU can be expressed as Υ k (t). Channel gain Υ k (t) is discretized and quantized into L levels, and represented as Γ={G0,···,G L-1}, G L-1 This represents the specific numerical value of the channel gain quantization. Furthermore, Υ k The transition probability from one state h to another state q is expressed as ψ. h,q (t). Therefore, the channel state transition probability matrix for vehicle k and RSU communication is represented as Ψ k (t)=[ψ h,q (t)] L×L According to Shannon's formula, the uplink transmission rate between vehicle k and RSU in time slot t can be expressed as:

[0034]

[0035] Where B is the total bandwidth for communication between all vehicles and the RSU, τ k (t) represents the bandwidth ratio of vehicle k to RSU communication, p k (t) represents the uplink transmit power of vehicle k, σ 2 This represents the power of Gaussian white noise.

[0036] The ML task model includes sensor data models from both the vehicle side and the RSU side. Based on the vehicle-road cooperative system framework, the ML task model is established, specifically including the following:

[0037] In time slot t∈Γ, the sensing ML task generated by the vehicle can be represented as s v,k (t)=(d v,k (t),c v,k (t),l v,k (t)), d v,k (t) represents the data size of the ML task for the k-th vehicle, c v,k (t) represents the amount of computation required to process the task, l v,k (t) represents the maximum tolerable latency for processing this task. Simultaneously, the sensors on the RSU will also generate corresponding sensing ML tasks, denoted as s. r,k (t)=(d r,k(t),c r,k (t),l r,k (t)). Vehicles, MEC servers, and cloud servers all have the ability to process ML tasks, with processing accuracies of P1, P2, and P3, respectively, where P1 < P2 < P3.

[0038] 2) A computational model is established based on the communication model and the ML task model. This computational model provides parameters for the vehicle, RSU, MEC server, and cloud server to process ML tasks. Their inference accuracy for processing ML tasks varies; higher inference accuracy results in safer driving for autonomous vehicles.

[0039] In this step, the computational model established based on the communication model and the ML task model includes the following:

[0040] For each vehicle's ML tasks, a portion can be processed locally, while the remainder can be offloaded to the RSU. Assume that in time slot t, vehicle k will process ML tasks s... v,k (t) The proportion of unloaded goods to RSU is η k (t). The computational delay for local vehicle processing is then: f k (t) represents the CPU cycle frequency of the k-th vehicle. At this time, the processing energy consumption of the vehicle is: E loc,k (t)=ε k (t)(f k ) 2 (1-η k (t)c v,k (t), ε k (t) represents the energy consumption coefficient of the vehicle chip.

[0041] In addition, the ML task of unloading from vehicle k to RSU is η k (t)s v,k (t), and the resulting transmission delay is: The energy consumption generated is: E m,k (t)=p k (t)T tran,k (t). Since corresponding ML tasks will also be generated on the RSU. r,k (t), the total number of tasks to be processed on the RSU is η. k (t)s v,k (t)+s r,k (t). The total tasks on the RSU can also be partially offloaded to the cloud server, assuming an offload ratio of γ. k (t). Then the tasks processed by the MEC server on the RSU are 1-γ of the total tasks. k (t), assuming that the computation frequency of the MEC server assigning k tasks to the vehicle is f at this time. m,kIf (t), then the processing latency of the task on the MEC server is expressed as:

[0042]

[0043] Assume the computation frequency of the cloud server in processing tasks is f. c (t), where the wired transmission rate from the RSU to the cloud server is R. c If (t), then the delay in vehicle k uploading the task to the cloud server for processing includes two parts: a transmission delay and a computation delay on the cloud server. Since the amount of processed data is small, its return transmission delay can be ignored. Therefore, the cloud server processing delay can be expressed as...

[0044]

[0045] 3) Based on the computational model, the computational offloading and resource allocation problem is modeled as a Markov decision model, and a policy network with multiple action outputs is introduced into the SAC algorithm according to the Markov decision model.

[0046] The problem model in this step specifically includes the following:

[0047] The latency caused by fusing the ML tasks on vehicle k and RSU in time slot t is expressed as:

[0048] T sum,k (t)=max{T loc,k (t),T tran,k (t)+T m,k (t),T tran,k (t)+T c,k (t)}

[0049] At this point, the total processing energy consumption of vehicle k's ML task is expressed as:

[0050] E k (t)=E loc,k (t)+E m,k (t)

[0051] Furthermore, since the ML models used by vehicles, MEC servers, and cloud servers are different, the processing accuracy obtained by fusing the ML tasks on vehicle k and RSU is expressed as follows:

[0052]

[0053] In vehicle-road cooperative systems, the processing accuracy, processing latency, and processing energy consumption of raw sensor information are all key indicators. Therefore, the objective function can be expressed as follows:

[0054]

[0055] st:C1:T k (t)≤l max (t)

[0056] C2:P k (t)≥P min (t)

[0057] C3:E k (t)≤I max (t)

[0058]

[0059]

[0060] C6:η1(t)...η k (t)∈[0,1]

[0061] C7:γ1(t)...γ k (t)∈[0,1].

[0062] Therefore, the computational unloading and resource allocation problem is modeled as a Markov decision process based on the computational model. This includes modeling the various parameters and indicators of the computational unloading and resource allocation problem as quadruples corresponding to typical Markov decision processes. The quadruples include the system's state space, action space, reward function, and state transition probabilities.

[0063] To solve the above optimization problem, based on the vehicle-road cooperative system model, a Markov decision process for calculating unloading and resource allocation is first constructed: {S,A,P(s t+1 |s t ,a t ),R(s t ,a t In the formula, the elements from left to right represent the state space, action space, state transition probability, and reward function of the problem, respectively, where the state transition probability P(s) is the state space, action space, state transition probability, and reward function. t+1 |s t ,a t ) indicates that the agent is in state s t Perform action a t The state transitions to state s. t+1 The probability of.

[0064] State Space: Since the ML tasks on each vehicle and RSU differ at different times, and the channel gain constantly changes according to the FSMC, the state s at time slot t is... t ∈S can be represented as

[0065] s t ={s v,1 (t)...,sv,k (t),s r,1 (t)...,s r,k (t),Υ1(t)...,Υ k (t)}

[0066] Action Space: At each time slot t, the actions that the agent can take include three aspects: the proportion of tasks offloaded to the MEC server and the cloud server, the proportion of vehicle communication bandwidth allocated, and the computing resources allocated by the MEC server to the vehicle. Therefore, action a t ∈A can be represented as

[0067] a t ={η1(t)...η k (t),γ1(t)...γ k (t),τ1(t)...τ k (t),f1(t)...f k (t)}

[0068] Reward function: Based on the objective function of this optimization problem, the reward function in this paper is expressed as follows:

[0069]

[0070] Where δ+μ+ω=1.

[0071] In vehicle-road cooperative systems, the MEC server acts as an agent in the SAC algorithm, controlling computation offloading and resource allocation decisions. When the MEC server receives ML task requests from vehicles and RSUs, it searches the current state s t The best action a t After making decisions and processing ML tasks, the MEC server builds a digital twin vehicle to assist in the vehicle's autonomous driving.

[0072] The Stochastic Policy Algorithm (SAC) is a stochastic policy algorithm based on the maximum entropy framework. Compared to most existing DRL algorithms, SAC maximizes both the reward and the policy entropy. The core idea of ​​maximum entropy is to consider as many actions as possible for each state, which means the neural network needs to explore all possible optimal paths. Therefore, SAC has stronger exploration capabilities and can converge to a larger reward value. The optimal policy of the SAC algorithm can be expressed as...

[0073]

[0074] Where γ∈[0,1] is the discount factor, representing the relationship between current and future rewards. α∈[0,1] is the temperature coefficient, representing the relative importance between entropy and reward. H(π(·|s t)) is the entropy term, which is determined by -log(·|s t The entropy is represented by , which indicates the degree of randomness in the policy probability distribution. The more random the policy, the greater the entropy.

[0075] The SAC algorithm is a stochastic policy algorithm based on the maximum entropy framework, comprising a policy network, an evaluation network, and a target evaluation network. The policy network outputs continuous actions, the evaluation network guides the policy network's updates, and the target evaluation network ensures a smoother training process for the evaluation network. The policy network uses π... φ (·|s t The network parameter is denoted by φ, and it outputs continuous actions. To suppress network overestimation, the SAC algorithm uses two evaluation networks to approximate the state-action value function. The evaluation network uses... It is indicated that its parameter is θ i Let i = 1, 2. During the training of the SAC algorithm, the smaller output value of the evaluation network is taken as the estimated value. Accordingly, the SAC algorithm also employs two target evaluation networks. Its parameters are i = 1, 2.

[0076] 4) Allow the agent to continuously interact with the environment and receive rewards, eventually converging to a computational offloading and resource allocation strategy that yields the maximum total reward as the optimal computational offloading and resource allocation strategy.

[0077] In this step, see Figure 2 The training of a Markov decision model can specifically include the following:

[0078] 4.1) Obtain the sequence of ML tasks to be executed.

[0079] 4.2) Initialize the policy network π φ (·|s t ), evaluation network and target evaluation network

[0080]

[0081] 4.3) Policy Network π φ (·|s t Output the corresponding action a based on the state behavior. t Execute action a t Receive reward r t And enter the next state s t+1 The experience of time slot t (s) t ,a t ,r t ,s t+1The parameters are added to the experience pool. The policy network and evaluation network are updated using the loss function, and the parameters are synchronized to the target evaluation network at regular intervals.

[0082] 4.4) Determine whether the reward has converged or whether the maximum number of iterations has been reached. If the reward has converged or the maximum number of iterations has been reached, the network parameter set can be obtained. Otherwise, the parameters of the synchronization policy network and the evaluation network are returned to step 2.

[0083] 4.5) Obtain the learning model and save the system network parameters.

[0084] 5) Execute ML tasks according to the optimal computational offloading and resource allocation strategy, and transmit the processing results to the MEC server. A digital twin vehicle is built on the MEC server, and then the digital twin vehicle predicts the behavior of the vehicle to assist in the autonomous driving of the vehicle.

[0085] Digital twin vehicles assist autonomous driving by using augmented reality technology and AI predictive analytics to make corresponding driving decisions for autonomous vehicles, thereby enhancing the intelligence of the vehicles.

[0086] In this step, the specific action 'a' of obtaining the optimal computational offloading and resource allocation strategy can be achieved through the policy network. t This action is performed to maximize the processing accuracy of ML tasks and minimize processing latency and energy consumption.

[0087] The above are merely specific embodiments of the present invention, but the design concept of the present invention is not limited thereto. Any non-substantial modifications made to the present invention using this concept shall be considered as infringing upon the protection scope of the present invention.

Claims

1. A method for computational offloading and resource allocation in a vehicle-road cooperative system, characterized in that, include A communication model and an ML task model are established based on the vehicle-road cooperative system framework, wherein the communication model is used to calculate the vehicle uplink communication rate. A computational model is established based on the communication model and the ML task model. This computational model calculates the relevant parameters for the vehicle, RSU, MEC server, and cloud server to process ML tasks, specifically including the following: For each vehicle's ML tasks, some are processed locally, and the rest are offloaded to the RSU; assuming that in time slot t, vehicle k will handle the ML tasks... The proportion of offloaded to RSU is The computational delay for local vehicle processing is: , Let the CPU cycle frequency of the k-th vehicle be . This indicates the amount of computation required to process the task; at this point, the energy consumption of the vehicle for processing is: , The energy consumption coefficient of the vehicle chip; In addition, the ML task of unloading from vehicle k to RSU is The resulting transmission delay is: , This indicates the size of the ML task data for the k-th vehicle. This represents the uplink transmission rate between vehicle k and RSU in time slot t; the energy consumption generated is also: , This refers to the uplink transmit power of vehicle k; since corresponding ML tasks will also be generated on the RSU. The total number of tasks pending on the RSU is + ; The total tasks on the RSU can also be partially offloaded to the cloud server, assuming the offload ratio is [percentage missing]. The tasks processed by the MEC server on the RSU are the total tasks. Assume that the computation frequency of task k assigned to vehicle by the MEC server at this time is... The processing latency of a task on the MEC server is then expressed as: ； The computational unloading and resource allocation problem is modeled as a Markov decision model based on the computational model. This includes modeling the parameters and indices of the computational unloading and resource allocation problem as quadruples corresponding to typical Markov decision processes. The quadruples include the system's state space, action space, reward function, and state transition probabilities. Based on the Markov decision model, a multi-action output policy network is introduced into the SAC algorithm. The optimal strategy of the SAC algorithm is expressed as: ； in It is a discount factor, representing the relationship between current rewards and future rewards; It is a temperature coefficient, representing the relative importance between entropy and reward; It is the entropy term, by The entropy represents the degree of randomness in the policy probability distribution; the more random the policy, the greater the entropy. The intelligent agent is allowed to continuously interact with the environment and receive rewards, eventually converging to a computational offloading and resource allocation strategy that yields the maximum total reward as the optimal computational offloading and resource allocation strategy. The ML task is executed according to the optimal computational offloading and resource allocation strategy, and the processing result is transmitted to the MEC server. A digital twin vehicle is built on the MEC server, and then the digital twin vehicle predicts the behavior of the vehicle to assist in the autonomous driving of the vehicle.

2. The computational offloading and resource allocation method for a vehicle-road cooperative system as described in claim 1, characterized in that, The vehicle-road cooperative system framework includes: multiple autonomous vehicles equipped with sensors, RSUs equipped with MEC servers and sensors, and cloud servers connected to the RSUs via a fiber optic network.

3. The computational offloading and resource allocation method for a vehicle-road cooperative system as described in claim 2, characterized in that, The MEC server, acting as the intelligent agent in the SAC algorithm, is used to control computation offloading and resource allocation decisions. When the MEC server receives ML task requests from the vehicle and RSU, it searches for the best action in the current state. After making a decision and processing the ML task, the MEC server constructs a digital twin vehicle to assist in the autonomous driving of the vehicle.

4. The computational offloading and resource allocation method for a vehicle-road cooperative system as described in claim 1, characterized in that, The communication model uses a Markov channel model to calculate the vehicle's signal-to-noise ratio (SNR), and the ML task model includes sensor data models from the vehicle side and sensor data models from the RSU side.

5. The computational offloading and resource allocation method for a vehicle-road cooperative system as described in claim 1, characterized in that, The action space includes the ratio of tasks offloaded to the MEC server and the cloud server, the vehicle communication bandwidth allocation ratio, and the computing resources allocated by the MEC server to the vehicle.

6. The computational offloading and resource allocation method for a vehicle-road cooperative system as described in claim 1, characterized in that, The SAC algorithm is a stochastic policy algorithm based on the maximum entropy framework, which includes a policy network, an evaluation network, and a target evaluation network. The policy network outputs continuous actions, the evaluation network is used to guide the update of the policy network, and the target evaluation network is used to ensure that the training and learning process of the evaluation network is more stable.

7. The computational offloading and resource allocation method for a vehicle-road cooperative system as described in claim 1, characterized in that, The digital twin vehicle's assistance in autonomous driving includes the digital twin vehicle making corresponding driving decisions for the autonomous vehicle through augmented reality technology and AI predictive analysis, thereby enhancing the vehicle's intelligence.

Images