Vehicle task offloading method based on mobility awareness in cloud edge collaborative environment

Abstract

The invention discloses a vehicle task offloading method based on mobility perception in a cloud edge collaborative environment. The method comprises the following steps: defining a vehicle task offloading problem in a vehicle edge environment as a vehicle task offloading optimization problem VTOP with user access delay minimization as a target; modeling the VTOP problem into a constrained Markov decision process, and introducing an MEC server to select a strategy experience pool to cope with the complexity of a mobile edge environment; secondly, a VTO-TD3 method is established on the basis of an on-line difference strategy (off-policy) type deep reinforcement learning algorithm, and the VTOP problem is solved; and then constructing a vehicle task unloading model and a learning framework in a vehicle edge environment, and establishing a VTO-TD3 algorithm formed by deep reinforcement learning. After initialization, unloading strategy optimization and calculation are carried out according to an access delay minimization target, an optimal vehicle task unloading strategy is obtained, and the problem of data-intensive and delay-sensitive vehicle calculation task unloading in a mobile edge environment is solved.

Description

Technical Field

[0001] The present invention relates to the field of vehicle mobile edge computing technology, and more specifically, to a vehicle task offloading method based on mobility perception in a cloud-edge collaborative environment. Background Art

[0002] With the development of intelligent transportation systems and intelligent connected vehicles, vehicles have been given more computing, storage, sensing and communication capabilities. In order to make safe decisions, intelligent connected vehicles need to calculate and analyze a large amount of data from sensors (about 1GB / s). Traditional vehicles can no longer meet the demand for computing and storage resources for large amounts of data. When dealing with complex, data-intensive, and delay-sensitive applications, vehicles face huge challenges. Offloading the vehicle's computing tasks to a remote cloud server is a solution. However, since offloading to a cloud server requires a high communication delay, the communication between the vehicle and the cloud server requires high bandwidth.

[0003] Mobile edge computing (MEC) technology combines IT service environment with cloud computing technology at the edge of the network to improve the computing and storage capabilities of the edge network and reduce network operation and service delivery delays; the vehicle network using MEC can meet the strict requirements of vehicles for service delay and communication reliability, and improve the quality of service (QoS) of vehicle users. However, when the location of the vehicle changes during the process of accessing services, due to changes in the coverage range or capacity limitations of the MEC server, it is necessary to select the best MEC server from multiple MEC edge servers for task offloading to provide services for the vehicle, minimize service access delays, and solve service access interruption problems.

[0004] In CN202311411378.6, a TD3-SMT task offloading method for minimizing latency in the Internet of Vehicles is disclosed, including the following steps: constructing a system model framework; step 2, modeling the task processing on the user side and the server side; step 3, initializing the computational offloading system state; step 4, training the TD3-SMT network and updating the current network parameters of the critic; step 5, using Softmax to estimate the value function; step 6, inputting the computational offloading system state into the pre-trained computational offloading TD3-SMT network; step 7, making an action decision based on the θn value obtained by the policy function to obtain the computational offloading system action. This patent enables user vehicles to optimize the unreasonable offloading ratio and offloading time problems of the Internet of Vehicles computational offloading while ensuring communication needs, and meet the high reliability and low latency requirements of many applications. However, this patent only considers the problem of vehicle task offloading in a mobile edge computing environment, and does not consider the changes in the locations of vehicle users in different time periods and the heterogeneity of MEC servers. Summary of the invention

[0005] The main technical problem to be solved by the present invention is to provide a vehicle task offloading method based on mobility perception in a cloud-edge collaborative environment in view of the deficiencies in the prior art in considering the mobility of vehicle users and the heterogeneity of edge devices.

[0006] The purpose of the present invention is achieved through the following technical solutions:

[0007] A vehicle task offloading method based on mobility perception in a cloud-edge collaborative environment, comprising the following steps:

[0008] S1. Construct a vehicle task offloading model in a collaborative environment between the cloud and the edge;

[0009] S2. Establish a deep reinforcement learning framework;

[0010] S3. Construct a deep neural network and initialize the parameters of the deep neural network;

[0011] S4. Initialize the successful experience pool M success and failure experience pool M failure ;

[0012] S5. Start iterative calculation;

[0013] S51. Randomly select a historical sample, obtain the initial state, and adjust the number of steps to be less than the maximum number of steps, that is, t<T max , and perform a loop operation:

[0014] S511. According to the state s t , select actions and add noise, a t =π θ (s t |θ)+ε,ε~N(0,1);

[0015] S512. Execute action a t , get reward r t and the next state s t+1 ;

[0016] S513. The four-tuple (s t ,a t ,r t ,s t+1 ) According to the size of the reward value, it is deposited into the success experience pool M success Or Failure Experience Pool M failure middle;

[0017] S514. Determine whether the experience pool is full. If not, go to step S511 and set s t+1 Instead of t Loop; otherwise, enter S515;

[0018] S515. Calculate from M success The TD error d of the samples drawn from k , update the sample priority p k =|d k |;

[0019] S515.From M success and M failure A total of m samples were collected Calculate the next action a through the target strategy network t+1 ;

[0020] S517. Calculate the target value y for each sample in the small batch experience sample t ;

[0021] S518. Update the main value network parameter w, the main strategy network parameter θ and the target value network parameters w′, θ′;

[0022] S52. End the loop and output the current average cumulative reward value;

[0023] S6. End the iterative calculation.

[0024] Furthermore, the vehicle task offloading model includes a MEC server, a vehicle service, a sub-service and a vehicle terminal. The MEC server is an entity that provides startup and operation for the vehicle service. The vehicle terminal sends a vehicle service request to the MEC server. The vehicle service is composed of various sub-services. During the offloading process, the sub-services are offloaded to each MEC server.

[0025] Furthermore, the representation of the components in the reinforcement learning framework includes:

[0026] State space P: At any time, the state space Pt is expressed as:

[0027] P t ={Ω(t),Φ}

[0028] Where Ω(t) is the dynamic information set of time period t, and Φ is the static information set;

[0029] Action space A: action a at time t t It is expressed as:

[0030] a t ={a t1 ,...,a ti ,...a tk}

[0031] Among them, a ti It is expressed as:

[0032]

[0033] The action space set A is expressed as:

[0034]

[0035] Reward r: Reward r obtained at time t t Expressed as:

[0036]

[0037] Among them, ST t The service access delay ST is at time t.

[0038] Furthermore, the method for calculating the service access delay ST includes:

[0039] Case 1 R1: Vehicle c j During the mobile process, when it is not within the coverage of any MEC server, the service access delay ST is calculated as:

[0040]

[0041] in Indicates the time when the service request is sent to the cloud server s0. Indicates the time when the service request is received. Indicates the execution time of the service on cloud server s0;

[0042] The second case R2: The vehicle is always within the coverage of the MEC server during the movement, and the MEC server has sufficient storage resources and computing resources to complete all sub-services. The service will not be interrupted during the access process. The service access delay ST is calculated as:

[0043]

[0044] middle is the communication time, Is vehicle service vs. i Execution time on the MEC server;

[0045] The third case R3: When the vehicle is moving, the vehicle terminal device is sometimes within the coverage of the MEC server and sometimes not. The service access delay ST is calculated as:

[0046]

[0047] Furthermore, the deep neural network includes a main strategy network π θ 、Target strategy network π θ′ , main value network Q w1 , target value network Q w1′ , main value network Qw2 , target value network Q w2′ , where the main strategy network π θ Read the current environment status P t , giving the current MEC server unloading adjustment a t , and interact with the environment to generate the next moment space state s t+1 and the current reward value r t ; Target policy network π θ′ Read the next moment state P t+1 Generate the next moment action a t+1 ; Main value network Q w1 , Q w2 P t 、a tt Evaluation; target value network Q w1′ , Q w2′ Responsible for P t+1 、a t+1 Make an evaluation.

[0048] Furthermore, the target value The calculation formula is expressed as:

[0049]

[0050] Where r is the reward function; g∈[0,1] is the reward discount coefficient; By the target policy network π θ' The calculation result is expressed as:

[0051]

[0052] Among them, ε'~clip[N(0,σ')-c,c] is the added noise based on normal distribution; c and -c are the upper and lower boundaries of the noise respectively.

[0053] Furthermore, the main value network parameter w is updated by minimizing the loss function L(w), which is expressed as:

[0054]

[0055] Where m is the number of training samples extracted from the experience pool, including the spatial state P, action a, reward value r at different times, etc.; i represents the i-th historical sample; is the target Q value;

[0056] The updated parameter w is expressed as:

[0057]

[0058] In the formula, μ w is the main value network learning rate.

[0059] Furthermore, the parameter θ of the main strategy network optimization is updated as:

[0060]

[0061] Where θ is the policy network learning rate.

[0062] Furthermore, the main strategy network π θ The gradient descent strategy is used to update the parameter θ, which is expressed as:

[0063]

[0064] Furthermore, the target network parameters w′, θ′ are updated as:

[0065] wj'=τwj+(1-τ)wj'

[0066] θ'=τθ+(1-τ)θ'

[0067] Where τ is the soft update coefficient and j is 1 or 2.

[0068] Compared with the prior art, the beneficial effects are:

[0069] The present invention defines the vehicle task offloading optimization problem in a cloud-edge collaborative environment as a vehicle task offloading optimization problem VTOP with the goal of minimizing user access delay. The VTOP problem is modeled as a constrained Markov decision process, and the MEC server selection strategy experience pool is introduced to cope with the complexity of the mobile edge environment. On the basis of fully considering the heterogeneity of edge devices in the MEC environment and the dynamics of the environment, it is proposed to use the VTO-TD3 method to implement a task offloading strategy based on vehicle mobility perception, which can effectively reduce user service access delay, while having good reliability and being unaffected by dynamic changes in the environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] Figure 1 This is a schematic diagram of the vehicle task offloading scenario in the cloud-edge collaborative environment.

[0071] Figure 2 It is the algorithm network structure diagram.

[0072] Figure 3 It is a schematic diagram of the training process.

[0073] Figure 4 It is a location relationship diagram between vehicle user access terminals and MEC servers in different time periods.

[0074] Figure 5 It is a distribution graph of user requests in different time periods.

[0075] Figure 6 It is the service access delay of vehicle users in different time periods of the three algorithms when the number of MEC servers remains unchanged.

[0076] Figure 7 It is the average value of vehicle user service access delay of the three algorithms when the number of MEC servers remains unchanged.

[0077] Figure 8 This is a diagram of the average service access delay of the three algorithms in different time periods when the number of MEC servers is doubled.

[0078] Fig. 9 is the average service access delay of vehicle users for the three algorithms when the number of MEC servers is doubled.

[0079] Fig.10 It is the vehicle user service access delay of the three algorithms in different time periods when the number of vehicle user requests doubles.

[0080] Fig.11 is the average service access delay of vehicle users for the three algorithms when the number of vehicle user requests doubles. DETAILED DESCRIPTION

[0081] The present invention will be further explained and illustrated below in conjunction with the embodiments, but the specific embodiments do not limit the present invention in any form.

[0082] Description of relevant symbols used in this embodiment:

[0083] symbol unit illustrate S - Server set, where s0 represents the cloud server VS - The set of services accessed by the vehicle C Vehicle User Set R GigaBytes Data size of MEC server resources P CPU cycles / second The processing capacity of each MEC server r - Coverage radius of MEC servers d Meter The distance between the vehicle terminal device and the MEC server wt CPU cycles / bit The size of computing resources required for the sub-services in the service to run ds MegaBytes The size of the data required to access the service <![CDATA[B 0 ]]> Mbps Bandwidth for data transmission between the cloud and vehicle terminal devices <![CDATA[B 1 ]]> Mbps Bandwidth for data transmission between MEC servers ST Millisecond Delayed vehicle service visits

[0084] Example 1

[0085] This embodiment provides a vehicle task offloading method based on mobility perception in a cloud-edge collaborative environment, the steps comprising:

[0086] S1. Construct a vehicle task offloading model, including MEC servers, vehicle services, sub-services and vehicle terminals. The vehicle terminal sends a vehicle service request to the MEC server. The vehicle service consists of various sub-services. During the offloading process, the sub-services need to be offloaded to each MEC server.

[0087] The MEC server is an entity that provides startup and operation for vehicle services. The MEC server set S is represented by S = {s0, ..., s i ,...,s |S|}, s0 represents the cloud server, and each MEC server is represented by s i =(s_lat i ,s_lon i ,s_c i ), where s_lat i ,s_loni Indicates the longitude and latitude of the location of the i-th MEC server, s_c i Indicates the CPU resource size of the i-th MEC server.

[0088] The vehicle service is a lightweight communication process that can be quickly and independently deployed on the MEC server based on container technology, which can complete certain business capabilities and can be called by vehicle users during service access. The vehicle service set is represented by VS = {vs1,...,vs i ,...,vs |VS|}, where vs i =(vs_b i ,vs_c i ) indicates service vs i The sub-service composition and the amount of CPU resources consumed when calling it.

[0089] Different sub-services make up vehicle services, and each sub-service completes different service functions. Represents the components of vehicle services vs i The various sub-services.

[0090] The vehicle terminal sends a service request. Due to the mobility of the vehicle, the location of the terminal device sending the request changes with time. C = {c1, ..., c i ,...}, where c i =(c_lat i ,c_lon i ), c_lat i , c_lon i Indicates terminal device c i The latitude and longitude of your location.

[0091] S2. Establish and strengthen the learning framework;

[0092] Assume that the cloud and the edge are network environment G, the number of MEC servers is K, the number of vehicle service requests is Q, and the time T is divided into 24 time periods. The representation of each component in the reinforcement learning framework is as follows:

[0093] State space P: At any time, the state space Pt is expressed as:

[0094] P t ={Ω(t),Φ}

[0095] Where Ω(t) is the dynamic information set of time period t, and Φ is the static information set.

[0096] Action space A: action a at time t t It is expressed as:

[0097] a t ={a t1 ,...,a ti ,...a tk}

[0098] Among them, a ti It is expressed as:

[0099]

[0100] The action space set A is expressed as:

[0101]

[0102] Reward r: The reward rt obtained at time t is expressed as:

[0103]

[0104] Among them, ST t The service access delay ST is at time t.

[0105] The calculation method of the service access delay ST includes:

[0106] Case 1 R1: Vehicle c j During the mobile process, when it is not within the coverage of any MEC server, the service access delay ST is calculated as:

[0107]

[0108] in Indicates the time when the service request is sent to the cloud server s0. Indicates the time when the service request is received. Indicates the execution time of the service on cloud server s0;

[0109] The second case R2: The vehicle is always within the coverage of the MEC server during the movement, and the MEC server has sufficient storage resources and computing resources to complete all sub-services. The service will not be interrupted during the access process. The service access delay ST is calculated as:

[0110]

[0111] in is the communication time, Is vehicle service vs. i Execution time on the MEC server;

[0112] The third case R3: When the vehicle is moving, the vehicle terminal device is sometimes within the coverage of the MEC server and sometimes not. The service access delay ST is calculated as:

[0113]

[0114] Agent: It is composed of a deep reinforcement learning algorithm and the unloading strategy obtained from it. Deep reinforcement learning is divided into three categories: strategy learning, value learning, and actor-critic. Actor-Critic combines the advantages of strategy learning and value learning, which can well solve the problem of continuous action space, realize single-step update, and improve learning efficiency. The state space is the basis for the agent to make decisions, and the choice of action space is directly affected by the current state. The agent evaluates the value of each action based on the current state and selects the action with the highest expected benefit to execute. Reward is the driving force for the agent to learn. The agent learns and optimizes its behavior strategy by constantly trying different actions and observing the feedback of rewards.

[0115] S3. Build a deep neural network, such as Figure 2 As shown, it includes the main strategy network π θ 、Target strategy network π θ′ , main value network Q w1 , target value network Q w1′ , main value network Q w2 , target value network Q w2′ Among them, the main strategy network π θ Responsible for reading the current environment status P t , giving the current MEC server unloading adjustment a t , and interact with the environment to generate the next moment space state s t+1 and the current reward value r t ; Target policy network π θ′ Responsible for reading the next moment state P t+1 Generate the next moment action a t+1 ; Main value network Q w1 , Q w2 Responsible for P t 、a t Evaluation; target value network Q w1′ , Q w2′ Responsible for P t+1 、a t+1 Make an evaluation.

[0116] S4. Initialize the parameters w1 and w2 of the two main value networks and the parameter θ of the main strategy network; initialize the target value network parameters w1'=w1, w2'=w2, θ'=θ.

[0117] S5. Initialize the successful experience pool M success and failure experience pool M failure , the initial value is set to 0;

[0118] S6. Start iterative calculation;

[0119] S61. Randomly select a historical sample and obtain the initial state. If the adjustment step number is less than the maximum limit step number, that is, t<T max , and perform a loop operation:

[0120] S611. According to the status s t , select actions and add noise, a t =π θ (s t |θ)+ε,ε~N(0,1);

[0121] S612. Execute action a t , get reward r t and the next state s t+1 ;

[0122] S613. The four-tuple (s t ,a t ,r t ,s t+1 ) According to the size of the reward value, it is deposited into the success experience pool M success Or Failure Experience Pool M failure middle;

[0123] S614. Determine whether the experience pool is full. If not, go to step S611 and set s t+1 Instead of t Loop; otherwise, enter S615;

[0124] S615. Calculate from M success The TD error d of the samples drawn from t , that is, the TD error at time t, V(s) is the state value function, d t =r t+1 +γV(s t+1 )-V(s t ), update the sample priority p t =|d t |。 。

[0125] S615.From M success and M failure A total of m samples were collected Calculate the next action a through the target strategy network t+1 ;

[0126] S617. Calculate the target value y for each sample in the small batch experience sample t , which is expressed as:

[0127]

[0128] Where r is the reward function; g∈[0,1] is the reward discount coefficient; By the target policy network π θ' The calculation result is expressed as:

[0129]

[0130] Where ε'~clip[N(0,σ')-c,c] is the added noise based on normal distribution; c and -c are the upper and lower boundaries of the noise respectively.

[0131] S618. Optimize the parameter w by minimizing the loss function L(w), where L(w) is expressed as:

[0132]

[0133] Where m is the number of training samples extracted from the experience pool, including the spatial state P, action a, reward value r at different times, etc.; i represents the i-th historical sample; is the target Q value.

[0134] According to the gradient rule, the main value network parameter w is updated as:

[0135]

[0136] In the formula, μ w is the main value network learning rate.

[0137] S619. Use the gradient descent strategy to optimize the main strategy network parameter θ, expressed as:

[0138]

[0139] Update the policy network parameters θ as:

[0140]

[0141] S620. Update the target network parameters w′, θ′, expressed as:

[0142] wj'=τwj+(1-τ)wj'

[0143] θ'=τθ+(1-τ)θ'

[0144] Where τ is the soft update coefficient; j is 1 or 2.

[0145] S62. End the loop and output the current average cumulative reward value;

[0146] S7. End the iterative calculation.

[0147] Example 2

[0148] This embodiment provides proof that VTOP is a constrained Markov decision process (Constrained MDP, CMDP).

[0149] In the MEC environment, the vehicle service access delay is set as the evaluation criterion of the vehicle mobility-aware task offloading strategy, and the system modeling is carried out with the goal of minimizing the service access delay. In the cloud-edge collaborative edge environment, the vehicle service access delay calculation method is divided into the following three cases:

[0150] Case 1 R1: Vehicle c j During the mobile process, if the device is not within the coverage of any MEC server, i MEC servers i The coverage radius is as follows:

[0151]

[0152] In this scenario, all service requests submitted by vehicles are processed by cloud server s0, which has sufficient storage and computing resources to complete all sub-services. The service access delay ST is calculated as:

[0153]

[0154] in the formula Indicates the time when the service request is sent to the cloud server s0. Indicates the time when the service request is received. and The time is the sum of the service propagation delay and the transmission delay. It represents the execution time of the service on cloud server s0. Since all sub-services are completed in the cloud, the communication delay between sub-services is not considered. and The calculation formula is expressed as:

[0155]

[0156] In the formula, ds i For vehicle c j Accessing Services vs i The amount of data required when B is the size of the cloud server s0 and the vehicle c j The network bandwidth between them. V is the signal propagation rate, d(s0,c j ) is the vehicle terminal c j The distance to the cloud server s0 is calculated as:

[0157]

[0158]

[0159] Where (c_lat j ,c_lon j ) and (s_lat0, s_lon0) are the vehicle terminal c j and the longitude and latitude of cloud server s0.

[0160] Execution time The calculation formula is:

[0161]

[0162] In the formula, Is it in service vs i Execute the jth subservice in The required computing resource size, p0 represents the CPU processing power of cloud server s0.

[0163] The second case R2: The vehicle is always within the coverage of the MEC server during the movement, and the MEC server has sufficient storage resources and computing resources to complete all sub-services, and the service will not be interrupted during the access process. The scenario description is expressed as:

[0164]

[0165] The calculation of service access delay ST is expressed as:

[0166]

[0167] in the formula is the communication time. Since the vehicle terminal is close to the MEC server, the service propagation time between the MEC server and the vehicle access terminal device is not considered temporarily. Only the service transmission time between MEC servers is considered.

[0168] The calculation formula is expressed as:

[0169]

[0170] In the formula, Representing vehicle services vs i Sub-services provided and The bandwidth between MEC servers. Is vehicle service vs. i The execution time on the MEC server is calculated as:

[0171]

[0172] In the formula, Represents execution service vs i The data size required for the jth sub-service, p q Represents MEC servers q In the processing sub-service The CPU processing power.

[0173] The third case R3: When the vehicle is moving, the vehicle terminal device is sometimes within the coverage of the MEC server and sometimes not. The scenario formula is expressed as:

[0174]

[0175] Due to the limited resource capacity and coverage of the MEC server, when the MEC server resources required for the execution of a sub-service are insufficient or the vehicle terminal device is not within the coverage of the MEC server, the service may be terminated. At this time, the unprocessed sub-service will be submitted to the cloud server s0 for processing, and the cloud server s0 will process all subsequent sub-services. The cloud server s0 and the MEC server jointly respond to the user's request.

[0176] The ST calculation formula is:

[0177]

[0178] In the formula, and It represents the communication time and execution time on the MEC server. The calculation formulas are as follows:

[0179]

[0180]

[0181] The calculation formula is:

[0182]

[0183] is the processing service time on cloud server s0, and the calculation formula is:

[0184]

[0185] All vehicle service requests are divided into 24 time windows, and the dynamic information set Ω is described in the MEC environment as:

[0186] Ω={Ω(1),Ω(2),...,Ω(T)}.

[0187] The dynamic information set Ω for each time period is described as:

[0188] Ω(t)={U(t),R(t),DM(t),UE(t),SM(t)},

[0189] In the formula, U(t) represents the number of vehicle service requests in time period t, R(t) is the available resources of MEC server, DM(t) represents the distance matrix between vehicle equipment and MEC server, UE(t) is the vehicle service request response matrix, and SM(t) is the selection matrix between MEC servers when vehicle task offloading.

[0190] In the MEC environment, the static information Φ includes the longitude and latitude of the MEC server location, the server coverage radius r, the service set VS, the server processing capacity P, and the bandwidth B between the cloud and the vehicle terminal device. 0 , and the bandwidth B between MEC servers 1 , defined as:

[0191] Φ={slat,slon,r,VS,P,B 0 ,B 1}

[0192] Under the dynamic and static environmental constraints in the cloud-edge collaborative environment, the VTOP problem is defined as:

[0193]

[0194] Finding the best uninstall strategy Minimize the average delay in vehicle service access.

[0195] The theorem can be derived from the above related description.

[0196] Theorem: The VTOP problem is a constrained Markov decision CMDP problem.

[0197] Proof: The VTOP problem satisfies the following four conditions.

[0198] (1) State countability: The states in the VTOP problem are finite. In the VTOP problem, since the vehicle user service requests are described in 24 time windows, the dynamic state in each time window is described as formula (14), where the number of vehicle user service requests and the available resources of the MEC server are all limited.

[0199] (2) Probabilistic independence of state transition: The state transition in the VTOP problem is probabilistically independent because the vehicle user service request and the MEC server resource size are independent in each time window. The state transition in each time window is only related to the state in the current time window and has nothing to do with the past state, which can be expressed as:

[0200] P[S t+1 |St ]=P[S t+1 |S1,...,S t ]

[0201] (3) Objective function with random rewards: The goal of the VTOP problem is to find the best MEC server offloading strategy to minimize the average service access delay. The delay of each service is random and changes over time. This objective function has random rewards. Maximizing the reward benefits while satisfying the long-term cost benefits, that is, finding the optimal offloading strategy

[0202] (4) Constraint: In the search for the optimal offloading strategy, the MEC server computing resource capacity is constrained, that is, ∑vs_c i ≤∑s_c j .

[0203] The VTOP problem meets the three necessary conditions of the Markov decision problem. At the same time, it is subject to the hardware conditions of the MEC server during task offloading, so it can be regarded as a CMDP problem.

[0204] Example 3

[0205] This implementation provides a simulation experiment. The data in the experiment mainly comes from the real data set of Australia EUA and some synthetic data. The location information of MEC server and vehicle user service request comes from the data set EUA. In the experiment, the geographical location of the base station is used as the location of the MEC server, and the geographical location of the terminal device is used as the location where the vehicle user service request is sent. Since there is no relevant public data set to provide service information, data such as application, server capacity and location information changes are synthesized, and the detailed description is as follows:

[0206] (1) MEC server and vehicle users. The latitude and longitude information of the base station location is obtained from the EUA dataset, and different coverage radii are set for the MEC server. The simulation software is used to show the location movement trajectory of the vehicle users in the dataset over time. The service request time is divided into 24 time periods, and the time periods t3, t7, and t8 are randomly selected. 10 ,t 21 The movement trajectory of Figure 4 As shown in the figure, the red points are the locations of the five edge servers, the blue circles are the coverage of each MEC server, and the dots of different colors represent the locations of different vehicle users when they send service requests. It can be concluded that the number and locations of vehicle users covered by the MEC server are changing in different time periods.

[0207] (2) User service requests. In different time periods, Poisson’s theorem is used to simulate the number of service requests sent by vehicle users. Figure 5As shown in Figure 2, the changes in the number and location of user service requests reflect the dynamics of the mobile edge environment.

[0208] (3) Bandwidth and computing power. A bandwidth matrix between MEC servers is randomly generated with a bandwidth range from 100MB to 400MB, where the bandwidth from MEC server to cloud server is set to 1000MB. The computing intensity of the five edge nodes and the cloud is set to 100Mbps, 150Mbps, 200Mbps, 100Mbps, 150Mbps, and 1000Mbps respectively;

[0209] (4) Resource and data size. The CPU and ROM resource capacity of the MEC server are randomly generated, and the task data size of the service request is randomly generated between 100MB and 500MB. After the MEC server receives the vehicle user service request, it allocates computing resources according to the task size.

[0210] The present invention selects two comparison methods: Q-learning, CORA and VTO-TD3 to evaluate the vehicle user service access delay results. The two methods are introduced as follows.

[0211] (1) Q-learning: Use the distributed stateless Q-learning algorithm to learn the current environment information and realize vehicle resource allocation.

[0212] (2) CORA: It uses deep reinforcement learning (DRL) technology to process high-dimensional and continuous state and action spaces to effectively learn the optimal offloading solution.

[0213] The simulation results are as follows:

[0214] (1) The number of MEC servers remains unchanged, and the average service access delay is compared;

[0215] Five base stations are randomly selected as MEC servers, and 50 terminal devices are used as vehicle users to send service request locations. The average service access delay of vehicle users in the three methods is compared in 24 time periods. Figure 6 It can be seen that in the 24 time periods, the methods with better offloading performance are VTO-TD3, CORA, and Q-learning. In the VTO-TD3 method, due to the use of experience replay and dual-network deterministic strategies, VTO-TD3 has the smallest average service access delay in most time periods, and its performance is better than the other two methods. CORA uses deep reinforcement learning to adapt to the dynamic environment of the vehicle edge. The average access delay is lower than VTO-TD3 in a few time periods, but higher than VTO-TD3 in most time periods. Q-learning does not fully consider the dynamics of the cloud-edge hybrid environment and the heterogeneity of MEC servers, and its performance is poorer than VTO-TD3 and CORA.

[0216] The average value of the vehicle user service access delay in 24 time periods is calculated, as shown in Table 2. It is found that the average access delay of VTO-TD3 is 932.7191ms, which is 5.82% lower than the average access delay of CORA (990.36075ms) and 12.22% lower than the average access delay of Q-learning (1062.536ms). Figure 7 As shown in the figure, VTO-TD3 outperforms both CORA and Q-learning.

[0217] Table 2

[0218] Uninstall method Average latency (ms) VTO-TD3 932.7191 CORA 990.36075 Q-learning 1062.536

[0219] (2) The number of vehicle user requests remains unchanged, and the average service access delay is compared;

[0220] 10 base stations are randomly selected as MEC servers, and 50 terminal devices are used as vehicle users to send service request locations. The number of service requests remains unchanged, and the average service access delays of the three methods are compared.

[0221] As the number of MEC servers increases, the range of choices available to vehicle users during task offloading increases, so the overall average access delay decreases relative to Figure 1. Figure 8 It can be seen that VTO-TD3 has the minimum service access delay in most of the 24 time periods, Q-learning has the minimum service access delay in a few time periods, and the minimum service access delays obtained by CORA and VTO-TD3 in multiple time periods are similar.

[0222] The average value of the vehicle user service access delay in each time period above is calculated, as shown in Table 3. The average access delay of VTO-TD3, 777.1189ms, is 4.9% lower than the average access delay of CORA, 817.156ms, and 11.534% lower than the average access delay of Q-learning, 878.43793ms. At the same time, compared with the user access delay under the five MEC servers in the above table, the average access delay under VTO-TD3 is reduced by 16.68%, CORA is reduced by 17.49%, and Q-learning is reduced by 17.33%. Fig. 9 It can be seen that when the number of MEC servers doubles, the VTO-TD3 method still has less service access delay.

[0223] Table 3

[0224] Uninstall method Average latency (ms) VTO-TD3 777.1189 CORA 817.156 Q-learning 878.43793

[0225] (3) Comparison of average service access delay as the number of vehicle user requests increases

[0226] In order to further evaluate the performance of the three methods under a large number of user requests, the number of user requests is doubled and the average access delay of user services is compared. Fig.10 It can be found that due to the limited resources of MEC servers, when the number of vehicle user service requests gradually increases, the number of service requests processed by the cloud server increases. Therefore, in multiple time periods, the average service access delay difference among the three methods is small.

[0227] The average value of the vehicle user service access delay in each time period is calculated, as shown in Table 4. It is found that as the number of requests increases, the average delay under the three methods increases, but due to the response of the cloud server, the average service access delay difference under the three methods is small. Fig.11 As shown in the figure, when the number of vehicle user service requests doubles, the VTO-TD3 method still has the minimum service access delay value and has good scalability.

[0228] Table 4

[0229] Uninstall method Average latency (ms) VTO-TD3 1062.66929 CORA 1106.69505 Q-learning 1127.48158

[0230] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the embodiments of the present invention. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the embodiments here. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the claims of the present invention.

Claims

1. A vehicle task offloading method based on mobility perception in a cloud-edge collaborative environment, characterized in that the steps include: S1. Construct a vehicle task offloading model in a collaborative environment between the cloud and the edge; S2. Establish a deep reinforcement learning framework; S3. Construct a deep neural network and initialize the parameters of the deep neural network; S4. Initialize the successful experience pool M success and failure experience pool M failure ; S5. Start iterative calculation; S51. Randomly select a historical sample, obtain the initial state, and adjust the number of steps to be less than the maximum number of steps, that is, t<T max , and perform a loop operation: S511. According to the state s t , select actions and add noise, a t =π θ (s t |θ)+ε,ε~N(0,1); S512. Execute action a t , get reward r t and the next state s t+1 ; S513. The four-tuple (s t ,a t ,r t ,s t+1 ) According to the size of the reward value, it is deposited into the success experience pool M success Or Failure Experience Pool M failure middle; S514. Determine whether the experience pool is full. If not, go to step S511 and set s t+1 Instead of t Loop; otherwise, enter S515; S515. Calculate from M success The TD error d of the samples drawn from k , update the sample priority p k =|d k |; S515. From M success and M failure A total of m samples were collected Calculate the next action a through the target strategy network t+1 ; S517. Calculate the target value y for each sample in the small batch experience sample t ; S518. Update the main value network parameter w, the main strategy network parameter θ and the target value network parameters w′, θ′; S52. End the loop and output the current average cumulative reward value; S6. End the iterative calculation.

2. According to claim 1, the vehicle task offloading method based on mobility perception in the cloud-edge collaborative environment is characterized in that: The vehicle task offloading model includes MEC servers, vehicle services, sub-services and vehicle terminals. The MEC server is an entity that provides startup and operation for vehicle services. The vehicle terminal sends a vehicle service request to the MEC server. The vehicle service is composed of various sub-services. During the offloading process, the sub-services are offloaded to each MEC server.

3. According to claim 1, the vehicle task offloading method based on mobility perception in the cloud-edge collaborative environment is characterized in that: The characterization of the components in the reinforcement learning framework includes: State space P: At any time, the state space Pt is expressed as: P t ={Ω(t),Φ} Where Ω(t) is the dynamic information set of time period t, and Φ is the static information set; Action space A: action a at time t t It is expressed as: a t ={a t1 ,...,a ti ,...a tk } Among them, a ti It is expressed as: The action space set A is expressed as: Reward r: Reward r obtained at time t t Expressed as: Among them, ST t The service access delay ST is at time t.

4. According to claim 3, the vehicle task offloading method based on mobility perception in the cloud-edge collaborative environment is characterized in that: The calculation method of the service access delay ST includes: Case 1 R1: Vehicle c j During the mobile process, when the device is not covered by any MEC server, the service access delay ST is calculated as: in Indicates the time when the service request is sent to the cloud server s0. Indicates the time when the service request is received. Indicates the execution time of the service on cloud server s0; The second case R2: The vehicle is always within the coverage of the MEC server during the movement, and the MEC server has sufficient storage resources and computing resources to complete all sub-services. The service will not be interrupted during the access process. The service access delay ST is calculated as: in is the communication time, Is vehicle service vs. i Execution time on the MEC server; The third case R3: When the vehicle is moving, the vehicle terminal device is sometimes within the coverage of the MEC server and sometimes not. The service access delay ST is calculated as:

5. According to claim 1, the vehicle task offloading method based on mobility perception in the cloud-edge collaborative environment is characterized in that: The deep neural network includes a main policy network π θ 、Target strategy network π θ′ , main value network Q w1 , target value network Q w1′ , main value network Q w2 , target value network Q w2′ , where the main strategy network π θ Read the current environment status P t , giving the current MEC server unloading adjustment a t , and interact with the environment to generate the next moment space state s t+1 and the current reward value r t ; Target policy network π θ′ Read the next moment state P t+1 Generate the next moment action a t+1 ; Main value network Q w1 , Q w2 P t 、a t Evaluation; target value network Q w1′ , Q w2′ Responsible for P t+1 、a t+1 Make an evaluation.

6. According to claim 1, the vehicle task offloading method based on mobility perception in the cloud-edge collaborative environment is characterized in that: Target value The calculation formula is expressed as: Where r is the reward function; g∈[0,1] is the reward discount coefficient; By the target policy network π θ' The calculation result is expressed as: Among them, ε'~clip[N(0,σ')-c,c] is the added noise based on normal distribution; c and -c are the upper and lower boundaries of the noise respectively.

7. According to claim 1, the vehicle task offloading method based on mobility perception in the cloud-edge collaborative environment is characterized in that: The main value network parameter w is updated by minimizing the loss function L(w), which is expressed as: Where m is the number of training samples extracted from the experience pool, including the spatial state P, action a, reward value r at different times, etc.; i represents the i-th historical sample; is the target Q value; The updated parameter w is expressed as: In the formula, μ w is the main value network learning rate.

8. According to claim 1, the vehicle task offloading method based on mobility perception in a cloud-edge collaborative environment is characterized in that: The parameter θ of the main policy network optimization is updated as: Where θ is the policy network learning rate.

9. The vehicle task offloading method based on mobility perception in a cloud-edge collaborative environment according to claim 8 is characterized in that: Main Strategy Network π θ The gradient descent strategy is used to update the parameter θ, which is expressed as:

10. The vehicle task offloading method based on mobility perception in a cloud-edge collaborative environment according to claim 1 is characterized in that: The target network parameters w′, θ′ are updated as: wj'=τwj+(1-τ)wj' θ'=τθ+(1-τ)θ' In the formula, is the soft New coefficient; j is 1 or 2.

Images