Side-channel collaborative multi-task computing offloading method, device and storage medium

Abstract

The present invention provides a method, device and storage medium for offloading multi-task computing in a side road collaboration, and relates to the field of mobile edge computing technology, including: building a three-layer distributed system model: a control layer, an edge service layer, and a mobile device layer; converting the computing offloading problem into a Markov optimization problem for solution; based on a double-delayed deep deterministic policy gradient algorithm, firstly gradually enhancing and optimizing the exploration noise of the main network, and then using a Q-value estimation method based on an interpolation function to improve the Q-value estimation accuracy, and finally introducing a priority experience playback double buffer pool method to improve the sampling efficiency of training, so as to form an improved double-delayed deep deterministic policy gradient algorithm, and then solve the Markov optimization problem. The present invention can effectively improve the efficiency of computing offloading, reduce the response time delay and energy consumption generated by the computing offloading process, and improve the vehicle user experience.

Description

Technical Field

[0001] The present invention relates to the field of mobile edge computing technology, and in particular to a method, device and storage medium for offloading edge collaborative multi-task computing. Background Art

[0002] The Internet of Vehicles (IoV) refers to an integrated network that uses sensors, communication technologies, and intelligent algorithms to connect vehicles to the Internet and realize intelligent interaction and data transmission between vehicles (V2V) and vehicles and infrastructure (V2I). IoV realizes functions such as information sharing and collaborative driving between vehicles. These application services often need to meet the requirements of low latency, high reliability, and high throughput. In the field of IoV research, highway side-road collaboration is a typical case scenario. However, the application data traffic on today's highways is growing exponentially, and the diversity and complexity of tasks pose severe challenges to the computing power and storage resources of on-board computing devices. Although offloading tasks to cloud servers is a feasible solution, the long transmission distance between vehicles and cloud servers will generate high communication delays and consume a lot of network bandwidth. Therefore, cloud computing does not have an advantage for computing tasks that are data-intensive and delay-sensitive, and mobile edge computing (MEC) has emerged. Computation offloading in mobile edge computing has been a research hotspot in recent years. By offloading computing tasks to edge servers, the latency and energy consumption of the computing process can be effectively reduced, thereby improving the quality of user experience. However, most of the current research scenarios for computational offloading focus on urban roads and residential roads, and there are fewer studies on highways. In addition, most studies on highways do not fully consider that due to the high-speed mobility of vehicles, the position changes of intelligent vehicles will cause their network transmission structure to change over time. Secondly, most computational offloading strategies use discretization algorithms, which are not suitable for continuous action problems of computational offloading.

[0003] Due to the rapid development of the current Internet of Vehicles, vehicle application services are growing exponentially in highway scenarios, such as safety alarms, road detection, real-time traffic conditions, navigation maps, etc., and these application services often need to meet the requirements of low latency, high reliability and high throughput. However, the vehicle's own communication, computing and storage resources are gradually unable to meet the computing requirements of the task, resulting in increasing time delays and energy consumption in the vehicle task computing process.

[0004] Therefore, how to offload the computation of vehicle tasks, reduce task response time, reduce computing energy consumption, and improve system performance has become a technical problem that needs to be solved urgently. Summary of the invention

[0005] The present invention aims to solve at least one of the technical problems existing in the prior art or related technology, and discloses a side-road collaborative multi-task computing offloading method, device and storage medium, which solves the increasingly prominent contradiction between the limited resources of vehicles and the exponentially growing demand for vehicle application services, and effectively reduces the response time and energy consumption of computing offloading process tasks.

[0006] The first aspect of the present invention discloses a roadside collaborative multi-task computing unloading method, comprising: building a three-layer distributed system model: establishing a control layer based on a control center, the control layer is used to centrally manage roadside units and mobile edge computing servers to achieve information sharing and management; establishing an edge service layer based on roadside units and mobile edge computing servers, the edge service layer is used to accept computing tasks; establishing a mobile device layer based on a mobile terminal of a vehicle, when the local computing resources of the mobile device layer cannot meet the computing requirements, the computing is unloaded to the edge service layer through a computing offloading strategy for computing;

[0007] The computation offloading problem is transformed into a Markov optimization problem for solution: The Markov optimization process of resource allocation and task offloading of mobile edge computing in vehicle networks is expressed as:

[0008] ,

[0009] Among them, S is the state space, A is the system state space, is the state transition probability, is the reward benefit; based on the behavioral decision under a given observation state, the mobile terminal obtains a reward from the reward function corresponding to the environment and observes the next state, with the goal of maximizing the cumulative reward;

[0010] Solving the Markov optimization problem based on the improved double-delay deep deterministic policy gradient algorithm: On the basis of the double-delay deep deterministic policy gradient algorithm, the exploration noise of the main network is first gradually enhanced and optimized, and then the Q-value estimation method based on the interpolation function is adopted to improve the Q-value estimation accuracy. Finally, the priority experience replay double buffer pool method is introduced to improve the sampling efficiency of training, so as to form an improved double-delay deep deterministic policy gradient algorithm, and then solve the Markov optimization problem.

[0011] In this technical solution, the traditional TD3 algorithm does not have an obvious advantage in early exploration convergence, and although TD3 alleviates the problem of overestimation of Q value caused by the DDPG algorithm, the TD3 algorithm will cause the underestimation bias of Q value, which will have a certain impact on performance. At the same time, the TD3 algorithm uses a random sampling method to sample from the experience replay pool during training, but this operation ignores the importance difference of different experience data in the replay buffer, and may also cause different samples to be repeatedly sampled or omitted, resulting in uneven distribution of training samples, affecting the generalization ability of the algorithm and reducing training efficiency. Therefore, the present invention designs an optimization algorithm based on TD3, namely, an improved double-delayed deep deterministic policy gradient algorithm: PD-TD3 (TwinDelayed Deep Deterministic Policy Gradient with Prioritized Double bufferpool Experience Replay), which can not only alleviate the above problems, but also minimize the average energy consumption and average delay consumption.

[0012] According to the side-by-side collaborative multi-task computing offloading method disclosed in the present invention, preferably, the Markov optimization process specifically includes: defining a state space : ,in, Indicates at time The queue length of the data buffer, represents the channel vector for uplink transmission, It indicates the ratio of the unit received power after uplink signal projection. Represents the transmission distance; defines the action space : , indicating that at time The data transmission path that the vehicle can choose is the behavior space; define the reward value : The reward value is an indicator used to evaluate the quality of the uninstallation strategy, expressed as:

[0013]

[0014] Among them, E com Represents the total energy consumption of the task calculation process, T com Represents the total delay of the task calculation process, is a non-negative weighting factor, indicating the preference for delay and energy consumption. The value enables the computation offloading strategy to make a trade-off between energy consumption and latency; in each discrete time slot On the mobile terminal, observe the status , and select an action based on the uninstallation strategy ; Then, based on the behavioral decision under a given observation state, the mobile terminal obtains a reward from the reward function corresponding to the environment , and observe the next state , the goal of the mobile terminal is to maximize the cumulative reward .

[0015] According to the side-path collaborative multi-task computing offloading method disclosed in the present invention, preferably, the step of gradually enhancing and optimizing the exploration noise of the main network specifically includes: based on the double-delay deep deterministic policy gradient algorithm, in the early stage of the algorithm, using a noise variance of 0.01~0.05 to speed up the convergence speed of learning; in the later stage of the algorithm, increasing the noise variance to find a balance between exploration and utilization to obtain a better strategy.

[0016] According to the side-path cooperative multi-task computing offloading method disclosed in the present invention, preferably, the process of adding exploration noise specifically includes: based on the Actor network of the double-delayed deep deterministic policy gradient algorithm, the exploration noise is from a variance of The noise is drawn from a normal distribution and then multiplied by the noise scaling factor :

[0017]

[0018] in, is the added noise, represents the noise variance, represents the noise scaling factor, represents the lower and upper limits of noise; the clip function represents the noise to be extracted (the noise obeys distribution), clipped within the range of (-c, c); the clip function format is clip(a,b,c), which means clipping the a data to the range of (b,c). The function format indicates that -c,c is used to represent the range.

[0019] According to the sidewalk cooperative multi-task computing offloading method disclosed in the present invention, preferably, a Q value estimation method based on an interpolation function is adopted to improve the Q value estimation accuracy, specifically including: adopting a method based on cubic B-spline interpolation Value estimation method: Represents the Q value obtained by the dual critic network. Construct a cubic B-spline interpolation function between:

[0020]

[0021] The number of control points is 4. represents the coefficient of the ith control point, represents the cubic B-spline basis function;

[0022] Use the confidence region algorithm to estimate the y value,

[0023] Select a confidence interval with a confidence level of 95%, and the confidence interval is expressed as:

[0024] C o n f i d e n c e _ I n t e r v a l = [ y ^ − z ψ , y ^ + z ψ ]

[0025] in, is the estimated value of the spline interpolation function in the interval, is the critical value of the standard normal distribution, is the standard deviation of the estimation error;

[0026] In the confidence interval, represents the standard deviation of the estimated error, and the midpoint of the confidence interval is selected as the estimated value.

[0027] .

[0028] According to the side-lane cooperative multi-task computing offloading method disclosed in the present invention, preferably, a priority experience replay double buffer pool method is introduced to improve the sampling efficiency of training, specifically including:

[0029] Use dual experience pools to store historical experience data, with the maximum delay tolerance of the task The attribute is the threshold, Indicates the actual uninstallation time of the task:

[0030]

[0031] When the uninstallation is successful, the experience is stored in the experience buffer , unloading failures are stored in the experience buffer ;

[0032] During sampling, multiple pieces of experience data are extracted from the successful experience pool with a probability of 0.8, and multiple pieces of experience data are extracted from the failed experience pool with a probability of 0.2:

[0033]

[0034] in, Indicates from The number of samples to be drawn from the experience buffer, represents the weight of extraction, is the number of samples required for training;

[0035] Each experience pool uses a priority experience replay mechanism, and the experience data in the pool is divided by the absolute value of the time difference error. To calculate the priority, the priority data is stored in the leaf node using the SumTree structure:

[0036]

[0037] in, represents the sample size, represents the target network Q value, is the calculated current Q value, Indicates the next state, Indicates the next action. is the target network parameter;

[0038] The probability of each data being sampled during sampling :

[0039]

[0040] in, Is The probability of decision, parameter Control the degree of use of priorities;

[0041] New experience data stored in the experience pool is given the highest priority; to ensure that each experience data has a different probability of being extracted and to increase the training speed, the priority weight is set :

[0042]

[0043] In the formula, represents the size of the priority experience replay pool, and the parameter β controls the degree of correction used.

[0044] According to the side-road collaborative multi-task computing offloading method disclosed in the present invention, preferably, the control layer and the edge service layer communicate using optical fiber links; optical fiber links are also used for communication between devices in the edge service layer; non-orthogonal multiple access technology is used for communication between vehicles in the mobile equipment layer and roadside units in the edge service layer, and continuous interference cancellation technology is used to reduce signal interference problems.

[0045] The second aspect of the present invention discloses a side-by-side collaborative multi-task computing unloading device, comprising: a memory for storing program instructions; a processor for calling the program instructions stored in the memory to implement a side-by-side collaborative multi-task computing unloading method such as any of the above-mentioned technical solutions.

[0046] A third aspect of the present invention discloses a computer-readable storage medium storing a program code, wherein the program code is used to implement a side-path collaborative multi-task computing offloading method as described in any of the above-mentioned technical solutions.

[0047] The beneficial effects of the present invention include at least: a three-layer distributed model is built for highway scenarios and takes into account the problem of task migration, and the system is stable and efficient. Algorithm advantages: First, in order to balance the early exploration and utilization of the TD3 algorithm and accelerate the early convergence speed of the algorithm, the present invention gradually enhances and optimizes the exploration noise variance of the main network. Secondly, in order to solve the problem of Q value underestimation bias of the target network of the TD3 algorithm, a Q value estimation method based on interpolation function and trust region is adopted to improve the accuracy of Q value estimation. Finally, considering the importance of different data, a priority experience replay double buffer pool method is introduced to improve the sampling efficiency of training. The present invention can effectively improve the efficiency of computational offloading, reduce the response time delay and energy consumption generated by the computational offloading process, and improve the vehicle user experience requirements. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 A schematic diagram of a system model of a side-path collaborative multi-task computing offloading method according to an embodiment of the present invention is shown.

[0049] Figure 2 A schematic flow chart of a side-path collaborative multi-task computing offloading method according to an embodiment of the present invention is shown.

[0050] Figure 3 A schematic diagram of transmission service migration according to an embodiment of the present invention is shown.

[0051] Figure 4 A schematic diagram of the traditional TD3 algorithm framework is shown.

[0052] Figure 5 A schematic diagram of an improved TD3 algorithm framework according to an embodiment of the present invention is shown.

[0053] Figure 6 A schematic block diagram of a side-path collaborative multi-task computing offloading device according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0054] In order to more clearly understand the above-mentioned objects, features and advantages of the present invention, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0055] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways different from those described herein. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0056] Most traditional highway research system models have only two layers, the edge layer and the mobile device layer. And they do not consider the task migration problem. Task migration refers to the edge node RSU only with the range Assume that in At this moment, the vehicle transfers its computing tasks to the RSU in binary full offload mode (Binary full offloading means that the entire task is transmitted and sent without splitting a task into segments.) Due to the high speed of the vehicle, the vehicle drove out after the calculation was completed. Service scope, i.e. , the calculation results cannot be transmitted back to the vehicle. Therefore, the control center will select the RSU closer to the vehicle Connect with the vehicle before the session ends to collaboratively transmit calculation results.

[0057] The existing calculation offloading strategy is the TD3 algorithm. The TD3 algorithm process is as follows:

[0058] 1. Initialization phase:

[0059] 1. Initialize neural network parameters: The TD3 algorithm uses multiple neural networks, including an Actor network for selecting actions and two Critic networks (Twin Critic) for estimating state value functions. At the same time, the parameters of the target Critic network are initialized.

[0060] 2. Set up the experience replay buffer: This is a memory space used to store past experience tuples, which usually contain information such as observations, actions, rewards, and next states.

[0061] 3. Initialize other hyperparameters: including learning rate, target update rate, noise parameters, etc. These parameters have an important impact on the performance and stability of the algorithm.

[0062] 2. Training phase:

[0063] 1. Select action: Use the Actor network to select an action based on the current state. To increase exploration, some exploration noise may be added to the action.

[0064] 2. Perform actions and observe the environment: Apply the selected action to the environment and observe the next state and reward fed back by the environment.

[0065] 3. Store the experience in the replay buffer: Store the observed state, action, reward, and next state information in the experience replay buffer.

[0066] 4. Sampling data from the replay buffer: Randomly extract a batch of data from the experience replay buffer for training the Critic network and updating the Actor network.

[0067] 5. Calculate gradients and update network parameters: Use the extracted data to calculate gradients through the Critic network and update the parameters of the Critic network. At the same time, the parameters of the target Critic network are also updated, usually using a soft update strategy.

[0068] 6. Update of Actor network: The update of Actor network is delayed, that is, it is updated after the Critic network is updated multiple times. This strategy helps to improve the stability of training.

[0069] 3. Loop iteration:

[0070] The above steps will be repeated continuously to form a cyclic iterative process. Through multiple iterations, the TD3 algorithm can gradually learn the optimal strategy, enabling the agent to achieve better performance in the environment.

[0071] Based on the TD3 algorithm of the above-mentioned calculation offloading strategy, an embodiment of the present invention discloses a side-path collaborative multi-task calculation offloading method, such as Figure 2 As shown, the method includes:

[0072] Step S101, building a three-layer distributed system model: establishing a control layer based on the control center, the control layer is used to centrally manage the roadside unit and the mobile edge computing server to achieve information sharing and management; establishing an edge service layer based on the roadside unit and the mobile edge computing server, the edge service layer is used to accept computing tasks; establishing a mobile device layer based on the mobile terminal of the vehicle, when the local computing resources of the mobile device layer cannot meet the computing requirements, the computing is unloaded to the edge service layer through the computing offloading strategy for computing;

[0073] Step S102, convert the computation offloading problem into a Markov optimization problem for solution: The Markov optimization process of resource allocation and task offloading of mobile edge computing in the vehicle network is expressed as:

[0074] ,

[0075] Among them, S is the state space, A is the system state space, is the state transition probability, is the reward benefit; based on the behavioral decision under a given observation state, the mobile terminal obtains a reward from the reward function corresponding to the environment and observes the next state, with the goal of maximizing the cumulative reward;

[0076] Step S103, solving the Markov optimization problem based on the improved double-delay deep deterministic policy gradient algorithm: on the basis of the double-delay deep deterministic policy gradient algorithm, firstly, the exploration noise of the main network is gradually enhanced and optimized, and then the Q value estimation method based on the interpolation function is adopted to improve the Q value estimation accuracy, and finally the priority experience replay double buffer pool method is introduced to improve the sampling efficiency of training, so as to form an improved double-delay deep deterministic policy gradient algorithm, and then solve the Markov optimization problem.

[0077] According to the above embodiments of the present invention, Figure 1 As shown, the three-layer distributed system model of the present invention includes: a control layer, an edge service layer and a mobile device layer; specifically, the present invention is oriented to the research scenario of a two-way straight highway with multiple users and multiple servers, and a three-layer distributed computing offloading system model is built to centrally control and utilize distributed computing resources to efficiently execute tasks:

[0078] (1) Control layer: Since most highways are located on the edge of cities or rural areas, they are sparsely distributed. Therefore, in order to ensure the reliability of computing offloading services, the control layer is composed of a control center with sufficient resources and control capabilities, which centrally manages RSU (Road Side Unit) and MEC servers to achieve computing information management and sharing.

[0079] (2) Edge service layer: This layer mainly consists of servers that receive calculations, including RSU and MEC servers with certain transmission and computing capabilities.

[0080] (3) Mobile device layer: This layer is composed of various types of mobile terminal vehicles. When the local computing resources cannot meet the computing requirements of any computationally intensive tasks of any mobile vehicle device, they are offloaded to the edge service layer for computing through an efficient offloading strategy.

[0081] The vehicles on the highway maintain a high speed and a constant speed for a certain period of time, and the vehicle set is recorded as , v n ∈ V ( n ∈ [ 0 , N ] ) The vehicle mission is described as ,in Indicates the task data size, Indicates the number of CPU cycles required to process vehicle tasks, It represents the maximum delay threshold that can be tolerated for processing vehicle tasks. RSU is installed near the road to receive vehicle tasks that need to be offloaded. The RSU set is recorded as , r m ∈ R ( m ∈ [ 0 , M ] ) , deploy MEC servers for each RSU to provide computing and storage services.

[0082] According to the above implementation, another embodiment of the present invention further discloses the operating principle of the system model including:

[0083] Communication model: divide the time into fine-grained parts and build a time The discrete time model of , the time set is represented as T = { t 0 , t 1 , ⋯ , t S } , t i ∈ T ( i ∈ [ 0 , S ] ) , is any moment in the time set. Indicates the vertical distance from the road centerline to the RSU, Indicates the communication coverage of RSU, Indicates the vehicle speed, Time Vehicle with RSU The Euclidean distance is expressed as :

[0084]

[0085] RSU only with scope The present invention considers the problem of service migration of calculation result feedback, such as Figure 3 Example. At this moment, the vehicle transfers its computing tasks to the RSU in binary full offload mode Due to the high speed of the vehicle, the vehicle drove out after the calculation was completed. Service scope, i.e. Therefore, the control center will select the RSU that is closer to the vehicle. Before the session ends, the vehicle is connected to cooperate in transmitting the calculation results. According to Shannon's formula, the vehicle Upload data to RSU The transmission rate is defined as:

[0086]

[0087] Indicates the uplink bandwidth; Indicates time slot Get off the vehicle Signal-to-Interference-plus-Noise Ration (SINR).

[0088] Network model: In the three-layer distributed computing offloading model framework, optical fiber links are used to communicate between the control layer and the edge service layer, and between the edge service layer devices. The vehicles in the mobile device layer communicate with the RSUs in the edge service layer using the non-orthogonal multiple access (NOMA) technology in wireless communications. Unlike traditional orthogonal multiple access technology, NOMA technology can accommodate more users through non-orthogonal resource allocation, allowing interference to be controlled through non-orthogonal resource allocation and tolerable increase in receiver complexity.

[0089] Considering the time-varying and block fading characteristics of the channel, a Gaussian Markov block fading autoregressive model is used to accurately describe the The channel attenuation of vehicle signal transmission, the vehicle channel vector is expressed as:

[0090]

[0091] In the formula is the error vector, showing a complex Gaussian distribution, For time slot The normalized channel correlation coefficient is At this moment, the RSU receiving signal is expressed as:

[0092]

[0093] In the formula Transmission power for offloading mission data to vehicles, is a complex data symbol with unit variance, Noise parameters representing the additive white Gaussian noise of the vehicle transmission signal.

[0094] In NOMA technology, since multiple users communicate at the same time and frequency and share spectrum resources, the signals between users will interfere with each other, which increases the complexity of system design and interference management. To solve the signal interference problem, Successive Interference Cancellation (SIC) technology is used. SIC technology needs to know the order of user calculation execution, that is, to determine the order of the starting users in each iteration. The order is based on the vehicle channel gain. The values ​​are sorted in ascending order. Time, signal to interference and noise ratio :

[0095]

[0096] Computing model: In order to provide users with diversified services, vehicles generate a large number of computing tasks. When the vehicle's computing power cannot meet user needs, the vehicle assigns the task to the MEC server for calculation. Therefore, the computing offloading process consists of local computing and offloaded computing.

[0097] 1. Local Computing

[0098] The local computing time consumption of the task is divided into the average waiting time of the task , local device task calculation time The average waiting time of local tasks is the ratio of the length of the local computing data waiting queue to the time:

[0099]

[0100] Indicated in The number of tasks arriving at the queue buffer at any given moment, that is, The length of the data waiting queue at each moment. express The local processor uses the Dynamic Voltage and Frequency Scaling (DVFS) technology, and its computing power is determined by the CPU frequency per unit time. Measure. The average computation time of local tasks is the time slot The ratio of the number of CPU cycles required to process 1 bit of task data to the computing power of the local processor:

[0101]

[0102] Therefore, the total time required for task-local computation is expressed as:

[0103]

[0104] Represents local computing power, the total energy consumption of local computing of the task:

[0105]

[0106] (II) Unloading calculation

[0107] The computation time consumed by task offloading is divided into the average waiting time of the task , Uninstall task transfer time , offload task calculation time The average waiting time of the offload task is the ratio of the length of the waiting queue for the offload calculation data to the time:

[0108]

[0109] Indicated in The number of tasks arriving at the unload queue buffer at a certain moment, i.e. The length of the queue waiting for data to be unloaded at any time. express The length of the data waiting queue at the moment. The transmission time of the offload task, including the uplink transmission time , the vehicle transmits task data to the RSU; downlink transmission time , RSU returns the calculation result to the vehicle. The uplink transmission time is the ratio of the task data volume to the transmission rate:

[0110]

[0111] Downlink transmission time takes into account RSU calculation completes vehicle Calculation task, and whether the vehicle has driven out Signal coverage , enter RSU coverage:

[0112]

[0113] Indicates the size of the calculation result data. represents the fiber transmission speed between RSUs. Therefore, the offload task transmission time is expressed as:

[0114]

[0115] The calculation time of the offload task is the ratio of the required offload task data volume to the server computing power:

[0116]

[0117] represents the computing capacity of the edge server. The total time required for task offloading calculation is expressed as:

[0118]

[0119] represents the vehicle transmission power, represents the server computing power, and the total energy consumption of task offloading computing is expressed as:

[0120]

[0121] In summary, the total delay of the task calculation process is ;

[0122] Calculate total energy consumption .

[0123] According to the above embodiment, another embodiment of the present invention further discloses the specific steps of converting the computation offloading problem into a Markov optimization problem for solution:

[0124] Considering the joint impact of multiple states on offloading decisions, the computational offloading problem is transformed into an optimal strategy problem under the Markov decision process: the agent changes its state by taking actions to obtain rewards and interact with the environment. The Markov optimization process of resource allocation and task offloading in MEC vehicle networks is expressed as ,in is the state space, is the system state space, is the state transition probability, The key elements of deep reinforcement learning methods for solving problems are state, action, and reward. , the specific model definition is as follows:

[0125] State Space : Comprehensively consider the MEC environment, ,in, Indicates at time The queue length of the data buffer, The channel vector for uplink transmission, The ratio of the unit received power of the uplink signal after projection, Indicates the transmission distance.

[0126] Action Space : Indicates at time The data transmission paths that vehicles can choose are behavior spaces.

[0127]

[0128] Reward Value : The reward value is an indicator used to evaluate the quality of the uninstallation strategy, expressed as:

[0129]

[0130] Among them, E com Represents the total energy consumption of the task calculation process, T com Represents the total delay of the task calculation process, is a non-negative weighting factor, indicating the preference for delay and energy consumption. The value can make the computation offloading strategy balance between energy consumption and latency. On the other hand, the agent observes its state , and chooses an action based on its strategy Then, based on the action decision given the observed state, the agent receives a reward from the reward function corresponding to its environment , and observe the next state , the goal of the agent is to maximize the cumulative reward .

[0131] like Figure 4 and Figure 5 As shown, the improvement of the present invention over the traditional TD3 algorithm is shown. Another embodiment of the present invention also discloses a specific method for improving the TD3 algorithm:

[0132] The double-delayed deep deterministic policy gradient (TD3) algorithm is an optimized version of the DDPG algorithm. It is also based on the Actor-Critic framework and is more suitable for continuous action space problems with large state-action spaces such as computational offloading. In order to alleviate the overestimation of the value function in DDPG, the TD3 algorithm draws on the learning strategy of double Q learning and introduces a double critic network to calculate and estimate the Q value of the next state. In order to offset the impact of the overestimation of the Q value, a smaller estimated Q value is selected as the target Q value result. At the same time, when the loss function is large, in order to alleviate the error propagation caused by updating the participant network, the TD3 algorithm designs a delayed policy update for the target network, that is, after updating the current network multiple times, the target network is only updated once. And truncated orthogonal distribution noise is added to the target action as regularization to achieve target policy smoothing and avoid the impact of peak shifting.

[0133]

[0134] in, represents the final Q value, is the target network Q value, represents the discount factor, is the target value function, are the target network parameters, is the added noise, is a normal distribution, is the standard deviation, Indicates the lower noise limit and the upper noise limit.

[0135] The traditional TD3 algorithm does not have a clear advantage in early exploration convergence, and although TD3 alleviates the problem of overestimation of Q value caused by the DDPG algorithm, the TD3 algorithm will cause the underestimation bias of Q value, which will have a certain impact on performance. At the same time, the TD3 algorithm uses a random sampling method to sample from the experience replay pool during training, but this operation ignores the importance difference of different experience data in the replay buffer, and may also cause different samples to be repeatedly sampled or omitted, resulting in uneven distribution of training samples, affecting the generalization ability of the algorithm and reducing training efficiency. Therefore, the present invention designs an optimization algorithm based on TD3, PD-TD3 (Twin Delayed Deep Deterministic Policy Gradient with Prioritized Double buffer pool Experience Replay), which can not only alleviate the above problems, but also minimize the average energy consumption and average delay consumption. The specific implementation method of the optimization is as follows:

[0136] 1. Explore enhanced exploration noise

[0137] Exploration noise is an important hyperparameter of the TD3 algorithm, which is used to add noise to the exploration actions during the exploration phase of the Actor network. In order to balance the exploration and exploitation in the TD3 algorithm, PD-TD3 uses an exploration enhancement method to adjust the exploration noise. In the early stages of the algorithm, a relatively small noise variance (variance range 0.01~0.05) is used to focus more on utilizing existing experience and speed up the convergence of learning. In the later stages, the noise variance is increased to find a balance between exploration and exploitation, so that the agent can gradually transition from utilizing existing experience to more extensive exploration during training to obtain better strategies. The exploration noise in the Actor network ranges from a variance of The noise is drawn from a normal distribution and then multiplied by the noise scaling factor :

[0138]

[0139] represents the noise variance, represents the initial noise variance, represents the number of training steps, represents the noise scaling factor, represents the initial noise scaling factor, represents the lower and upper limits of noise. The clip function represents the noise to be extracted (the noise obeys distribution), clipped in the range (-c, c).

[0140] 2. Accurate Q value

[0141] In the TD3 algorithm of the dual critic network, the minimum value method is used to To solve the underestimation bias, the PD-TD3 algorithm target network double critic adopts cubic B-spline interpolation based Value estimation method. Construct a cubic B-spline interpolation function between:

[0142]

[0143] in, The number of control points is 4. represents the coefficient of the ith control point, represents the cubic B-spline basis function;

[0144] Use the confidence region algorithm to estimate the appropriate y value. Choose a confidence interval with a confidence level of 95%, expressed as:

[0145] C o n f i d e n c e _ I n t e r v a l = [ y ^ − z ψ , y ^ + z ψ ]

[0146] in, is the estimated value of the spline interpolation function in the interval, is the critical value of the standard normal distribution, is the standard deviation of the estimated error. Within the confidence interval, represents the standard deviation of the estimated error, and the midpoint of the confidence interval is selected as the estimated value.

[0147] .

[0148] 3. Dual Priority Experience Buffer Pool

[0149] To improve sampling efficiency, the PD-TD3 algorithm uses dual experience pools to store historical experience data, with the maximum delay tolerance of the task. The attribute is the threshold, Indicates the actual unloading time of the task.

[0150]

[0151] When the uninstallation is successful, the experience is stored in the experience buffer , unloading failures are stored in the experience buffer When sampling, a certain amount of experience is extracted from the successful experience pool with a high probability, and a portion of data is extracted from the failed experience pool with a low probability. This not only takes into account the importance of failed experience data, but also improves sampling efficiency.

[0152]

[0153] Indicates from The number of samples to be drawn from the experience buffer, represents the weight of extraction, is the number of samples required for training. Each experience pool uses a priority experience replay mechanism, and the experience data in the pool is calculated by the absolute value of TD-error. To calculate the priority, the priority data is stored in the leaf node using the SumTree structure

[0154]

[0155] represents the sample size, Indicates the next state, Indicates the next action. is the target network parameter. The probability of each data being sampled during sampling :

[0156]

[0157] Is The probability of decision, parameter Control the degree of use of priority. New experience data stored in the experience pool will be given the highest priority so that new information can be learned faster. However, if the priority of new experience is too high, it may cause the experience distribution to be too biased towards new experience, making the neural network training process unable to converge. In order to ensure that each experience data has a different probability of being extracted and improve the training speed, set the priority weight :

[0158]

[0159] In the formula, is the size of the priority experience replay pool, parameter Controls the extent to which corrections are used.

[0160] like Figure 6 As shown, according to another embodiment of the present invention, a side-by-side collaborative multi-task computing unloading device 600 is also disclosed, including: a memory 601, used to store program instructions; a processor 602, used to call the program instructions stored in the memory to implement the side-by-side collaborative multi-task computing unloading method as in the above embodiment.

[0161] According to another embodiment of the present invention, a computer-readable storage medium is disclosed. The computer-readable storage medium stores program code, and the program code is used to implement the side-path collaborative multi-task computing offloading method as described in the above embodiment.

[0162] All or part of the steps in the various methods of the above embodiments can be completed by controlling the relevant hardware through a program, and the program can be stored in a readable storage medium, and the storage medium includes a read-only memory (ROM), a random access memory (RAM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), a one-time programmable read-only memory (OTPROM), an electronically erasable programmable read-only memory (EEPROM), a compact disc (CD-ROM) or other optical disc storage, magnetic disk storage, magnetic tape storage, or any other readable medium that can be used to carry or store data.

[0163] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for offloading side-by-side collaborative multi-task computing, characterized in that: include: Build a three-layer distributed system model: establish a control layer based on the control center, which is used to centrally manage roadside units and mobile edge computing servers to achieve information sharing and management; An edge service layer is established based on the roadside unit and the mobile edge computing server, and the edge service layer is used to accept computing tasks; a mobile device layer is established based on the mobile terminal of the vehicle, and when the local computing resources of the mobile device layer cannot meet the computing requirements, the computing is offloaded to the edge service layer through the computing offloading strategy for computing; The computation offloading problem is transformed into a Markov optimization problem for solution: The Markov optimization process of resource allocation and task offloading of mobile edge computing in vehicle networks is expressed as: {S,A,P(s t+1 ∣s t ,a t ),R(s t ,a t )}, Among them, S is the state space, A is the system state space, P(s t+1 ∣s t ,a t ) is the state transition probability, R(s t ,a t ) is the reward benefit; based on the behavioral decision under a given observation state, the mobile terminal obtains a reward from the reward function corresponding to the environment and observes the next state, with the goal of maximizing the cumulative reward; Solving the Markov optimization problem based on the improved double-delay deep deterministic policy gradient algorithm: On the basis of the double-delay deep deterministic policy gradient algorithm, the exploration noise of the main network is firstly gradually enhanced and optimized, and then the Q-value estimation method based on the interpolation function is adopted to improve the Q-value estimation accuracy, and finally the priority experience replay double buffer pool method is introduced to improve the sampling efficiency of training, so as to form the improved double-delay deep deterministic policy gradient algorithm, and then solve the Markov optimization problem; The step of gradually enhancing and optimizing the exploration noise of the main network specifically includes: Based on the double-delayed deep deterministic policy gradient algorithm, in the early stage of the algorithm, a noise variance of 0.01 to 0.05 is used to accelerate the convergence of learning; in the later stage of the algorithm, the noise variance is increased to find a balance between exploration and utilization to obtain a better strategy; The process of adding exploration noise specifically includes: based on the Actor network of the double-delayed deep deterministic policy gradient algorithm, the exploration noise is increased from a variance of The noise is drawn from a normal distribution and multiplied by the noise scaling factor χ: Where μ is the added noise, represents the noise variance, χ represents the noise scaling factor, -c, c represents the noise lower limit and noise upper limit, and clip represents clipping the extracted noise within the range of (-c, c). The extracted noise obeys distributed.

2. The side-channel collaborative multi-task computing offloading method according to claim 1 is characterized in that: The Markov optimization process specifically includes: Define the state space S v (t i ): S ν (t i )={L ν (t i ),h ν (t i ),ρ ν (t i-1 ),d ij (t i )}, Among them, L v (t i ) indicates that at time t i The queue length of the data buffer, h v (t i ) represents the channel vector for uplink transmission, ρ v (t i-1 ) represents the ratio of unit received power after uplink signal projection, d ij (t i ) represents the transmission distance; Define the action space A v (t i ): A={a1,a2,a3,...,a n }, indicating that at time t i The data transmission paths that vehicles can choose are the behavior space; Define the reward value R v (t i ): The reward value is an indicator used to evaluate the quality of the uninstallation strategy, expressed as: R ν (t i )=-10oh ν E com -(1-h ν )T com Among them, E com Represents the total energy consumption of the task calculation process, T com represents the total delay of the task calculation process, ω v is a non-negative weighting factor, indicating the preference for delay and energy consumption. By setting ω v The value enables the computation offloading strategy to make a trade-off between energy consumption and delay. At each discrete time slot Δt, the mobile terminal observes the state S∈S v (t i ), and select an action A∈A according to the offloading strategy v (t i ); Then, based on the behavioral decision under the given observation state, the mobile terminal obtains a reward R from the reward function corresponding to the environment v (t i ), and observe the next state S′. The goal of the mobile terminal is to maximize the cumulative reward R v .

3. The side-channel collaborative multi-task computing offloading method according to claim 1 is characterized in that: The step of using the Q value estimation method based on the interpolation function to improve the Q value estimation accuracy specifically includes: The Q value estimation method based on cubic B-spline interpolation is adopted: Represents the Q value obtained by the dual critic network. Construct a cubic B-spline interpolation function between: n is the number of control points 4, c i represents the coefficient of the i-th control point, B i,3 (x) represents the cubic B-spline basis function; Use the confidence region algorithm to estimate the y value, A confidence interval with a confidence level of 95% is selected and the confidence interval is expressed as: in, is the estimated value of the spline interpolation function in the interval, z is the critical value of the standard normal distribution, and ψ is the standard deviation of the estimation error; In the confidence interval, σ represents the standard deviation of the estimated error. The midpoint of the confidence interval is selected as the estimated value.

4. The side-channel collaborative multi-task computing offloading method according to claim 1 is characterized in that: The step of introducing the priority experience replay double buffer pool method to improve the sampling efficiency of training specifically includes: Use dual experience pools to store historical experience data, with the maximum delay tolerance of the task The attribute is the threshold, T v,t Indicates the actual uninstallation time of the task: When the uninstallation is successful, the experience is stored in the experience buffer B1, and when the uninstallation fails, it is stored in the experience buffer B2; During sampling, multiple pieces of experience data are extracted from the successful experience pool with a probability of 0.8, and multiple pieces of experience data are extracted from the failed experience pool with a probability of 0.2: in, represents the number of samples to be extracted from the B1 and B2 experience buffer pools, θ represents the weight of extraction, and Y is the number of samples required for training; Each experience pool uses a priority experience playback mechanism. The experience data in the pool is prioritized by the absolute value of the time difference error |Loss|. The priority data is stored in the leaf node using the SumTree structure: Among them, Y represents the sample size, y represents the target network Q value, is the calculated current Q value, s′ represents the next state, a′ represents the next action, θ v is the target network parameter; The probability P(i) of each data being sampled during sampling: in, is the probability determined by |Loss|, and the parameter u controls the degree of use of the priority; New experience data stored in the experience pool is given the highest priority; Set the priority weight W i : In the formula, F β represents the size of the priority experience replay pool, and the parameter β controls the degree of correction used.

5. The side-channel collaborative multi-task computing offloading method according to any one of claims 1 to 4, characterized in that: The control layer and the edge service layer communicate using optical fiber links; the devices in the edge service layer also communicate with each other using optical fiber links; the vehicles in the mobile equipment layer and the roadside units in the edge service layer communicate using non-orthogonal multiple access technology, and continuous interference cancellation technology is used to reduce signal interference problems.

6. A side-channel collaborative multi-task computing unloading device, characterized in that: include: A memory for storing program instructions; A processor, used to call the program instructions stored in the memory to implement the side-by-side collaborative multi-tasking computing offloading method as described in any one of claims 1 to 5.

7. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program code, and the program code is used to implement the side-path collaborative multi-tasking computing offloading method as described in any one of claims 1 to 5.

Images