A task computing offloading method based on matd3 in vehicle-mounted edge computing

Abstract

The present application belongs to the field of parallel and distributed high-performance computing, and specifically relates to a task computing offloading method based on MATD3 in vehicle-mounted edge computing, which introduces a task scheduling priority algorithm and provides a new idea for vehicle networking computing offloading. By considering the execution cost and internal dependency of the task to determine the task priority, the task offloading sequence can be optimized, the total processing delay of the application program can be reduced, and a multi-agent reinforcement learning method, especially a strategy based on MATD3, is adopted to cooperatively control multiple agents, fully considering the competition for computing resources among multiple vehicles, task dependency, limited computing resources of RSU, vehicle mobility and other complex factors, which can better adapt to the dynamically changing environment of vehicle networking, determine the optimal offloading strategy of vehicle tasks, and thus minimize the offloading failure rate and energy consumption of the tasks in the system.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of parallel and distributed high-performance computing, and particularly relates to a vehicle-mounted edge computing method considering the dependency structure of sub-tasks between new vehicle-mounted applications generated by intelligent and connected vehicles and the dynamic changes of edge nodes under the environment of the vehicles. BACKGROUND

[0002] With the progress of wireless access technology and the increasing popularity of intelligent and connected vehicles (ICV), vehicle networking has become an important paradigm of 5G networks. Vehicle networking makes various reliable vehicle services possible, especially new vehicle-mounted applications such as automatic driving, augmented reality, and vehicle-mounted multimedia services, which promote the development of intelligent transportation and improve traffic safety and efficiency. These applications are usually computationally intensive, which naturally leads to high energy consumption and delay. However, due to the physical characteristics of vehicles, vehicles are limited in computing resources and memory size, and it is difficult to guarantee the needs of new applications by relying solely on resource-constrained vehicle terminals. In order to solve the contradiction between the needs of these application programs and the limitations of resource-constrained ICVs, researchers have introduced mobile edge computing into vehicle networking and proposed vehicular edge computing (VEC). VEC meets the demand for expanding computing capacity of vehicles by moving cloud computing resources down to the network edge close to users and allowing vehicles to offload applications to roadside units (RSUs) equipped with MEC servers, significantly improving the quality of service for vehicle tasks.

[0003] Currently, the demand for data communication and computing resources of in-vehicle applications is increasing, however, how to make offloading decisions and resource allocation in time-varying and uncertain vehicle networking scenarios is a key challenge. Designing an efficient vehicle computing offloading strategy to further utilize computing and communication resources to assist in executing new applications generated during the ICV driving process has become a key research issue. Traditional methods such as Lyapunov optimization and game theory algorithms can provide approximate optimal solutions in a short time, but these algorithms have poor robustness and are difficult to implement real-time decision-making. As a promising artificial intelligence technology, deep reinforcement learning (DRL) has great advantages in dealing with uncertain decision-making problems. Compared with heuristic methods, DRL methods have stronger robustness. However, traditional single-agent algorithms are not suitable for multi-vehicle environments, as the behavior of each vehicle will affect the entire environment. When a vehicle changes its strategy, it also changes the environment of other vehicles, which leads to continuous changes in the environment, thereby increasing the instability of the learning process and making it difficult to meet the convergence conditions. Moreover, in a dynamic and unstable multi-vehicle environment, past experiences may quickly become obsolete, and key techniques such as experience replay in traditional single-agent algorithms cannot be used directly. SUMMARY

[0004] (I) Technical problems solved

[0005] The purpose of the present application is to provide a computing offloading strategy that takes into account the structural differences of new in-vehicle applications generated by intelligent connected vehicles and the dynamic changes of edge nodes in the environment in which the vehicle is located, with the aim of reducing energy consumption and application offloading failure rate in computing offloading. This method specifically considers the influence of vehicle mobility and application dependency structure in the VEC environment, effectively reducing energy consumption and application offloading failure rate in vehicle edge computing.

[0006] To achieve the above-mentioned purpose of the application, the technical scheme of the present application is to establish a Markov decision process model for the computing offloading problem in a multi-vehicle scenario. The application is divided into multiple tasks, and the dependency relationship is described using a directed acyclic graph. On this basis, a task scheduling priority algorithm is proposed to further reduce application offloading delay.

[0007] (II) Technical solutions

[0008] To achieve the above-mentioned purpose, the present application provides the following technical scheme: a task computing offloading method based on MATD3 in vehicle edge computing, comprising the following steps:

[0009] Step 1, introduce a task scheduling priority algorithm to obtain the task priority according to the execution cost and internal dependency relationship of the task, wherein the execution cost is related to the number of RSUs within the communication range of the vehicle;

[0010] Step 2, convert the task offloading problem into an optimal policy problem under Markov decision process, define state space, action space and reward function, wherein the state space includes vehicle state and MEC server state, the action space is to determine the target MEC server of each task, and the reward function is related to the application offloading failure rate and energy consumption;

[0011] Step 3, design a strategy based on distributed deep reinforcement learning, which includes TD3 algorithm, MATD3 algorithm and MATD3-based computing offloading strategy;

[0012] Step 4, in the TD3 algorithm, set the actor network and critic network, the actor network outputs actions according to the observed state, the critic network includes two Q networks for evaluating action quality, update the Q network by minimizing the loss, update the actor network according to the sampled policy gradient, and the parameters of the target network are updated according to the online network parameters;

[0013] Step 5, in the MATD3 algorithm, multiple TD3-based agents are designated to learn collaborative control strategies, using a centralized training and decentralized execution framework, centralized Q value calculation considers information of all agents, loss function is defined according to Q value calculation, actor network gradient is calculated according to loss function, and parameters of target network are updated according to specific equation;

[0014] Step 6, in the MATD3-based computing offloading strategy, an intelligent module is set on each ICV, including actor network and critic network, during the training process, the main actor network and the main critic network are trained centrally, and the target actor network and the target critic network are executed decentralized, the offloading process is divided into interaction stage and training stage, the interaction stage includes initializing network parameters, generating applications, obtaining task scheduling priority, selecting actions and interacting with the environment, and the training stage includes updating the main critic network, updating the main actor network, updating the target critic network and the target actor network.

[0015] Preferably, the vehicle state includes the position of the vehicle itself, the CPU cycle required by the current task, the size of the computing task and the computing complexity parameter, and the MEC server state includes the number of unoccupied CPU cores and the computing capacity in the MEC server in the RSU server.

[0016] Preferably, in the interaction phase of the MATD3-based computing offloading strategy, each ICV initializes the parameters of the actor network and the critic network from the data center, generates an application and decomposes with a certain probability, obtains the scheduling priority of the task according to the task priority algorithm, obtains the initial state of the VEC environment, selects an action according to the environment state at the beginning of the time slice and interacts with the environment, obtains the reward and the next state, and stores the conversion in the experience replay buffer D.

[0017] Preferably, in the training phase of the MATD3-based computing offloading strategy, network training includes three steps:

[0018] Step a, the data center trains the main critic network, samples a random batch of conversions from the experience replay buffer D, calculates the Q value of the main critic network, and updates the parameters of the main critic network by minimizing the loss function;

[0019] Step b, the parameters of the main actor network are updated by the deterministic policy gradient descent method;

[0020] Step c, the target actor network parameters and the target critic network parameters of each ICV are updated according to a specific equation.

[0021] Preferably, during network training, the actor network outputs action instructions according to the current state, the critic network estimates the Q value of the action, and guides the actor network to update the policy, and the action selection of each ICV only depends on the local state information observed by itself.

[0022] (Three) beneficial effects

[0023] Compared with the prior art, the present application provides a MATD3-based task computing offloading method in vehicle edge computing, which has the following beneficial effects:

[0024] The MATD3-based task computing offloading method in vehicle edge computing introduces a task scheduling priority algorithm, providing a new idea for vehicle networking computing offloading. By considering the execution cost and internal dependency of the task to determine the task priority, it helps to optimize the task offloading sequence and reduce the total processing delay of the application program. A multi-agent reinforcement learning method is adopted, especially a MATD3-based strategy, which cooperatively controls multiple agents, fully considers the competition for computing resources among multiple vehicles, task dependency, limited computing resources of RSU, and vehicle mobility and other complex factors, and can better adapt to the dynamically changing environment of vehicle networking, determine the optimal offloading strategy of vehicle tasks, and thus minimize the offloading failure rate and energy consumption of tasks in the system. BRIEF DESCRIPTION OF DRAWINGS

[0025] Figure 1 The present application is a schematic diagram of a structural vehicle edge network model;

[0026] Figure 2 Decomposition model diagram for application of the structure of the present application;

[0027] Figure 3 MATD3-based distributed computing offloading algorithm structure diagram for the structure of the present application. DETAILED DESCRIPTION

[0028] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0029] Please refer to Figures 1-3 A task computing offloading method based on MATD3 in vehicle-mounted edge computing, comprising the following steps:

[0030] Step 1, introduce a task scheduling priority algorithm, obtain the task priority according to the execution cost and internal dependency of the task, wherein the execution cost is related to the number of RSUs within the communication range of the vehicle;

[0031] Specifically:

[0032] 1. Construct a system model: including constructing a network model, a first application model, a second application model and a computing model.

[0033] Construct a network model:

[0034] The VEC network model is as shown in Figure 1 There are n ICVs in the road, denoted as {γ1, γ2,..., γ n}, and m RSUs, denoted as {Λ1, Λ2,.., Λ m}; the coverage radii of the RSUs are {r1, r2,..., r m}, and the communication radius of the vehicle is r v . We consider the case where the RSU resources are limited, model the RSU as a multi-core processor, and assume that each RSU has z CPU cores, i.e. at the same time, at most z tasks can be processed. There is a data center on the access network side, which is responsible for training deep reinforcement models and can also collect information from each MEC server, such as the remaining CPU core situation in the MEC server, the computing power of each CPU core, etc.

[0035] To better describe the application generation, transmission and processing process in ICV, we divide the vehicle travel time into t time slices, each with a length of l. Within the same time slice, the VEC system is quasi-static, i.e. the relative position of the vehicle and the wireless channel state remain unchanged. In each time slice, the ICV generates applications of different application types with a probability of ε.

[0036] The steps of computation offloading are as follows:

[0037] (1) When the ICV generates an application, a request is sent to the nearby data center to obtain information about the nearby RSUs;

[0038] (2) The ICV formulates an offloading strategy according to the known computing resources of the RSUs and the dependency relationship of the offloaded tasks;

[0039] (3) The ICV executes the offloading strategy and schedules the tasks to be executed locally or offloaded to the designated RSUs according to the current state of the system;

[0040] (4) The RSU executes the tasks and sends the running results to the ICV, and sends its own information to the data center.

[0041] A first application model is constructed:

[0042] The applications generated by vehicles are usually computationally intensive and are usually composed of multiple tasks with dependency relationships. The limited computing capacity of the ICV cannot meet the deadline constraints of the application. Therefore, in order to make full use of the computing resources of the RSUs within the communication range of the ICV, the application is decomposed into multiple computing tasks with dependency relationships. The ICV application decomposition model is shown in Figure 2 i The application generated by the ICV can be divided into multiple computing tasks, denoted as where G i is the dependency relationship between tasks, is the deadline constraint of the ICV i application. There is a dependency relationship between the tasks after application decomposition, which can be represented by a directed acyclic graph G i = (N i , E i ), where is the set of tasks after application decomposition; the directed edge set constructed by the dependency relationship between tasks can be represented as E i = {e i，u，v |f(e i，u，v ) = 1, 1 ≤ u, v ≤ |N i |, u ≠ v}, f(e i，u，v ) = 1 indicates that there is a directed edge τ i,u → τ i,v , τ​i,v For τ i,v , the predecessor task of τ For each task, define d i，u , the input data size of τ i，u , c , the number of CPU cycles required to complete τ i , and D i，u , the maximum allowed delay of τ

[0043] Construct the second application model:

[0044] We define B as the channel bandwidth, p i as the transmission power of ICV, h as the channel fading factor, and σ 2 as the noise power. According to Shannon's theorem, the uplink transmission rate between ICV i and RSU j at the kth time slice can be approximated as:

[0045]

[0046] Construct the computation model:

[0047] The computation model can be divided into two parts: local execution of tasks and RSU execution of tasks

[0048] Local execution of tasks:

[0049] The execution delay of task u on ICV i locally is:

[0050]

[0051] where C is the local computing capability of ICV i (in CPU frequency per second).

[0052] The power of executing task u on ICV i is:

[0053]

[0054] where k icv is the vehicle energy consumption coefficient (in CPU frequency per second).

[0055] The energy consumption of processing task u on ICV i is:

[0056]

[0057] RSU performs tasks

[0058] RSU processes according to the current number of free CPU cores, and the tasks executed need to meet all the completed predecessor tasks.

[0059] ICV i The upload delay of task u on ICV represents:

[0060]

[0061] In RSU j The execution delay represents:

[0062]

[0063] Task u in RSU j The queuing delay of the task represents

[0064]

[0065] Where K is the set of tasks unloaded to RSU j .

[0066] ICV i The execution delay of task u on RSU j :

[0067]

[0068] The power uploaded by the vehicle to RSU j Processing is:

[0069]

[0070] Where κ rsu is the energy consumption coefficient, is the local computing power of RSU j . The transmission energy of task u on ICV i to RSU j :

[0071]

[0072] The energy consumption of task u on ICV i processing on RSU j :

[0073]

[0074] 2. Problem definition:

[0075] If the application is not processed before the deadline constraint, the application is considered to be offloaded unsuccessfully. We use 0-1 decision variables to represent the ICV i The task offloading mode taken: v i ICV represents the current time slice i Whether to select local processing of tasks; ICV represents the current time slice i Whether to offload tasks to RSU within the communication range j . RSU has task data to process data. Our goal is to seek the optimal offloading to minimize the task offloading failure rate energy consumption, and define the system cost of ICV i

[0076]

[0077] Wherein ICV at the kth time slice i Energy consumption generated by local processing of computing tasks, Communication energy consumption generated in the process of offloading task u to RSU at the kth time j ξ, Γ are weight factors, and ξ+Γ=1.

[0078] The offloading failure penalty of the kth time slice, The total deadline constraint of the offloading failed application in the current time slice is represented by Qk, and then It can be represented as:

[0079]

[0080] The optimization problem can be jointly expressed as the problem of minimizing the total cost in the system, described as follows:

[0081]

[0082] The main goal of the above optimization problem is to minimize the total cost of the system by making reasonable offloading decisions.

[0083] The following are further constraints of the optimization problem.

[0084]

[0085]

[0086] C3: a i，j ∈{0,1,2,...,g} (15)

[0087] ​Constraint C1 ensures that the execution delay does not exceed the maximum allowed delay. C2 and C3 are constraints on decision variables. C2 constraint indicates that a task can only be executed on one server, and C3 further describes the specific offloading destination of the task. To solve the problem, it is of great significance to obtain the optimal task offloading decision with the minimum task offloading failure rate and the minimum task energy consumption. For this purpose, we design a task offloading algorithm based on MARL to solve this problem.

[0088] Step 2, transform the dependent task offloading problem into an optimal strategy problem under Markov decision process, define state space, action space and reward function, wherein the state space includes vehicle state and MEC server state, the action space is to determine the target MEC server of each task, and the reward function is related to the offloading failure rate and energy consumption of the application;

[0089] Specifically:

[0090] A task scheduling priority algorithm is introduced, and the state, action and reward function are defined to approximate the optimization problem to MDP. Finally, we use a deep reinforcement learning algorithm based on multi-agent double-delay deep deterministic policy gradient (MATD3) to optimize the offloading strategy to minimize the system cost.

[0091] Construct a task scheduling priority algorithm:

[0092] The task offloading decision process is divided into two steps: offloading sequence and offloading decision. In order to reduce the total processing delay of the application, it is necessary to obtain the optimal task offloading sequence to reduce the task offloading. This paper proposes two important concepts of task τ i，u , namely priority and execution cost. The priority of the task is obtained according to the execution cost and internal dependency, and the execution cost can be expressed as

[0093]

[0094] Where g is the number of RSUs within the communication range of the vehicle. In the task described by the DAG graph, we define the last task as the exit task, and each task has at least one path to the exit task, which we call the exit critical path ep.

[0095] In this paper, we use a backward recursive method to calculate the priority of the task. Specifically, we first calculate the priority of the task, and then calculate the priority of each non-exit task according to the structure of the DAG graph. When calculating the priority of the non-exit task, the exit critical path with the highest priority will be selected. The priority of task τ i，u can be defined as

[0096]

[0097] Constructing MDP model:

[0098] Reinforcement learning can be described as an intelligence that interacts with the environment, constantly learns for a specific goal, and obtains the maximum reward value. In this paper, we use a distributed structure to obtain the optimal offloading strategy, and the agent refers to all ICVs, and the environment refers to the VEC environment. In the following, according to the model proposed in this paper, the state space, action space and reward function of the system are defined.

[0099] In the Markov decision process, the agent changes its state by taking action, obtains reward and interacts with the environment in a cycle. In this paper, the task offloading problem will be transformed into the optimal strategy problem under MDP.

[0100] (1) State space: The state space is the state of the ICV i at the current time and the state of the MEC server.

[0101] The vehicle state is composed of the position of the vehicle itself The required CPU cycles c of the current task i，u The size d of the computing task i，u And the computing complexity c i，u The MEC state includes the number of unoccupied CPU cores z in the RSU server k The computing power

[0102]

[0103] (2) Action space: In VEC, ICV i needs to make offloading decisions according to the system state and decision strategy to determine the target MEC server for each task. In the kth time slice, let denote the offloading decision of task u, then the action space of ICV i can be expressed as:

[0104]

[0105] (3) Reward function: In the time period t, the environment gives a reward according to the action taken. Our goal is to minimize the offloading failure rate and energy consumption of the application. Therefore, in the kth time slice, assuming that the set of applications that fail to offload in the current time slice is Qk, then the reward function of ICV i is:

[0106]

[0107] where α+β=1.

[0108] Step 3, design a strategy based on distributed deep reinforcement learning, including TD3 algorithm, MATD3 algorithm and MATD3-based computing offloading strategy;

[0109] Specifically: this problem involves the competition of multiple vehicles for computing resources, and determining the offloading decision needs to consider the dependency relationship of tasks, the limited computing resources of RSU and the mobility of vehicles. In addition, the offloading decision of each vehicle will affect the offloading failure rate and energy consumption. To solve this problem, we designed a strategy based on distributed deep reinforcement learning to solve the proposed computing offloading problem, which consists of three parts: 1) TD3 algorithm; 2) MATD3 algorithm; 3) MATD3-based computing offloading strategy;

[0110] Step 4, in the TD3 algorithm, set the actor network and critic network, the actor network outputs actions according to the observed state, the critic network includes two Q networks for evaluating action quality, update Q networks by minimizing loss, update actor networks according to sampled policy gradient, update target network parameters according to online network parameters;

[0111] TD3 network is divided into actor network and critic network. Among them, the actor network (π φ ) takes the observed state as input and outputs actions through multiple fully connected layers. Inspired by double Q learning, the critic network in TD3 consists of two Q-networks . These two critic networks can independently evaluate the quality of actions generated by the actor network, which takes the state and action as input and outputs the corresponding Q value. By using two critic networks, TD3 aims to reduce overestimation and variance in Q value estimation, thereby improving the stability and reliability of the training process.

[0112] To improve the stability of training, we introduce the target networks of the critic and the actor, which are represented as Q networks are updated by minimizing loss as shown in equation (22), and actor networks are updated according to sampled policy gradient as shown in equation (24). The parameters of the target network are updated according to the soft update of the online network parameters.

[0113]

[0114] where ρ represents the soft update coefficient.

[0115] Step 5, in the MATD3 algorithm, multiple TD3-based agents are assigned to learn collaborative control strategies, using a centralized training and decentralized execution framework, centralized Q value calculation considers all agent information, loss function is defined according to Q value calculation, actor network gradient is calculated according to loss function, target network parameters are updated according to specific equations;

[0116] In the MATD3 method, multiple TD3-based agents are designated to learn a cooperative control policy. Each agent has its own actor network and critic network. Therefore, the MATD3 method is considered as a multi-agent extension of single-agent TD3. MATD3 adopts a framework of centralized training and decentralized execution, in which Q-value estimation considers the information of all agents, but agents still perform their actions according to local measurements. In this context, the centralized Q-value calculation is rewritten as shown in equation (24):

[0117]

[0118] where γ is the discount factor. Then the loss function can be defined as:

[0119]

[0120] Then the gradient of the actor network can be expressed as

[0121]

[0122] The parameters of the target network are updated using the following equation:

[0123]

[0124] Step 6, in the MATD3-based computing offloading policy, an intelligent module is set up on each ICV, containing an actor network and a critic network. During the training process, the master actor network and the master critic network are trained centrally, and the target actor network and the target critic network are executed in a decentralized manner. The offloading process is divided into an interaction phase and a training phase. The interaction phase includes steps such as initializing network parameters, generating applications, obtaining task scheduling priorities, selecting actions and interacting with the environment. The training phase includes steps such as updating the master critic network, updating the master actor network, updating the target critic network and the target actor network.

[0125] The implementation process of the computing offloading proposed in this paper is shown in Figure 3 Each ICV in the system has an intelligent module, and each intelligent module has an actor network and a critic network. During the training process, the master actor network and the master critic network are trained in a centralized manner, while the target actor network and the target critic network are executed in a decentralized manner.

[0126] In the distributed execution process, the ICV i first downloads the training weights of the actor network from the data center and loads them into its own target actor network. Then the ICV observes the environment to obtain the state The actor network mounted on the ICV i outputs the action and executes it to obtain the next state and instant rewards and get a set of experiences The data center puts the joint state s, joint action a, joint reward r, and joint state s' of all vehicles into the experience replay buffer D. When the stored experiences reach a predefined value, a small batch of samples is sampled from the experience replay buffer.

[0127] We divide the offloading process into two stages, the interaction stage and the training stage.

[0128] 1) Interaction stage: The details of the interaction stage are shown in Algorithm 1. Each ICV initializes the parameters of the actor network and critic network from the data center. ICVs will generate an application with a certain probability, decompose it to get the scheduling priority of the task according to the task priority algorithm, and get the initial state of the VEC environment. At the beginning of the time slice, the ICV selects an action according to the obtained environment state and interacts with the environment to obtain the reward and the next state. In addition, the transition is stored in the experience replay buffer D.

[0129] 2) Training stage: We provide the details of the learning stage in Algorithm 2. Network training contains three steps: first, update the main critic network. Second, update the main actor network. Finally, update the target critic network and the target actor network. This stage starts based on known actor network and critic network parameters and other parameters.

[0130] For each ICV, network training contains three steps: first, update the main critic network. Second, update the main actor network. Finally, update the target critic network and the target actor network.

[0131] The data center trains the main critic network, and the goal of the training process is to maximize the Q value of the vehicle in equation (24). The data center samples a random batch of transitions from the experience replay buffer D, calculates the Q value of the main critic network, and updates the parameters of the main critic network by minimizing the loss function in equation (25).

[0132] Update the parameters of the main actor network: The parameters of the main actor network can be updated by the deterministic policy gradient descent method in equation (26).

[0133] Update the parameters of the target network: The target actor network parameters and the target critic network parameters of each ICV are updated according to equation (27)

[0134] During network training, the actor network should obtain the local state of the vehicle, and the critic network needs the action of all vehicles to output the Q value. After completing the training process, the execution process only needs the target actor network, and each agent can only select the optimal action according to its state.

[0135] In this process, the actor network outputs action instructions according to the current state, and the critic network on the ICV is responsible for estimating the Q value of the action, that is, the numerical expectation of the long-term return, thereby guiding the actor network to update the strategy in the direction of improving the cumulative reward. Although the Q value estimation of the critic network integrates all the information of the ICV, the action selection of each ICV only depends on the local state information observed by itself.

[0136] In this embodiment, the vehicle state includes the position of the vehicle itself, the CPU cycle required by the current task, the size of the computing task and the computing complexity parameter, and the MEC server state includes the number of unoccupied CPU cores and the computing capacity in the MEC server in the RSU server.

[0137] In this embodiment, in the interaction phase of the MATD3-based computing offloading strategy, each ICV initializes the parameters of the actor network and the critic network from the data center, generates an application and decomposes it with a certain probability, obtains the scheduling priority of the task according to the task priority algorithm, obtains the initial state of the VEC environment, selects an action according to the environment state at the beginning of the time slice and interacts with the environment, obtains the reward and the next state, and stores the conversion in the experience replay buffer D.

[0138] In this embodiment, in the training phase of the MATD3-based computing offloading strategy, network training includes three steps:

[0139] Step a, the data center trains the main critic network, samples a random batch of conversions from the experience replay buffer D, calculates the Q value of the main critic network, and updates the parameters of the main critic network by minimizing the loss function;

[0140] Step b, the parameters of the main actor network are updated by the deterministic policy gradient descent method;

[0141] Step c, the target actor network parameters and the target critic network parameters of each ICV are updated according to a specific equation.

[0142] In this embodiment, in the network training process, the actor network outputs action instructions according to the current state, the critic network estimates the Q value of the action, and guides the actor network to update the strategy, and the action selection of each ICV only depends on the local state information observed by itself.

[0143]

[0144]

[0145]

[0146] In order to explain the possible application scenarios, technical principles, specific implementation schemes, and the purposes and effects of the present application in detail, the following will be described in detail in combination with the specific embodiments listed and the accompanying drawings. The embodiments described herein are only used to more clearly illustrate the technical solutions of the present application, and therefore only serve as examples, and cannot limit the protection scope of the present application.

[0147] In this paper, the term "embodiment" means that the specific features, structures or characteristics described in combination with the embodiment can be included in at least one embodiment of the present application. The term "embodiment" appearing at various positions in the specification does not necessarily refer to the same embodiment, and does not particularly limit the independence or association between other embodiments. In principle, in the present application, as long as there is no technical contradiction or conflict, each technical feature mentioned in each embodiment can be combined in any way to form a corresponding implementable technical solution.

[0148] Unless otherwise defined, the meaning of the technical terms used herein is the same as that generally understood by those skilled in the art to which the present application belongs; the use of related terms herein is only for the purpose of describing specific embodiments, and is not intended to limit the present application.

[0149] Although the embodiments of the present application have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and variations can be made to these embodiments without departing from the principles and spirit of the present application, and the scope of the present application is defined by the appended claims and their equivalents.

Claims

1. A method for offloading task calculation based on MATD3 in vehicle-mounted edge computing, characterized in that: The steps include: Step 1: Introduce a task scheduling priority algorithm to obtain the task priority based on the execution cost and internal dependencies of the task, where the execution cost is related to the number of RSUs within the vehicle's communication range; Step 2: Convert the dependent task offloading problem into an optimal policy problem under a Markov decision process. Define the state space, action space, and reward function. The state space includes the vehicle state and MEC server state. The action space determines the target MEC server for each task. The reward function is related to the application's offloading failure rate and energy consumption. Step 3: Design a distributed deep reinforcement learning strategy, which includes the TD3 algorithm, the MATD3 algorithm, and a computation offloading strategy based on MATD3. Step 4. In the TD3 algorithm, an actor network and a critic network are set up. The actor network outputs actions based on the observed state. The critic network includes two Q networks for evaluating the quality of actions. The Q network is updated by minimizing the loss. The actor network is updated according to the sampled policy gradient. The parameters of the target network are soft-updated according to the online network parameters. Step 5. In the MATD3 algorithm, multiple TD3-based agents are assigned to learn collaborative control strategies, using a centralized training and distributed execution framework. The centralized Q-value calculation considers the information of all agents, the loss function is defined based on the Q-value calculation, the gradient of the actor network is calculated based on the loss function, and the parameters of the target network are updated according to specific equations; Step 6. In the MATD3-based computation offloading strategy, an intelligent module is set up on each ICV, including an actor network and a critic network. During the training process, the main actor network and the main critic network are trained centrally, while the target actor network and the target critic network are executed decentralized. The offloading process is divided into an interaction phase and a training phase. The interaction phase includes the steps of initializing network parameters, generating applications, obtaining task scheduling priorities, selecting actions, and interacting with the environment. The training phase includes the steps of updating the main critic network, updating the main actor network, updating the target critic network, and updating the target actor network.

2. The method for offloading task calculation based on MATD3 in vehicle-mounted edge computing according to claim 1, characterized in that: The vehicle status includes the vehicle's own location, the CPU cycles required for the current task, the size of the computing task and the computational complexity parameters; the MEC server status includes the number of unoccupied CPU cores and computing power in the MEC server in the RSU server.

3. The method for offloading task calculation based on MATD3 in vehicle-mounted edge computing according to claim 2, characterized in that: In the interaction phase of the MATD3-based computation offloading strategy, each ICV initializes the parameters of the actor network and critic network from the data center, generates and decomposes applications with a certain probability, obtains the scheduling priority of the task according to the task priority algorithm, obtains the initial state of the VEC environment, selects an action based on the environment state at the beginning of the time slice and interacts with the environment, obtains the reward and the next state, and stores the transition in the experience replay buffer D.

4. The method for offloading task calculation based on MATD3 in vehicle-mounted edge computing according to claim 3 is characterized in that: In the training phase of the MATD3-based computation offloading strategy, network training consists of three steps: Step a: The data center trains the master critic network, samples random batches of transitions from the experience replay buffer D, calculates the Q value of the master critic network, and updates the parameters of the master critic network by minimizing the loss function; Step b, update the parameters of the main actor network through the deterministic policy gradient descent method; Step c: Update the target actor network parameters and target critic network parameters of each ICV according to the specific equation.

5. The method for offloading task calculation based on MATD3 in vehicle-mounted edge computing according to claim 4 is characterized in that: During network training, the actor network outputs action instructions based on the current state, and the critic network estimates the Q value of the action to guide the actor network to update its strategy. The action selection of each ICV depends only on its own observed local state information.

Images