MATD3-based task calculation unloading method in vehicle-mounted edge calculation

Abstract

The invention belongs to a computing unloading strategy in the field of parallel and distributed high-performance computing, and particularly relates to a task computing unloading method based on MATD3 in vehicle-mounted edge computing, a task scheduling priority algorithm is introduced, and a new thought is provided for computing unloading of the Internet of Vehicles. The task priority is determined by considering the execution cost and the internal dependency relationship of tasks, optimization of the task unloading sequence is facilitated, the total processing delay of an application program is reduced, a multi-agent reinforcement learning method is adopted, especially a strategy based on MATD3 is adopted, multiple agents are cooperatively controlled, and the task unloading efficiency is improved. Complex factors such as competition of multiple vehicles for computing resources, task dependency, RSU limited computing resources and vehicle mobility are fully considered, the method can better adapt to the dynamic change environment of the Internet of Vehicles, the optimal unloading strategy of the vehicle tasks is determined, and therefore the unloading failure rate and energy consumption of the tasks in the system are minimized.

Description

Technical Field

[0001] The present invention belongs to a computing offloading strategy in the field of parallel and distributed high-performance computing, and specifically relates to on-board edge computing that takes into account the dependency structure of subtasks between new on-board applications generated by intelligent connected vehicles and the dynamic changes of edge nodes in the vehicle's environment. Background Art

[0002] With the advancement of wireless access technology and the increasing popularity of intelligent and connected vehicles (ICV), the Internet of Vehicles has become an important paradigm for 5G networks. The Internet of Vehicles makes various reliable vehicle services possible, especially new in-vehicle applications such as autonomous driving, augmented reality, and in-vehicle multimedia services, which promote the development of intelligent transportation and improve traffic safety and efficiency. These applications are usually computationally intensive, which naturally leads to high energy consumption and latency. However, due to the physical characteristics of vehicles, vehicles are limited in computing resources and memory size. It is difficult to guarantee the needs of new applications by relying solely on resource-constrained in-vehicle terminals. In order to resolve the contradiction between the needs of these applications and the limitations of resource-constrained ICVs, researchers introduced mobile edge computing into the Internet of Vehicles and proposed vehicular edge computing (VEC). VEC meets the ever-expanding demand for vehicle computing power and significantly improves the service quality of vehicle tasks by moving cloud computing resources down to the edge of the network close to users and allowing vehicles to offload applications to roadside units (RSUs) equipped with MEC servers.

[0003] At present, the demand for data communication and computing resources in vehicle applications is increasing day by day. However, how to make offloading decisions and resource allocation in time-varying and uncertain IoV scenarios is a key challenge. Designing efficient vehicle computing offloading strategies and further utilizing computing and communication resources to assist in executing new applications generated during ICV driving has become a key research issue. Traditional methods such as Lyapunov optimization and game theory algorithms can provide approximate optimal solutions in a short time, but these algorithms have poor robustness and are difficult to achieve real-time decision-making. As a promising artificial intelligence technology, deep reinforcement learning (DRL) has great advantages in dealing with uncertain decision-making problems. Compared with heuristic methods, DRL methods have stronger robustness. However, traditional single-agent algorithms are not suitable for multi-vehicle environments, because the behavior of each vehicle affects the entire environment. When a vehicle changes its strategy, it also changes the environment of other vehicles, which leads to continuous changes in the environment, thereby increasing the instability of the learning process and making it difficult to meet the convergence conditions. Moreover, in a dynamic and unstable multi-vehicle environment, past experience may quickly become no longer applicable, and key technologies in traditional single-agent algorithms such as experience replay cannot be directly used. Summary of the invention

[0004] 1. Technical issues to be resolved

[0005] The purpose of the present invention is to provide a computing offloading strategy that takes into account the structural differences of new vehicle-mounted applications generated by intelligent connected vehicles and the dynamic changes of edge nodes in the vehicle environment, with the aim of reducing the energy consumption in computing offloading and the failure rate of application offloading. This method specifically considers the impact of vehicle mobility and application dependency structure in the VEC environment, effectively reducing the energy consumption and application offloading failure rate in vehicle-mounted edge computing.

[0006] In order to achieve the purpose of the above invention, the design technical solution of the present invention is: the computation offloading problem in the multi-vehicle scenario is established as a Markov decision process model. The application is divided into multiple tasks, and the dependency relationship is described using a directed acyclic graph. On this basis, a task scheduling priority algorithm is proposed to further reduce the application offloading delay.

[0007] (II) Technical solution

[0008] To achieve the above object, the present invention provides the following technical solution: a task calculation offloading method based on MATD3 in vehicle-mounted edge computing, comprising the following steps:

[0009] Step 1: Introduce a task scheduling priority algorithm to obtain the task priority based on the execution cost and internal dependencies of the task, where the execution cost is related to the number of RSUs within the vehicle's communication range;

[0010] Step 2: Convert the dependent task offloading problem into the optimal strategy problem under the Markov decision process, define the state space, action space and reward function, where the state space includes the vehicle state and the MEC server state, the action space is to determine the target MEC server for each task, and the reward function is related to the offloading failure rate and energy consumption of the application;

[0011] Step 3: Design a strategy based on distributed deep reinforcement learning, which includes the TD3 algorithm, the MATD3 algorithm, and a computational offloading strategy based on MATD3;

[0012] Step 4. In the TD3 algorithm, an actor network and a critic network are set up. The actor network outputs actions based on the observed state. The critic network includes two Q networks for evaluating the quality of actions. The Q network is updated by minimizing the loss. The actor network is updated according to the sampled policy gradient. The parameters of the target network are soft-updated according to the online network parameters.

[0013] Step 5. In the MATD3 algorithm, multiple TD3-based agents are assigned to learn collaborative control strategies, using a centralized training and distributed execution framework. The centralized Q-value calculation considers the information of all agents, the loss function is defined based on the Q-value calculation, the gradient of the actor network is calculated based on the loss function, and the parameters of the target network are updated according to specific equations;

[0014] Step 6. In the computational offloading strategy based on MATD3, an intelligent module is set on each ICV, including an actor network and a critic network. During the training process, the main actor network and the main critic network are trained centrally, and the target actor network and the target critic network are executed decentralized. The offloading process is divided into an interaction phase and a training phase. The interaction phase includes steps such as initializing network parameters, generating applications, obtaining task scheduling priorities, selecting actions and interacting with the environment. The training phase includes steps such as updating the main critic network, updating the main actor network, updating the target critic network and the target actor network.

[0015] Preferably, the vehicle status includes the location of the vehicle itself, the CPU cycles required for the current task, the size of the computing task and the computing complexity parameters, and the MEC server status includes the number of unoccupied CPU cores and computing power in the MEC server in the RSU server.

[0016] Preferably, in the interaction phase of the MATD3-based computation offloading strategy, each ICV initializes the parameters of the actor network and the critic network from the data center, generates and decomposes applications with a certain probability, obtains the scheduling priority of the task according to the task priority algorithm, obtains the initial state of the VEC environment, selects an action according to the environment state at the beginning of the time slice and interacts with the environment, obtains the reward and the next state, and stores the transformation in the experience replay buffer D.

[0017] Preferably, in the training phase of the MATD3-based computation offloading strategy, network training includes three steps:

[0018] Step a: The data center trains the master critic network, samples random batches of transformations from the experience replay buffer D, calculates the Q value of the master critic network, and updates the parameters of the master critic network by minimizing the loss function.

[0019] Step b, update the parameters of the main actor network through the deterministic policy gradient descent method;

[0020] Step c: Update the target actor network parameters and target critic network parameters of each ICV according to the specific equation.

[0021] Preferably, during the network training process, the actor network outputs action instructions based on the current state, the critic network estimates the Q value of the action, and guides the actor network to update the strategy. The action selection of each ICV depends only on its own observed local state information.

[0022] (III) Beneficial effects

[0023] Compared with the prior art, the present invention provides a method for offloading task calculation based on MATD3 in vehicle-mounted edge computing, which has the following beneficial effects:

[0024] The MATD3-based task computing offloading method in the vehicle edge computing introduces a task scheduling priority algorithm, which provides a new idea for the computing offloading of the Internet of Vehicles. By considering the execution cost and internal dependencies of the task to determine the task priority, it helps to optimize the task offloading order and reduce the total processing delay of the application. The multi-agent reinforcement learning method, especially the MATD3-based strategy, is used to coordinate the control of multiple agents, fully considering the complex factors such as the competition for computing resources among multiple vehicles, task dependencies, RSU limited computing resources, and vehicle mobility. It can better adapt to the dynamically changing environment of the Internet of Vehicles and determine the optimal offloading strategy for vehicle tasks, thereby minimizing the offloading failure rate and energy consumption of tasks in the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 This is a schematic diagram of the vehicle edge network model of the present invention;

[0026] Figure 2 It is a schematic diagram of the decomposition model of the structure application of the present invention;

[0027] Figure 3 This is a schematic diagram of the structure of the distributed computing offloading algorithm based on MATD3 of the present invention. DETAILED DESCRIPTION

[0028] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0029] See also Figure 1-3 , a task computing offloading method based on MATD3 in vehicle-mounted edge computing, comprising the following steps:

[0030] Step 1: Introduce a task scheduling priority algorithm to obtain the task priority based on the execution cost and internal dependencies of the task, where the execution cost is related to the number of RSUs within the vehicle's communication range;

[0031] Specific:

[0032] 1. Build a system model: including building a network model, a first application model, a second application model and a computing model.

[0033] Build the network model:

[0034] VEC network model Figure 1 As shown, there are n ICVs on the road, represented by {γ1,γ2,...,γ n}, there are m RSUs, represented by {Λ1, Λ2, .., Λ m}; The coverage radius of RSU is {r1, r2, ..., r m}, the communication radius of the vehicle is r v We consider the case of limited RSU resources and model RSU as a multi-core processor. Assume that each RSU has z CPU cores, that is, it can process at most z tasks at the same time. There is a data center on the access network side. The data center is responsible for training the deep reinforcement model and can also collect information from each MEC server, such as the remaining CPU cores in the MEC server and the computing power of each CPU core.

[0035] In order to better describe the application generation, transmission and processing process in ICV, we divide the vehicle driving time into t time slices, each time slice is l. In the same time slice, the VEC system is quasi-static, that is, the relative position of the vehicle and the state of the wireless channel remain unchanged. In each time slice, ICV generates applications of different application types with a probability of ε.

[0036] The steps to calculate the unloading are as follows:

[0037] (1) When the ICV generates an application, it sends a request to a nearby data center to obtain information about nearby RSUs;

[0038] (2) ICV formulates an offloading strategy based on the known computing resources of the RSU and the dependencies of the offloading tasks;

[0039] (3) ICV executes the offloading strategy and schedules the task to be executed locally or offloaded to the specified RSU according to the current state of the system;

[0040] (4) RSU executes the task and sends the operation results to ICV, and sends its own information to the data center.

[0041] Build the first application model:

[0042] Vehicle-generated applications are usually computationally intensive applications, usually composed of multiple tasks with dependencies. The limited computing power of ICV cannot meet the deadline constraints of the application. Therefore, in order to fully utilize the computing resources of RSUs within the communication range of ICV, the application is decomposed into multiple computing tasks with dependencies between them. The ICV application decomposition model is as follows: Figure 2 As shown, ICV i The resulting application can be divided into multiple computing tasks, represented as Among them G i is the dependency relationship between tasks, ICV i The deadline constraint of the application. There are dependencies between the tasks after the application is decomposed, which can be represented by a directed acyclic graph G i =(N i , E i ) indicates that is the set of tasks after application decomposition; the directed edge set constructed by the dependencies between tasks can be expressed as E i ={e i，u，v |f(e i，u，v )=1,1≤u,v≤|N i |,u≠v},f(e i，u，v )=1 means there is a τ i,u → i,v Directed edges, τi,v is τ i,v For each task, it is defined as Including input data size d i，u , calculate the number of CPU cycles c required i，u , and the maximum allowed delay of the task In the VEC scenario, each task can choose to process it locally or in an RSU. Define g as the number of RSUs within the ICV communication range. For ICV i , there are g+1 (1≤g≤m+1) unloading decisions, and the decision variable a is defined i，u Indicates an uninstall decision.

[0043] Build the second application model:

[0044] We define the channel bandwidth as B and the ICV transmission power as p i , the channel fading factor is h, σ 2 is the noise power. According to Shannon’s theorem, in the kth time slice, ICV i To RSU j The data uplink transmission rate between can be approximated as:

[0045]

[0046] Build a computational model:

[0047] The computing model can be divided into two parts: local execution task and RSU execution task

[0048] Local execution tasks:

[0049] ICV i The local execution delay of task u on is:

[0050]

[0051] in ICV i The local computing power (CPU frequency in cycles / second).

[0052] ICV i The power required to perform the task is:

[0053]

[0054] where k icv is the vehicle energy consumption factor (CPU frequency in cycles / second).

[0055] ICV i The energy consumption of processing task u is:

[0056]

[0057] RSU Execution Task

[0058] RSU processes tasks based on the number of available CPU cores. The tasks to be executed must satisfy the requirements of all their predecessor tasks being completed.

[0059] ICV i The upload delay of task u on is expressed as:

[0060]

[0061] In RSU j The execution delay on is expressed as:

[0062]

[0063] Task u in RSU j The queuing delay of the above task is expressed as

[0064]

[0065] Where K is offloaded to RSU j A collection of tasks.

[0066] ICV i On the task u in RSU j Execution Delay:

[0067]

[0068] Vehicle upload to RSU j The power processed is:

[0069]

[0070] where κ rsu is the energy consumption coefficient, For RSU j Local computing power. ICV i Task u to RSU j The transmission energy consumption is:

[0071]

[0072] ICV i On the task u in RSU j The energy consumption of the above processing is:

[0073]

[0074] 2. Problem definition:

[0075] If the application is not processed before the deadline, it is considered that the application has failed to be uninstalled. We use a 0-1 decision variable to represent ICV i Task offloading method adopted: v i Indicates the current time slice ICV i Whether to choose to process the task locally; Indicates the current time slice ICV i Whether to choose to offload tasks to RSUs within communication range j The RSU processes data only when it has task data. Our goal is to seek optimal offloading to minimize the energy consumption of task offloading failure rate and define ICV i The system cost is

[0076]

[0077] in ICV at the kth time slice i Energy consumption generated by local processing of computing tasks, Unload task u to RSU at time k j The communication energy consumption generated in the process, ξ, Γ are weight factors, and ξ+Γ=1.

[0078] is the unloading failure penalty for the kth time slice, is the total deadline constraint for applications that failed to be uninstalled in the current time slice, and Qk represents the set of applications that failed to be uninstalled in the current time slice. It can be expressed as:

[0079]

[0080] The optimization problem can be jointly formulated as the problem of minimizing the total cost in the system, as follows:

[0081]

[0082] The main goal of the above optimization problem is to minimize the total system cost by making reasonable offloading decisions.

[0083] The following are further constraints on the optimization problem.

[0084]

[0085]

[0086] C3:a i，j ∈{0,1,2,...,g} (15)

[0087] Constraint C1 ensures that the execution delay does not exceed the maximum allowed delay. C2 and C3 are constraints on the decision variables. The C2 constraint indicates that the task can only be executed on one server, and C3 further describes the specific offloading destination of the task. In order to solve the problem, it is of great significance to obtain the optimal task offloading decision with the minimum task offloading failure rate and the minimum task energy consumption. To this end, we designed a MARL-based task offloading algorithm to solve this problem.

[0088] Step 2: Convert the dependent task offloading problem into the optimal strategy problem under the Markov decision process, define the state space, action space and reward function, where the state space includes the vehicle state and the MEC server state, the action space is to determine the target MEC server for each task, and the reward function is related to the offloading failure rate and energy consumption of the application;

[0089] Specific:

[0090] A task scheduling priority algorithm is introduced, and the state, action and reward functions are defined to approximate the optimization problem as an MDP. Finally, a deep reinforcement learning algorithm based on Multi-Agent Double Delayed Deep Deterministic Policy Gradient (MATD3) is used to optimize the computation offloading strategy to minimize the system cost.

[0091] Build a task scheduling priority algorithm:

[0092] The task offloading decision process is divided into two steps: offloading order and offloading decision. In order to reduce the total processing delay of the application, it is necessary to obtain the optimal task offloading order to reduce the offloading of tasks. This paper proposes a task τ i，u There are two important concepts in the task, namely priority and execution cost. The task priority is obtained based on the execution cost and internal dependencies. The execution cost can be expressed as

[0093]

[0094] Where g is the number of RSUs within the communication range of the vehicle. In the tasks described by the DAG graph, we define the last task as the exit task, and each task has at least one path to the exit task, which we call the exit critical path ep.

[0095] In this paper, we use the reverse recursive method to calculate the priority of tasks. Specifically, the priority of the task is first calculated, and then the priority of each non-exit task is reversely calculated according to the structure of the DAG graph. When calculating the priority of non-exit tasks, the exit critical path with the highest priority will be selected. Task τ i，u The priority can be defined as

[0096]

[0097] Constructing the MDP model:

[0098] Reinforcement learning can be described as a kind of intelligence that interacts with the environment and continuously learns for a specific goal to obtain the maximum reward value. In this paper, we adopt a distributed structure to obtain the optimal unloading strategy, the agent refers to all ICVs, and the environment refers to the VEC environment. In the following, the state space, action space, and reward function of the system are defined according to our proposed model.

[0099] In the Markov decision process, the agent changes its state by taking actions, obtains rewards and interacts with the environment in a cycle. This paper transforms the problem of dependent task offloading into the optimal strategy problem under MDP.

[0100] (1) State space: The state space is the ICV at the current moment i status and the status of the MEC server.

[0101] The vehicle state is determined by the vehicle's own position The CPU cycles c required by the current task i，u , calculate the size of the task d i，u And the computational complexity c i，u The MEC state includes the number of unoccupied CPU cores in the RSU server and the number of unoccupied CPU cores in the MEC server. k , computing power

[0102]

[0103] (2) Action Space: In VEC, ICV i It is necessary to make an offloading decision based on the system status and decision-making strategy and determine the target MEC server for each task. represents the offloading decision of task u, then ICV i The action space can be expressed as:

[0104]

[0105] (3) Reward function: In time period t, the environment gives rewards based on the actions taken. Our goal is to minimize the application uninstallation failure rate and energy consumption. Therefore, in the kth time slice, assuming that the set of applications that failed to be uninstalled in the current time slice is Qk, then ICV i The reward function is:

[0106]

[0107] Where α+β=1.

[0108] Step 3: Design a strategy based on distributed deep reinforcement learning, which includes the TD3 algorithm, the MATD3 algorithm, and a computational offloading strategy based on MATD3;

[0109] Specifically: This problem involves competition for computing resources among multiple vehicles. Determining the offloading decision requires considering the dependencies of tasks, the limited computing resources of RSUs, and the mobility of vehicles. In addition, the offloading decision of each vehicle will affect the offloading failure rate and energy consumption. To address this problem, we designed a strategy based on distributed deep reinforcement learning to solve the proposed computation offloading problem. The framework consists of three parts, namely 1) TD3 algorithm; 2) MATD3 algorithm; 3) computation offloading strategy based on MATD3;

[0110] Step 4. In the TD3 algorithm, an actor network and a critic network are set up. The actor network outputs actions based on the observed state. The critic network includes two Q networks for evaluating the quality of actions. The Q network is updated by minimizing the loss. The actor network is updated according to the sampled policy gradient. The parameters of the target network are soft-updated according to the online network parameters.

[0111] The TD3 network is divided into an actor network and a critic network. φ ) takes the observed state as input and outputs actions through multiple fully connected layers. Inspired by double Q learning, the critic network in TD3 consists of two Q-networks The two critic networks are able to independently evaluate the quality of actions produced by the actor network. They receive states and actions as input and output corresponding Q-values. By using two critic networks, TD3 aims to reduce overestimation and variance in Q-value estimation, thereby improving the stability and reliability of the training process.

[0112] To improve the stability of training, we introduce the target networks of critic and actor, which are denoted as The Q network is updated by minimizing the loss according to equation (22), and the actor network is updated according to the sampling policy gradient according to equation (24), as shown in equation (25). The parameters of the target network are soft-updated according to the online network parameters.

[0113]

[0114] Where ρ represents the soft update coefficient.

[0115] Step 5. In the MATD3 algorithm, multiple TD3-based agents are assigned to learn collaborative control strategies, using a centralized training and distributed execution framework. The centralized Q-value calculation considers the information of all agents, the loss function is defined based on the Q-value calculation, the gradient of the actor network is calculated based on the loss function, and the parameters of the target network are updated according to specific equations;

[0116] In the MATD3 method, multiple TD3-based agents are designated to learn a collaborative control policy. Each agent has its own actor network and critic network. Therefore, the MATD3 method is regarded as a multi-agent extension of the single-agent TD3. MATD3 adopts a centralized training and distributed execution framework, in which the Q-value estimation takes into account the information of all agents, but the agents still perform their actions based on local measurements. In this context, the centralized Q-value calculation is rewritten as shown in (24):

[0117]

[0118] Among them, γ is the discount factor. Then the loss function can be defined as:

[0119]

[0120] Then, the gradient of the actor network can be expressed as

[0121]

[0122] The parameters of the target network are updated using the following equations:

[0123]

[0124] Step 6. In the computational offloading strategy based on MATD3, an intelligent module is set on each ICV, including an actor network and a critic network. During the training process, the main actor network and the main critic network are trained centrally, and the target actor network and the target critic network are executed decentralized. The offloading process is divided into an interaction phase and a training phase. The interaction phase includes steps such as initializing network parameters, generating applications, obtaining task scheduling priorities, selecting actions and interacting with the environment. The training phase includes steps such as updating the main critic network, updating the main actor network, updating the target critic network and the target actor network.

[0125] The implementation process of computation offloading proposed in this paper is as follows Figure 3 As shown in the figure. Each ICV in the system has an intelligent module, and each intelligent module has an actor network and a critic network. During the training process, the main actor network and the main critic network are trained in a centralized manner, while the target actor network and the target critic network are performed in a decentralized manner.

[0126] During distributed execution, ICV i First, the training weights of the actor network are downloaded from the data center and loaded into its target actor network. Then ICV observes the environment to get the state On ICV i The actor network outputs actions on And execute to get the next state and instant rewards And get a set of experience The data center puts the joint state s, joint action a, joint reward r, and joint state s' of all vehicles into the experience replay buffer D. When the stored experience reaches a predefined value, a mini-batch of samples is sampled from the experience replay buffer.

[0127] We divide the offloading process into two phases, the interaction phase and the training phase.

[0128] 1) Interaction phase: The details of the interaction phase are shown in Algorithm 1. Each ICV initializes the parameters of the actor network and the critic network from the data center. ICV will generate applications with a certain probability, and after decomposition, it obtains the scheduling priority of the task according to the task priority algorithm and obtains the initial state of the VEC environment. At the beginning of the time slice, ICV selects an action based on the obtained environment state and interacts with the environment to obtain the reward and the next state. In addition, the conversion Stored in the experience replay buffer D.

[0129] 2) Training phase: We provide the details of the learning phase in Algorithm 2. Network training consists of three steps: First, update the master critic network. Second, update the master actor network. Finally, update the target critic network and target actor network. This phase starts based on the known actor network and critic network parameters and other parameters.

[0130] For each ICV, network training consists of three steps: First, update the master critic network. Second, update the master actor network. Finally, update the target critic network and target actor network.

[0131] The data center trains the master critic network, and the goal of the training process is to maximize the Q value of the vehicle in Equation (24). The data center samples a random batch of transformations from the experience replay buffer D, calculates the Q value of the master critic network, and updates the parameters of the master critic network by minimizing the loss function by Equation (25).

[0132] Update the parameters of the main actor network: The parameters of the main actor network can be updated by the deterministic policy gradient descent method (26).

[0133] Update the parameters of the target network: Update the target actor network parameters and target critic network parameters of each ICV according to formula (27)

[0134] During the network training process, the actor network should obtain the local state of the vehicle, and the critic network needs the actions of all vehicles to output the Q value. After completing the training process, the execution process only requires the target actor network, and each agent can only make the optimal action selection based on its state.

[0135] In this process, the actor network outputs action instructions based on the current state, and the critic network on the ICV is responsible for estimating the Q value of this action, that is, the numerical expectation of long-term return, thereby guiding the actor network to update its strategy in the direction of increasing cumulative rewards. Although the Q value estimate of the critic network integrates the information of all ICVs, the action selection of each ICV depends only on its own observed local state information.

[0136] In this embodiment, the vehicle state includes the position of the vehicle itself, the CPU cycle required for the current task, the size of the computing task and the computing complexity parameters, and the MEC server state includes the number of unoccupied CPU cores and computing power in the MEC server in the RSU server.

[0137] In this embodiment, in the interaction phase of the MATD3-based computation offloading strategy, each ICV initializes the parameters of the actor network and the critic network from the data center, generates and decomposes applications with a certain probability, obtains the scheduling priority of the task according to the task priority algorithm, obtains the initial state of the VEC environment, selects an action according to the environment state at the beginning of the time slice and interacts with the environment, obtains the reward and the next state, and stores the transformation in the experience replay buffer D.

[0138] In this embodiment, in the training phase of the MATD3-based computation offloading strategy, network training includes three steps:

[0139] Step a: The data center trains the master critic network, samples random batches of transformations from the experience replay buffer D, calculates the Q value of the master critic network, and updates the parameters of the master critic network by minimizing the loss function.

[0140] Step b, update the parameters of the main actor network through the deterministic policy gradient descent method;

[0141] Step c: Update the target actor network parameters and target critic network parameters of each ICV according to the specific equation.

[0142] In this embodiment, during the network training process, the actor network outputs action instructions based on the current state, the critic network estimates the Q value of the action, and guides the actor network to update the strategy. The action selection of each ICV only depends on its own observed local state information.

[0143]

[0144]

[0145]

[0146] In order to explain in detail the possible application scenarios, technical principles, specific schemes that can be implemented, and the purposes and effects that can be achieved, the following is a detailed description of the specific embodiments listed in conjunction with the accompanying drawings. The embodiments described herein are only used to more clearly illustrate the technical solutions of the present application, and are therefore only used as examples, and cannot be used to limit the scope of protection of the present application.

[0147] Reference to "embodiment" herein means that the specific features, structures or characteristics described in conjunction with the embodiment may be included in at least one embodiment of the present application. The term "embodiment" appearing in various places in the specification does not necessarily refer to the same embodiment, nor does it particularly limit its independence or association with other embodiments. In principle, in the present application, as long as there is no technical contradiction or conflict, the various technical features mentioned in the embodiments can be combined in any way to form a corresponding implementable technical solution.

[0148] Unless otherwise defined, the technical terms used in this document have the same meanings as those generally understood by those skilled in the art to which this application belongs; the use of relevant terms in this document is only for describing specific embodiments and is not intended to limit this application.

[0149] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A task computing offloading method based on MATD3 in vehicle-mounted edge computing, characterized in that: The steps include: Step 1: Introduce a task scheduling priority algorithm to obtain the task priority based on the execution cost and internal dependencies of the task, where the execution cost is related to the number of RSUs within the vehicle's communication range; Step 2: Convert the dependent task offloading problem into the optimal strategy problem under the Markov decision process, define the state space, action space and reward function, where the state space includes the vehicle state and the MEC server state, the action space is to determine the target MEC server for each task, and the reward function is related to the offloading failure rate and energy consumption of the application; Step 3: Design a strategy based on distributed deep reinforcement learning, which includes the TD3 algorithm, the MATD3 algorithm, and a computational offloading strategy based on MATD3; Step 4. In the TD3 algorithm, an actor network and a critic network are set up. The actor network outputs actions based on the observed state. The critic network includes two Q networks for evaluating the quality of actions. The Q network is updated by minimizing the loss. The actor network is updated according to the sampled policy gradient. The parameters of the target network are soft-updated according to the online network parameters. Step 5. In the MATD3 algorithm, multiple TD3-based agents are assigned to learn collaborative control strategies, using a centralized training and distributed execution framework. The centralized Q-value calculation considers the information of all agents, the loss function is defined based on the Q-value calculation, the gradient of the actor network is calculated based on the loss function, and the parameters of the target network are updated according to specific equations; Step 6. In the computational offloading strategy based on MATD3, an intelligent module is set on each ICV, including an actor network and a critic network. During the training process, the main actor network and the main critic network are trained centrally, and the target actor network and the target critic network are executed decentralized. The offloading process is divided into an interaction phase and a training phase. The interaction phase includes steps such as initializing network parameters, generating applications, obtaining task scheduling priorities, selecting actions and interacting with the environment. The training phase includes steps such as updating the main critic network, updating the main actor network, updating the target critic network and the target actor network.

2. According to claim 1, the method for offloading task calculation based on MATD3 in vehicle-mounted edge computing is characterized in that: The vehicle status includes the location of the vehicle itself, the CPU cycle required for the current task, the size of the computing task and the computing complexity parameters, and the MEC server status includes the number of unoccupied CPU cores and computing power in the MEC server in the RSU server.

3. According to claim 2, the method for offloading task calculation based on MATD3 in vehicle-mounted edge computing is characterized in that: In the interaction phase of the MATD3-based computation offloading strategy, each ICV initializes the parameters of the actor network and the critic network from the data center, generates and decomposes applications with a certain probability, obtains the scheduling priority of the task according to the task priority algorithm, obtains the initial state of the VEC environment, selects an action according to the environment state at the beginning of the time slice and interacts with the environment, obtains the reward and the next state, and stores the transition in the experience replay buffer D.

4. The method for offloading task calculation based on MATD3 in vehicle-mounted edge computing according to claim 3 is characterized in that: In the training phase of the MATD3-based computation offloading strategy, network training includes three steps: Step a: The data center trains the master critic network, samples random batches of transformations from the experience replay buffer D, calculates the Q value of the master critic network, and updates the parameters of the master critic network by minimizing the loss function. Step b, update the parameters of the main actor network through the deterministic policy gradient descent method; Step c: Update the target actor network parameters and target critic network parameters of each ICV according to the specific equation.

5. According to claim 4, the method for offloading task calculation based on MATD3 in vehicle-mounted edge computing is characterized in that: During the network training process, the actor network outputs action instructions based on the current state, the critic network estimates the Q value of the action, and guides the actor network to update the strategy. The action selection of each ICV depends only on its own observed local state information.

Images