Deep reinforcement learning vehicle edge calculation unloading method based on time sequence state perception and related equipment

Abstract

The invention provides a deep reinforcement learning vehicle edge calculation unloading method based on time sequence state perception and related equipment. According to the method, an LSTM network is nested in a feedforward neural network of an actor network and a critic network of a DDPG network, and a deep reinforcement learning network is constructed; defining a state space, an action space and a reward function of the deep reinforcement learning network, and training the deep reinforcement learning network by using the sample data in the experience buffer area to obtain a vehicle edge calculation unloading model; wherein when the state space is defined, priority numerical values are given to vehicles according to vehicle attributes and task sizes, and the problem of time delay sensitivity in a vehicle edge environment is solved; and the historical state and the current state of the vehicle are analyzed by using the LSTM network, and the time dependency relationship between the historical state and the current state is captured, so that the perception of the state change of the vehicle is realized, the accuracy of motion prediction is improved, the motion is better predicted, and the system cost is reduced.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a deep reinforcement learning vehicle edge computing unloading method based on temporal state perception and related equipment. Background Art

[0002] In the field of intelligent transportation, autonomous driving technology uses advanced environmental perception and precise vehicle control to improve driving efficiency for drivers. Autonomous driving technology involves heterogeneous tasks, computing, and wireless communications in the Internet of Vehicles (IoV), which have extremely high requirements for latency. However, due to the limited computing power of the vehicle, it cannot meet the service needs of the vehicle environment. At the same time, with the increase in the number of vehicles, the amount of data has increased sharply and more complex computing tasks have brought challenges such as high latency and limited network bandwidth.

[0003] Computation offloading is one of the key technologies of mobile edge computing (MEC). By transferring part or all of the tasks to the edge server, it effectively addresses the limitations of computing resources on mobile devices and the requirements for real-time performance and energy consumption. Based on the advantages of MEC and combined with Internet of Vehicles technology, researchers proposed the concept of vehicle edge computing (VEC), which enables distributed edge servers to effectively promote computation offloading and interaction between vehicles and roadside units through wireless access networks, significantly reducing the burden of computation-intensive tasks on vehicles.

[0004] The current computational offloading methods mainly include optimization or game-based methods, reinforcement learning (RL)-based methods, and deep reinforcement learning (DRL)-based dynamic offloading algorithms. Among them, the more typical deep reinforcement learning (DRL)-based dynamic offloading algorithms are the dual deep Q network (DDQN) algorithm that optimizes task offloading and resource allocation in edge cloud computing. By analyzing resource utilization and the current state of the edge cloud network, it dynamically makes offloading decisions to meet computing and communication needs. There is also a computational offloading algorithm based on AHP-DQN, which uses the analytic hierarchy process (AHP) to accurately allocate vehicle tasks and formulates offloading strategies based on real-time channel gains. This method effectively solves the challenges of limited terminal storage capacity and diversified network services encountered in the computational offloading process. However, the above DRL methods are only applicable to discrete actions, and if used in continuous action spaces, it may lead to dimensionality explosion. Therefore, the policy network is integrated into the original DQN structure, and some researchers have proposed the deep deterministic policy gradient algorithm (DDPG). This algorithm significantly improves the processing capability in the continuous action space and optimizes the task offloading strategy. Although the DDPG-based algorithm has achieved good results in the continuous action space, in the vehicle edge environment, factors such as vehicle mobility, channel condition attenuation, and task load queue changes cause dynamic changes in the environment, resulting in the network topology and computing resource availability becoming more dynamic and complex. At the same time, the delay sensitivity of different vehicle tasks will also affect the task offloading decision process, and some vehicles cannot obtain the required computing resources in time. Summary of the invention

[0005] Due to the delay sensitivity of vehicle tasks and the dynamic nature of the vehicle edge environment, implementing effective real-time offloading decisions and resource allocation while ensuring the quality of vehicle service is a challenging task. Therefore, in order to at least partially solve the delay sensitivity problem or the dynamic problem of the vehicle edge environment, the present invention provides a deep reinforcement learning vehicle edge computing offloading method based on temporal state perception and related equipment.

[0006] In a first aspect, the present invention provides a deep reinforcement learning vehicle edge computing unloading method based on temporal state perception, comprising:

[0007] Step 1: Nest the LSTM network into the feedforward neural network of the actor network and critic network of the DDPG network to construct a deep reinforcement learning network;

[0008] Step 2: define the state space of the deep reinforcement learning network; wherein each state vector in the state space is used to indicate the task queue length, vehicle coordinate position, actual processed task size, vehicle priority value and remaining computing resources of a single vehicle at time slot t; correspondingly, the LSTM network is used to receive the real-time state vector and integrate the real-time state vector with the historical state vector;

[0009] Step 3: defining an action space of the deep reinforcement learning network; wherein each action in the action space is used to indicate the task offloading ratio of the vehicle selected and the selected vehicle at time slot t;

[0010] Step 4: defining the reward function of the deep reinforcement learning network as a function that is negatively correlated with the system cost; wherein the system cost includes the total delay and total energy consumption spent on completing the computing task;

[0011] Step 5: Use the sample data in the experience buffer to train the deep reinforcement learning network to obtain a vehicle edge computing offloading model;

[0012] Step 6: Obtain the real-time state vector and input it into the vehicle edge computing unloading model to obtain the real-time unloading decision and resource allocation results.

[0013] Furthermore, the vehicle coordinate position is updated in the following manner:

[0014]

[0015] in, Indicates vehicle VE i At the coordinate position of time slot t, q i (t+1) represents vehicle VE i The coordinate position of the next time slot after time slot t, Δ represents the duration of a time slot, v i (t) represents vehicle VE i speed, β(t)∈[0,2π].

[0016] Furthermore, the task queue length is updated as follows:

[0017]

[0018] in, and Respectively represent vehicle VE i The length of the task queue between time slot t and the next time slot; L max Indicates the maximum number of computing task loads that vehicles can queue; Indicates vehicle VE iThe computational tasks generated in time slot t, L b Indicates the average data size of a computing task; Indicates vehicle VE i The size of the task actually processed in time slot t, is a Boolean variable. If vehicle VE i The task was successfully processed. otherwise

[0019] Furthermore, it also includes: determining the vehicle priority value according to the attributes of the vehicle and the length of the task queue; the attributes of the vehicle include ordinary vehicles and emergency vehicles.

[0020] Furthermore, the system cost is calculated as follows:

[0021]

[0022] in, and Respectively represent the total delay of completing the computing task after and before normalization, and They represent the total energy consumption of completing the computing task after and before normalization, λ1 and λ2 are weighted parameters, and T Local =L b c / f max is the execution delay of the total local maximum computing power, f max represents the maximum computing power of the vehicle, c represents the number of CPU cycles required to process a task, and E Local =κL b c(f max ) 2 is the total local maximum computing power consumption, κ is the effective capacitance coefficient, Z i Indicates vehicle VE i The priority value of Indicates vehicle VE i The task offloading ratio in time slot t, Represents a vehicle index collection, represents the time slot index set, N is the total number of vehicles, T represents the number of time slots, represents the computing power of the MEC server in time slot t, Indicates the maximum computing capacity of the MEC server, L b Indicates the average data size of a computing task, Indicates vehicle VE i The size of the task actually processed in time slot t.

[0023] Furthermore, in step 5, before the sample data in the experience buffer is input into the deep reinforcement learning network, it also includes:

[0024] The difference between the maximum value and the minimum value of each variable in the sample data is used as the scaling factor of each variable, and each variable is normalized using the scaling factor of each variable. The normalized sample data is used to train the deep reinforcement learning network.

[0025] Furthermore, in step 5, during the training process, the loss function used by the critic network is:

[0026]

[0027] in, Express expectations, represents the predicted Q value of the target network of the critic network, r t For the input state variable s t Next, perform action a t The reward obtained when , γ is the discount factor, Indicates execution of action a t After that, the state variable s is observed t+1 After executing action a t+1 The Q value obtained when θ is the parameter of the deep reinforcement learning network.

[0028] In a second aspect, the present invention provides a deep reinforcement learning vehicle edge computing unloading system based on temporal state perception, comprising:

[0029] The network construction module is used to embed the LSTM network in the actor network and the feedforward neural network of the critic network of the DDPG network to construct a deep reinforcement learning network;

[0030] A state space module, used to define the state space of the deep reinforcement learning network; wherein each state vector in the state space is used to indicate the task queue length, vehicle coordinate position, actual processed task size, vehicle priority value and remaining computing resources of a single vehicle in time slot t; correspondingly, the LSTM network is used to receive a real-time state vector and integrate the real-time state vector with a historical state vector;

[0031] An action space module, used to define the action space of the deep reinforcement learning network; wherein each action in the action space is used to indicate the task offloading ratio of the vehicle selected and the selected vehicle at time slot t;

[0032] A reward module, used to define a reward function of the deep reinforcement learning network as a function that is negatively correlated with a system cost; wherein the system cost includes a total delay and a total energy consumption spent on completing a computing task;

[0033] A training module, used to train the deep reinforcement learning network using sample data in the experience buffer to obtain a vehicle edge computing offloading model;

[0034] The unloading calculation module is used to obtain the real-time state vector input into the vehicle edge computing unloading model to obtain the real-time unloading decision and resource allocation results.

[0035] In a third aspect, the present invention provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the method described in the first aspect.

[0036] In a fourth aspect, the present invention provides a non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the method described in the first aspect.

[0037] Beneficial effects of the present invention:

[0038] (1) Considering the sensitivity of different vehicles to task delays, the present invention proposes a task offloading and resource allocation scheme in a VEC environment. According to vehicle attributes and task size, a vehicle priority division strategy is designed to optimize the vehicle scheduling sequence and task allocation, thereby achieving differentiated processing for different vehicles and tasks and solving the delay sensitivity problem faced in the vehicle edge environment.

[0039] (2) Since the unloading strategy depends on the dynamics of the environment, the present invention uses the Markov decision process (MDP) to describe the unloading problem. By analyzing dynamic environmental factors such as vehicle position, task queue load, and channel attenuation in real time, a deep deterministic policy gradient calculation unloading algorithm (DDPG-TSA) with time series state perception is designed. The present invention modifies the initial layer of the DDPG network to process time series data more efficiently. By analyzing the historical state and current state of the vehicle, the influence of the historical state on the current decision is fully utilized, and the time dependency between the historical state and the current state is captured. The perception of vehicle state changes is realized, the accuracy of action prediction is improved, the action is better predicted, and the system cost is reduced.

[0040] (3) In terms of numerical simulation, the present invention discusses the impact of hyperparameters on algorithm performance in order to select the optimal parameter combination. By comparing with other baseline algorithms, it is verified that the DDPG-TSA algorithm can effectively reduce system costs. The numerical simulation results verify that the method of the present invention is significantly better than other baseline methods in different scenarios, which shows that the method proposed in the present invention has higher efficiency and performance in solving the problem of computational offloading. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 A schematic diagram of a vehicle edge computing network environment provided by an embodiment of the present invention;

[0042] Figure 2 A principle block diagram of a deep reinforcement learning vehicle edge computing offloading method based on temporal state perception provided by an embodiment of the present invention;

[0043] Figure 3 The influence of different empirical buffer sizes provided in the embodiments of the present invention on the convergence performance of the DDPG-TSA algorithm of the present invention;

[0044] Figure 4 The influence of different learning rates provided in the embodiments of the present invention on the convergence performance of the DDPG-TSA algorithm of the present invention;

[0045] Figure 5 The influence of different discount factors provided in the embodiments of the present invention on the convergence performance of the DDPG-TSA algorithm of the present invention;

[0046] Figure 6 The impact of the total number of vehicles on the system cost under different algorithms provided in the embodiments of the present invention;

[0047] Figure 7 The influence of the distance between the MEC server and the vehicle on the system cost under different algorithms provided in the embodiments of the present invention;

[0048] Figure 8 The system cost of different algorithms provided by the embodiments of the present invention under different MEC server computing capabilities;

[0049] Fig. 9 Performance comparison results of different algorithms provided for the embodiments of the present invention: (a) Task discard rate under different task sizes; (b) Task discard rate under different MEC server computing capabilities;

[0050] Fig.10 A schematic diagram of the structure of a deep reinforcement learning vehicle edge computing unloading system based on temporal state perception provided by an embodiment of the present invention;

[0051] Fig.11 This is a structural block diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0052] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution in the embodiment of the present invention will be clearly described below in conjunction with the drawings in the embodiment of the present invention. Obviously, the described embodiment is a part of the embodiment of the present invention, not all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0053] In order to facilitate understanding of the technical solution of the present invention, the network environment or network architecture to which the technical solution of the present invention is applied is briefly introduced here. Figure 1 The vehicle edge computing network architecture shown in the figure mainly includes vehicles and base stations (BS). Among them, a fixed MEC server is equipped on the base station side to provide computing services specifically for vehicles in the coverage area. It can be understood that these vehicles have the necessary computing resources and can exchange and process data with the MEC server through wireless communication channels. In the present invention, represents a set of vehicles, each of which can choose to offload tasks to the MEC server or perform tasks locally. The entire system time is divided into time slots of equal length, i.e. represents a set of time slots, wherein each time slot lasts for Δ seconds. For ease of description, the symbols used in the present invention are summarized below, as shown in Table 1.

[0054] Table 1 Symbols and meanings

[0055]

[0056] like Figure 2 As shown, the embodiment of the present invention improves the traditional DDPG algorithm on the basis of the traditional DDPG algorithm to provide a deep reinforcement learning vehicle edge computing offloading method based on time series state perception, including the following steps:

[0057] S101: The LSTM network is embedded in the feedforward neural network of the actor network and the critic network of the DDPG network to construct a deep reinforcement learning network.

[0058] S102: define the state space of the deep reinforcement learning network; wherein each state vector in the state space is used to indicate the task queue length (i.e., the size of the task to be processed), the vehicle coordinate position, the actual processed task size, the vehicle priority value, and the remaining computing resources of the MEC server for a single vehicle in time slot t; correspondingly, the LSTM network is used to receive the real-time state vector and integrate the real-time state vector with the historical state vector;

[0059] S103: defining an action space of the deep reinforcement learning network; wherein each action in the action space is used to indicate a task offloading ratio of a vehicle selected and a selected vehicle at time slot t;

[0060] S104: defining a reward function of the deep reinforcement learning network as a function that is negatively correlated with a system cost; wherein the system cost includes a total delay and a total energy consumption spent on completing a computing task;

[0061] S105: Using the sample data in the experience buffer to train the deep reinforcement learning network to obtain a vehicle edge computing offloading model;

[0062] S106: Obtain a real-time state vector and input it into the vehicle edge computing unloading model to obtain a real-time unloading decision and resource allocation result.

[0063] Specifically, in the traditional DDPG network structure, each layer is a fully connected layer network, and only the current state data is considered, while the historical state data is ignored. This design may lead to a learning strategy that is not optimal. Since the LSTM network can process time series data more effectively, in order to improve learning efficiency, the present invention uses the LSTM network to modify the initial layer of the DDPG network, and uses the LSTM network to integrate the historical state and current state of the vehicle, making full use of the impact of the historical state on the current decision, capturing the time dependency between the historical state and the current state, and using the integrated data as the input of the actor network and the critic network to achieve perception of vehicle state changes, better process complex state space data, improve the accuracy of action prediction, and predict actions faster and better, reducing system costs.

[0064] In one embodiment, in the VEC network proposed in the embodiment of the present invention, the location of the MEC server is set to be fixed, using coordinates p = [p x ,p y ,H] T represents, H represents the height of the base station. Assume that the coordinates of the vehicle at time slot t are represented as After Δ seconds, the coordinate position q i (t+1) is expressed as:

[0065]

[0066] in, Indicates vehicle VE i At the coordinate position of time slot t, q i (t+1) represents vehicle VE i The coordinate position of the next time slot after time slot t, Δ represents the duration of a time slot, vi (t) represents vehicle VE i speed, β(t)∈[0,2π].

[0067] It should be noted that only when the vehicle's coordinate position is within the coverage of the MEC server can the vehicle offload the task to the MEC server for calculation. Therefore, the following constraints are met:

[0068]

[0069] in, Indicates that the MEC server coverage is within the area.

[0070] Based on the above embodiments, the computing task of the vehicle in time slot t is represented as in, Indicates the length of the task queue. Indicates the size of the task actually processed, c indicates the number of CPU cycles required to process a task, Indicates the maximum tolerable delay of the task, that is, whether the task is executed locally or offloaded to the edge server, the execution time cannot exceed Z i Indicates the priority value of the vehicle, Z i The larger the value, the higher the priority of the vehicle. The embodiment of the present invention assumes that the tasks are fine-grained and can be divided into subsets of any size. Based on this, the task queue length The task queue length can be associated with the remaining tasks of the previous time slot and the newly generated computing tasks. The update method is expressed as:

[0071]

[0072] in, and Respectively represent vehicle VE i The length of the task queue between time slot t and the next time slot; Indicates vehicle VE i The computing tasks generated in time slot t are modeled as independent and identically distributed Poisson processes with parameter ξ; L b Indicates the average data size of a computing task. max Indicates the maximum number of computing task loads that vehicles can queue. is a Boolean variable. If the vehicle VE i The task was successfully processed. otherwise Indicates that the task is discarded.

[0073] Based on the above embodiments, in the embodiment of the present invention, the vehicle priority is divided based on two key factors: the vehicle attribute and the task queue length. Specifically, the vehicle is divided into ve emer andve nor Two categories, emergency vehicles belong to ve emer , ordinary vehicles belong to ve nor , emergency vehicles are given higher priority. Secondly, tasks are divided into l large and l small There are two types, the task size range is [0,100]Mbits and belongs to l small , otherwise it belongs to l large If the attributes of the vehicles are the same, the smaller task will be executed first. If the larger task is processed first, it may cause a longer waiting time for other tasks, affecting the response speed of the task and the overall performance of the system. For specific divisions, refer to Table 2. In this embodiment, the vehicle priority value setting is divided into 3 types. The larger the value, the higher the corresponding priority.

[0074] Table 2 Vehicle priority

[0075]

[0076] Based on the above embodiments, the construction process of the system cost calculation model in this embodiment mainly includes four construction steps: local calculation model, edge calculation model, system model and cost normalization.

[0077] The specific process is as follows:

[0078] (1) Local computing model, that is, modeling the situation where the vehicle performs tasks locally. Assume that the local execution CPU frequency By applying dynamic voltage and frequency scaling techniques, the device can balance execution time and energy consumption by adjusting the CPU frequency in each cycle. for:

[0079]

[0080] Accordingly, the energy consumption of local processing is expressed as:

[0081]

[0082] Where κ is the effective capacitance coefficient; based on the actual chip architecture, Indicates the energy consumption of one CPU cycle.

[0083] (2) Edge computing model, that is, modeling the situation of MEC server executing tasks. Offload to the edge server for execution, the computation offloading process is divided into three parts: first, the vehicle transmits the task to the MEC server via a wireless link; then, the MEC server allocates computing resources to the offloaded task; finally, the MEC server transmits the computation result back to the corresponding vehicle. In the present invention, since the data size of the computation result is much smaller than the task size and the downlink transmission rate is high, the transmission delay and energy consumption of data sent from the MEC server to the vehicle are ignored.

[0084] The delay of the vehicle uplinking the task to the MEC server It is expressed as:

[0085]

[0086] in, Indicates vehicle VE i The task offloading ratio in time slot t, represents the uplink transmission rate of the vehicle, expressed as:

[0087]

[0088] Where B represents the system bandwidth, σ 2 represents the noise power, p loss represents the path loss power, Indicates signal blocking flag ( Indicates signal blocking, 0 indicates no blocking), P i represents the uplink transmission power of the vehicle, δ is the channel power gain when the reference distance is 1m, and p represents the location coordinates of the MEC server.

[0089] Accordingly, the uplink transmission energy consumption for:

[0090]

[0091] After receiving the offload task, the MEC server processes the offloaded task and executes the delay It is expressed as:

[0092]

[0093] Correspondingly, the energy consumption of MEC server It is expressed as:

[0094]

[0095] Among them, P MEC Indicates the power of the MEC server when it is running.

[0096] Based on the above calculation process, the total delay and energy consumption in edge computing mode are expressed as:

[0097]

[0098] (3) System model

[0099] The system cost considered in the present invention includes the maximum delay of task execution and the total energy consumption of local computing and edge computing. The maximum delay of task execution and the total energy consumption are calculated according to equations (4), (5), (11), and (12). It is expressed as:

[0100]

[0101] in, The task calculation delay must not be greater than the task's tolerance delay. The task will be discarded. Defined as:

[0102]

[0103] The total energy consumption is expressed as:

[0104]

[0105] (4) Cost normalization.

[0106] Next, we normalize the latency and energy consumption respectively. The normalized latency is expressed as:

[0107]

[0108] Among them, T Local =L b c / f max is the execution delay of the total local maximum computing power. Similarly, the normalized energy consumption is expressed as:

[0109]

[0110] Among them, E Local =κL b c(f max ) 2 is the total local maximum computing power consumption.

[0111] In summary, the embodiment of the present invention adopts the following method to calculate the system cost C, namely:

[0112]

[0113] st(16),(17)(18b)

[0114]

[0115] in, and Respectively represent the total delay of completing the computing task after and before normalization, and They represent the total energy consumption of completing the computing task after normalization and before normalization, respectively. λ1 and λ2 are weighted parameters, which weigh the delay and energy consumption. From formula (18d), we can see that the linear sum of the delay weight and the energy consumption weight is 1. Local =L b c / f max is the execution delay of the total local maximum computing power, f max represents the maximum computing power of the vehicle, c represents the number of CPU cycles required to process a task, and E Local =κL b c(f max ) 2 is the total local maximum computing power consumption, κ is the effective capacitance coefficient, Z i Indicates vehicle VE i The priority value of Indicates vehicle VE i The task offloading ratio in time slot t, Represents a vehicle index collection, represents the time slot index set, N is the total number of vehicles, T represents the number of time slots, represents the computing power of the MEC server in time slot t, Indicates the maximum computing capacity of the MEC server, L b Indicates the average data size of a computing task, Indicates vehicle VE i The size of the task actually processed in time slot t.

[0116] It is understandable that if the vehicle chooses to perform the task completely locally, the MEC server will not allocate computing resources for it, that is, According to the last constraint in the system cost (i.e., inequality (18f)), the computing resources allocated by MEC to the task cannot be greater than the maximum computing capacity of the MEC server.

[0117] The system cost is the objective function of the present invention. From the above content, it can be seen that the objective function and constraints are NP-hard problems. When the number of devices increases, the complexity of the problem increases sharply. Traditional optimization-based methods may not be able to adapt to dynamic characteristics and make intelligent decisions. Therefore, the embodiment of the present invention proposes a method based on deep reinforcement learning to solve this problem.

[0118] In the embodiment of the present invention, the training process of the deep reinforcement learning network is mainly approximated as a Markov decision process (MDP). Generally speaking, the MDP consists of four tuples: They represent state, action, state transition probability and reward respectively. At a certain time t, the agent observes the current state in an unknown environment. Choose and execute an acceptable action according to the policy π The policy π is usually stochastic and is defined as the probability distribution of the current state mapped to the action, i.e. Then, the agent receives an immediate reward r t =r t (s t ,a t ), according to the state transition probability p(s t+1 |s t ,a t ) Enter the next state s t+1 Therefore, the state transition of the environment depends on the agent's current state s t Action performed a t .

[0119] Specifically in the embodiment of the present invention, the state space includes various states of the MEC server and the vehicle. Therefore, the embodiment of the present invention describes the state vector of a certain time slot t as follows:

[0120]

[0121] in, Indicates vehicle VE i The task queue length is set to ensure that the number of tasks for the vehicle at any time does not exceed the task queue length. Indicates vehicle VE i The coordinate position of Indicates vehicle VE i The size of the task actually processed in time slot t. In addition, represents the remaining computing resources of the MEC server in time slot t, Z i Indicates vehicle VE i The priority value of the .

[0122] It is understandable that in practical applications, there are situations where the communication environment of the vehicle is poor. At this time, the vehicle and the MEC server may not be able to communicate. Therefore, a variable indicating the quality of the communication environment may be added to the above state vector. Specifically, this embodiment adopts To indicate the signal blocking sign, the state vector of a time slot t can be described as:

[0123] Specifically in the embodiment of the present invention, the action space is the unloading decision and resource allocation result in each time slot, including the vehicle selected for calculation unloading and the task unloading ratio corresponding to each vehicle. Therefore, the action vector of a certain time slot t can be expressed as follows:

[0124]

[0125] in, represents the index of the vehicle selected for computational unloading at time slot t, The task offloading ratio for the corresponding vehicle. Dynamically allocate the task offloading ratio according to the vehicle selection to optimize the utilization of network and computing resources. The agent selects the appropriate offloading ratio based on the current environment state. Offload the task to the MEC server, where Specifically, when When , the task is executed completely locally; when When , all computing tasks are offloaded to the MEC server for execution; when When data is offloaded to the edge server for processing, and the remaining The data is processed locally.

[0126] Specifically in the embodiment of the present invention, the behavior of the vehicle is driven by rewards, and this reward mechanism is usually closely related to the objective function. The objective function of the present invention is to minimize the system cost, which is contrary to the goal of deep reinforcement learning (DRL) to pursue the maximum long-term discounted reward. Therefore, the reward function designed in the embodiment of the present invention needs to be negatively correlated with the objective function to ensure that the cost of the entire vehicle edge computing (VEC) system is minimized. For example, directly setting the reward function to

[0127] r t =-C (21)

[0128] Where C represents the system cost. The agent’s reward value r t The constraints in formula (18) should be met to ensure the validity of the calculation results, and the reward is set to a negative value to reflect the situation where the constraints are not met.

[0129] Furthermore, combined with Figure 2 As shown, the DDPG-TSA proposed in the present invention is based on DDPG and uses two different DNNs to approximate the critic network and the actor network. At the same time, the critic network and the actor network each contain a target network with the same structure and initial parameters.

[0130] In one embodiment, during the training of a deep neural network (DNN), the input data distribution of each layer changes as the parameters of the previous layer change. This change may cause the input data to gradually approach the limit boundary of the nonlinear function, thereby slowing down the convergence of the training process. To overcome this challenge, an embodiment of the present invention proposes a state normalization algorithm, which optimizes the training efficiency of a deep neural network by preprocessing the state space. Specifically, the normalization algorithm includes: using the difference between the maximum and minimum values ​​of each variable as a scaling factor, and scaling the state s in various ranges. t Normalization is performed. This strategy can unify the distribution range of input data, effectively accelerate the convergence speed of the training process, and thus improve the overall training effect. The pseudo code of the normalization algorithm is shown in Table 3.

[0131] Table 3

[0132]

[0133] The above Algorithm 1 describes the steps of state normalization. Five scaling factors are used in the normalization algorithm, each of which is explained as follows: ξ1 is used to normalize the scaling factor of the task queue length, ξ2 is used to scale the vehicle's coordinates, ξ3 is used to scale the remaining task size of the vehicle, ξ4 normalizes the remaining computing resources of the MEC server, and ξ5 is used to normalize the priority of the vehicle.

[0134] On the basis of the above embodiments, in the reinforcement learning process in the embodiment of the present invention, in order to learn the optimal unloading strategy, the reward of a certain state is defined as the future long-term discounted reward, which is expressed as Among them, γ∈[0,1] represents the discount factor. Then, the action value function (also called Q value function) is introduced into the loss function to describe the t Next, take action a according to strategy π t Expected returns:

[0135]

[0136] in, Express expectations.

[0137] According to existing research, the optimal strategy π that maximizes the long-term discounted reward * Under this condition, the Q-value function satisfies the Bellman optimality equation:

[0138]

[0139] Optimal strategy π * It can be obtained by

[0140]

[0141] Next, the present invention attempts to iteratively update the Q-value function using a recursive method in each time slot, where the agent updates the Q-value function from the experience tuple (s t ,a t ,r t ,s t+1 ), the Q-value function satisfies the following iterative formula:

[0142]

[0143] Among them, α∈(0,1) represents the learning rate, To predict the Q value, is the current Q value, and the time difference method (TD) is used to measure the difference between the current Q value and the Q value of the next time step. When a suitable learning rate is selected, the Q value converges to the optimal

[0144] Specifically, the LSTM layer in the actor network is responsible for receiving the current state information s of the vehicle t , and compare it with the historical state information h t-1 Integrate the information t The input φ of the next layer of network is calculated through the forget gate of the LSTM network t , and finally save the integrated information h through the output gate t , for the next step of calculation. Then φ t Input into the next layer of network to speed up the learning of the optimal strategy. t The calculation formula is as follows:

[0145] φ t =f(h t ;ω)(26)

[0146] Among them, f is determined by LSTM, and ω is the network parameter of LSTM.

[0147] The actor network uses the tanh activation function to limit the network output to the range of [-1,1]. Finally, the output action is converted into an actual action in the continuous action space. The critic network has the same network structure as the actor network, and evaluates the action according to the current state-action pair, that is, the Q value. Therefore, the critic network loss function in the embodiment of the present invention is updated as follows:

[0148]

[0149] in,

[0150]

[0151] The policy gradient of the actor network is updated using the chain rule as follows:

[0152]

[0153] Based on the above content, in the embodiment of the present invention, the pseudo code of the training process of the deep reinforcement learning network is shown in Table 4. The specific details are as follows: First, the normalized state space is input into the LSTM network, and then input into the actor network after processing, and the action a is output. t In order to fully explore the state space, we need to balance the exploitation and exploration issues by adding noise n t Construct action space and obtain action a t =μ(s t )+n t Where n t Obey Gaussian noise distribution μ e and σ e,t Represent the mean and standard deviation respectively. The agent performs action a t After that, observe the next state s t+1 and instant rewards t , and then the experience group (s t ,a t ,r t ,s t+1 ) is stored in the experience buffer. During the algorithm training process, the agent randomly extracts K small batch samples (s t ,a t ,r t ,s t+1 ) is input into the actor network and the critic network. The critic network outputs the current Q value, and the actor target network outputs the action.

[0154] According to formula (28), the critic target network is based on the state s t+1 and action μ′(s t+1 ) Calculate the target value y t , in order to minimize the loss function, the critic network is updated using the Adam optimizer. Then the actor network μ(s|θ μ ) will be action a = μ (s t ) is input to the critic network to implement action a = μ(s t ) parameter Updated by the optimizer. Using these two gradients, the actor network is approximately updated according to the following formula:

[0155]

[0156] Finally, use τ to soft-update the actor target network and the critic target network:

[0157]

[0158] Table 4

[0159]

[0160] In order to verify the performance of the solution of the present invention in the vehicle edge environment, the relevant numerical simulation process is provided below. First, we explored the impact of different hyperparameters on the stability and performance of the algorithm to select the best parameter combination. Then, we compared it with other baseline methods to verify the superior performance of the DDPG-TSA algorithm proposed in this invention in solving the computational offloading problem in different scenarios.

[0161] (I) Simulation settings

[0162] In the VEC system, the base station with the MEC server is located in the center, the height of the base station is fixed at 100m, and its signal coverage is a circular area. Vehicles with computing tasks are randomly distributed in the range of In the method proposed by the present invention, the calculation frequencies of the MEC server and the vehicle are set to f MEC =5GHz and f max =1.2GHz. The vehicle's transmission power P i =0.3W, the operating power of the MEC server P MEC =0.5W. Noise power σ 2 =-100dBm

[35] , the channel power gain δ at a reference distance of 1m is -50dB

[35] . The CPU cycles required per bit is c = 1000 cycles / bit. We assume that the learning rates of the actor network and the critic network are the same, α = 1e-6. The experience buffer D size is 2000, the discount factor γ = 0.0001, and the mini-batch size is set to 32. The detailed simulation parameters are shown in Table 3.

[0163] Table 5 Simulation parameters

[0164]

[0165] For performance comparison, the present invention is compared with the following four baseline methods:

[0166] Offloading Only: In this method, the vehicle completely offloads all computing tasks to the MEC server, and the entire computing resources of the MEC server are allocated to all vehicles.

[0167] Locally Only: All vehicles perform their computing tasks completely locally, without offloading data to the MEC server.

[0168] DQN: The DQN algorithm based on discrete action space is compared and analyzed with the DDPG-TSA algorithm proposed in this invention.

[0169] DDPG: By comparing with the original DDPG algorithm, it is verified that the DDPG-TSA algorithm has better performance.

[0170] (II) Results Analysis

[0171] (1) Parameter analysis

[0172] During the experiment, the difference in parameter settings significantly affected the performance, convergence speed and stability of the algorithm. We will explore the impact of three key parameters on the performance of the algorithm: the experience replay buffer, the learning rate and the discount factor. This study aims to select the optimal parameter configuration to enable the algorithm to achieve its best performance level.

[0173] The capacity of the experience replay buffer has a significant impact on the convergence speed and stability of the algorithm. We set different buffer sizes to evaluate the convergence performance of the algorithm. Figure 3 As shown in the figure, when the buffer size is 2000, the algorithm converges quickly to the local optimal solution. However, as the number of iterations increases, the System Cost shows obvious local oscillations. This phenomenon is due to the limitation of the buffer size, which fails to obtain sufficient feature information and thus fails to learn the optimal strategy. When the buffer capacity is expanded to 200,000, the training cycle is extended due to the huge buffer, which hinders effective learning. When the buffer capacity is set to 20,000, the algorithm converges to the global optimum at 330 Episodes and shows high stability in subsequent iterations. Therefore, we set the experience replay buffer size to 20,000.

[0174] Figure 4The convergence and stability of the algorithm under different learning rates are demonstrated. When α=1e-4, although the system cost converges quickly to the local optimal solution, the system cost begins to gradually increase after 280 episodes and gradually converges, but the optimal solution is not as good as when α=1e-6. This is because during the neural network training process, the relatively large learning rate makes the actor network and the critic network update steps larger. When α=1e-5, the algorithm cannot converge. When α=1e-6, the system cost converges to the optimal solution in 340 episodes, and even with the increase in the number of iterations, the global optimal solution can still be obtained. The stability and convergence of the algorithm are satisfied. On the other hand, when α=1e-7, the algorithm cannot converge and the system cost cannot reach the optimal solution. This is because the lower learning rate causes the neural network to update more slowly, and more iterations are required to converge. Therefore, we determine 1e-6 as the optimal learning rate for network training.

[0175] exist Figure 5 In this paper, we evaluated the impact of different γ on the convergence performance of the algorithm. The discount factor is used to balance the importance of current rewards and future rewards. The experimental results show that when γ is 0.0001, the algorithm can converge to the global optimal solution and has the best performance. When the γ value is large, the agent is more inclined to future rewards and may make decisions that are beneficial in the long run but unfavorable in the short run. Therefore, choosing an appropriate γ value can improve the performance of the algorithm. In the following experiments, we set the γ value to 0.0001.

[0176] (2) Performance comparison

[0177] In this section, we compare the performance of the DDPG-TSA algorithm with the baseline algorithm in terms of the number of vehicles, the computing power of the MEC server, the distance between the base station and the vehicle, and the task abandonment rate.

[0178] Figure 6The impact of different numbers of vehicles on the system cost of each algorithm when the computing power of the MEC server is 5GHz is shown. It can be found that as the number of vehicles increases, the system cost increases. However, compared with the other four algorithms, the DDPG-TSA algorithm we proposed not only has a lower system cost, but also increases more slowly. When the number of vehicles is 10, although the system cost of Offloading Only does not exceed that of Locally Only, the trend shows that as the number of vehicles continues to increase, the system cost of Offloading Only will reach the highest. This is mainly because the MEC server needs to take on more tasks, resulting in tight computing and communication resources, thereby increasing latency and energy consumption. Compared with Offloading Only, Locally Only, DQN, and DDPG, our method reduces the system cost by approximately 48%, 51%, 16%, and 9%, respectively.

[0179] Figure 7 The effect of the distance between the MEC server and the vehicle on the total system cost is shown. Specifically, the increase in distance leads to a decrease in the wireless channel gain. According to formula (7), the data transmission rate of the vehicle will be affected accordingly, resulting in increased latency and energy consumption. For the Offloading Only method, the effect of increased distance is particularly obvious, and the system cost increases rapidly. When a certain distance is reached, the system cost exceeds Locally Only. The method of the present invention shows even better stability. Even when the distance between the MEC server and the vehicle increases, the system cost is always kept at the lowest level. This shows that our method can effectively adapt to changes in wireless channel conditions, ensure data transmission efficiency, and control system cost within a reasonable range.

[0180] Next, we compare the system costs of different methods by adjusting the computing power of MEC servers. Figure 8 As shown in the figure, with the enhancement of the computing power of the MEC server, the system costs of the other four methods, except the Locally Only method, are significantly reduced. The difference between Offloading Only and DQN and DDPG is getting smaller and smaller. This is because the MEC server has sufficient computing resources to allocate to the vehicle, thereby reducing the system cost. The method proposed in the present invention can make the optimal offloading decision based on the computing resources of the MEC server, effectively reducing the system cost. Since Locally Only does not offload the task to the MEC server for execution, its system cost remains basically unchanged. Compared with the Offloading Only, Locally Only, DQN and DDPG methods, DDPG-TSA can still reduce costs by approximately 44%, 61%, 21% and 19%, respectively.

[0181] Fig. 9 The changes in task abandonment rate are shown from two dimensions: task size and MEC server computing power. Fig. 9 In (a), we observe that as the task size increases, the task discard rate of all algorithms increases. Among them, the discard rate of the Offloading Only and Locally Only algorithms increases more significantly, which indicates that neither local computing resources nor edge server resources are sufficient to handle all computing tasks under latency constraints. The DQN and DDPG algorithms have slowed down the rising trend of the task discard rate to a certain extent by adjusting the offloading strategy, but compared with the DDPG-TSA algorithm, the discard rate curve increases more significantly. Fig. 9 (b) shows the impact of changes in the computing power of the MEC server on the task discard rate. As the computing power of the MEC server increases, the task discard rate of the Offloading Only strategy shows large fluctuations, which shows that it is extremely sensitive to changes in computing resources. In contrast, the task discard rate of the Locally Only strategy has hardly changed and remains at a certain level. The other three DRL-based strategies remain at a low level, especially when the MEC server has sufficient computing resources, the task discard rate of Offloading Only is comparable to that of the DRL method. The DDPG-TSA algorithm we proposed always maintains the lowest task discard rate in various scenarios and selects the best offloading strategy during task offloading.

[0182] The present invention aims to study the problem of computational offloading in the edge environment of vehicles, especially for the situation where mobile vehicle tasks arrive randomly in continuous time. In this case, it is necessary to make offloading decisions and allocate computing resources quickly and accurately. In order to solve this problem, the present invention proposes an online dynamic computational offloading framework based on the DDPG algorithm, and introduces an LSTM network to combine historical state information and current state information to improve the accuracy of the algorithm's prediction action. At the same time, considering that different vehicles have different sensitivities to task delays in actual scenarios, we prioritize vehicles according to vehicle attributes and task sizes. In order to improve the convergence and stability of the algorithm, we normalize the state space. Subsequently, simulation experiments are performed to evaluate the impact of different hyperparameters on the performance of the algorithm to select the optimal parameter combination. Finally, comparisons with baseline methods are made from multiple aspects to verify that our method has obvious advantages in reducing system costs.

[0183] Fig.10 A deep reinforcement learning vehicle edge computing unloading system based on temporal state perception is provided in an embodiment of the present invention, comprising a network construction module, a state space module, an action space module, a reward module, a training module and an unloading computing module.

[0184] Among them, the network construction module is used to embed the LSTM network in the feedforward neural network of the actor network and the critic network of the DDPG network to construct a deep reinforcement learning network; the state space module is used to define the state space of the deep reinforcement learning network; wherein each state vector in the state space is used to indicate the task queue length, vehicle coordinate position, actual processed task size, vehicle priority value and remaining computing resources of a single vehicle in time slot t; correspondingly, the LSTM network is used to receive the real-time state vector and integrate the real-time state vector with the historical state vector; the action space module is used to define the action space of the deep reinforcement learning network; wherein each action in the action space is used to indicate the task offloading ratio of the vehicle selected and the selected vehicle in time slot t; the reward module is used to define the reward function of the deep reinforcement learning network as a function that is negatively correlated with the system cost; wherein the system cost includes the total delay and total energy consumption spent on completing the computing task; the training module is used to train the deep reinforcement learning network using the sample data in the experience buffer to obtain a vehicle edge computing offloading model; the offloading calculation module is used to obtain the real-time state vector input to the vehicle edge computing offloading model to obtain real-time offloading decision and resource allocation results.

[0185] It should be noted that the computing offloading system provided in the embodiment of the present invention is to implement the above method. Its specific functions can be referred to the above method embodiments, which will not be repeated here.

[0186] Fig.11 An example of a physical structure diagram of an electronic device is shown in FIG. Fig.11As shown, the electronic device may include: a processor (processor) 1101, a communication interface (Communications Interface) 1102, a memory (memory) 1103 and a communication bus 1104, wherein the processor 1101, the communication interface 1102, and the memory 1103 communicate with each other through the communication bus 1104. The processor 1101 can call the logic instructions in the memory 1103 to execute the vehicle edge computing method, which includes: nesting the LSTM network in the feedforward neural network of the actor network and the critic network of the DDPG network to construct a deep reinforcement learning network; defining the state space of the deep reinforcement learning network; wherein each state vector in the state space is used to indicate the task queue length, vehicle coordinate position, actual processed task size, vehicle priority value and remaining computing resources of a single vehicle in time slot t; correspondingly, the LSTM network is used to receive the real-time state vector and integrate the real-time state vector with the historical state vector; defining the action space of the deep reinforcement learning network; wherein each action in the action space is used to indicate the task offloading ratio of the vehicle selected and the selected vehicle in time slot t; defining the reward function of the deep reinforcement learning network as a function that is negatively correlated with the system cost; wherein the system cost includes the total delay and total energy consumption spent to complete the computing task; using the sample data in the experience buffer to train the deep reinforcement learning network to obtain a vehicle edge computing offloading model; obtaining the real-time state vector and inputting it into the vehicle edge computing offloading model to obtain a real-time offloading decision and resource allocation result.

[0187] In addition, when the logic instructions in the above-mentioned memory 1103 are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0188] An embodiment of the present invention further provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute the methods provided by the above-mentioned method embodiments, for example, including: nesting the LSTM network in the actor network and the feedforward neural network of the critic network of the DDPG network to construct a deep reinforcement learning network; defining the state space of the deep reinforcement learning network; wherein each state vector in the state space is used to indicate the task queue length, vehicle coordinate position, actual processed task size, vehicle priority value, and MEC server remaining value of a single vehicle in time slot t. computing resources; correspondingly, the LSTM network is used to receive a real-time state vector and integrate the real-time state vector with a historical state vector; define the action space of the deep reinforcement learning network; wherein each action in the action space is used to indicate the task offloading ratio of the vehicle selected and the selected vehicle in time slot t; define the reward function of the deep reinforcement learning network as a function that is negatively correlated with the system cost; wherein the system cost includes the total delay and total energy consumption spent on completing the computing task; use the sample data in the experience buffer to train the deep reinforcement learning network to obtain a vehicle edge computing offloading model; obtain the real-time state vector and input it into the vehicle edge computing offloading model to obtain a real-time offloading decision and resource allocation result.

[0189] The embodiment of the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the methods provided by the above-mentioned method embodiments are implemented, for example, including: embedding the LSTM network in the feedforward neural network of the actor network and the critic network of the DDPG network to construct a deep reinforcement learning network; defining the state space of the deep reinforcement learning network; wherein each state vector in the state space is used to indicate the task queue length, vehicle coordinate position, actual processed task size, vehicle priority value and remaining computing resources of the MEC server of a single vehicle in time slot t; correspondingly, the LSTM network is used The invention relates to receiving a real-time state vector and integrating the real-time state vector with a historical state vector; defining an action space of the deep reinforcement learning network; wherein each action in the action space is used to indicate the task offloading ratio of the vehicle selected and the selected vehicle in time slot t; defining a reward function of the deep reinforcement learning network as a function negatively correlated with the system cost; wherein the system cost includes the total delay and total energy consumption spent on completing the computing task; training the deep reinforcement learning network using sample data in an experience buffer to obtain a vehicle edge computing offloading model; obtaining a real-time state vector and inputting it into the vehicle edge computing offloading model to obtain a real-time offloading decision and resource allocation result.

[0190] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0191] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A deep reinforcement learning vehicle edge computing offloading method based on temporal state perception, characterized in that: include: Step 1: Nest the LSTM network into the feedforward neural network of the actor network and critic network of the DDPG network to construct a deep reinforcement learning network; Step 2: define the state space of the deep reinforcement learning network; wherein each state vector in the state space is used to indicate the task queue length, vehicle coordinate position, actual processed task size, vehicle priority value and remaining computing resources of a single vehicle at time slot t; correspondingly, the LSTM network is used to receive the real-time state vector and integrate the real-time state vector with the historical state vector; Step 3: defining an action space of the deep reinforcement learning network; wherein each action in the action space is used to indicate the task offloading ratio of the vehicle selected and the selected vehicle at time slot t; Step 4: defining the reward function of the deep reinforcement learning network as a function that is negatively correlated with the system cost; wherein the system cost includes the total delay and total energy consumption spent on completing the computing task; Step 5: Use the sample data in the experience buffer to train the deep reinforcement learning network to obtain a vehicle edge computing offloading model; Step 6: Obtain the real-time state vector and input it into the vehicle edge computing unloading model to obtain the real-time unloading decision and resource allocation results.

2. The deep reinforcement learning vehicle edge computing unloading method based on temporal state perception according to claim 1 is characterized in that: The vehicle coordinate position is updated in the following manner: in, Indicates vehicle VE i At the coordinate position of time slot t, q i (t+1) represents vehicle VE i The coordinate position of the next time slot after time slot t, Δ represents the duration of a time slot, v i (t) represents vehicle VE i speed, β(t)∈[0,2π].

3. The deep reinforcement learning vehicle edge computing unloading method based on temporal state perception according to claim 1 is characterized in that: The task queue length is updated as follows: in, and Respectively represent vehicle VE i The length of the task queue between time slot t and the next time slot; L max Indicates the maximum number of computing task loads that vehicles can queue; Indicates vehicle VE i The computational tasks generated in time slot t, L b Indicates the average data size of a computing task; Indicates vehicle VE i The size of the task actually processed in time slot t, is a Boolean variable. If vehicle VE i The task was successfully processed. otherwise 4. The deep reinforcement learning vehicle edge computing unloading method based on temporal state perception according to claim 1 is characterized in that: Also includes: The vehicle priority value is determined according to the attributes of the vehicle and the length of the task queue; the attributes of the vehicle include ordinary vehicles and emergency vehicles.

5. The deep reinforcement learning vehicle edge computing unloading method based on temporal state perception according to any one of claims 1 to 3 is characterized in that: The system cost is calculated as follows: in, and Respectively represent the total delay of completing the computing task after and before normalization, and They represent the total energy consumption of completing the computing task after and before normalization, λ1 and λ2 are weighted parameters, and T Local =L b c / f max is the execution delay of the total local maximum computing power, f max represents the maximum computing power of the vehicle, c represents the number of CPU cycles required to process a task, and E Local =κL b c(f max ) 2 is the total local maximum computing power consumption, κ is the effective capacitance coefficient, Z i Indicates vehicle VE i The priority value of Indicates vehicle VE i The task offloading ratio in time slot t, represents the vehicle index set, T represents the time slot index set, N is the total number of vehicles, T represents the number of time slots, represents the computing power of the MEC server in time slot t, Indicates the maximum computing capacity of the MEC server, L b Indicates the average data size of a computing task, Indicates vehicle VE i The size of the task actually processed in time slot t.

6. The deep reinforcement learning vehicle edge computing unloading method based on temporal state perception according to claim 1 is characterized in that: In step 5, before the sample data in the experience buffer is input into the deep reinforcement learning network, it also includes: The difference between the maximum value and the minimum value of each variable in the sample data is used as the scaling factor of each variable, and each variable is normalized using the scaling factor of each variable. The normalized sample data is used to train the deep reinforcement learning network.

7. The deep reinforcement learning vehicle edge computing unloading method based on temporal state perception according to claim 1 is characterized in that: In step 5, during the training process, the loss function used by the critic network is: in, Express expectations, represents the predicted Q value of the target network of the critic network, r t For the input state variable s t Next, perform action a t The reward obtained when , γ is the discount factor, Indicates execution of action a t After that, the state variable s is observed t+1 After executing action a t+1 The Q value obtained when θ is the parameter of the deep reinforcement learning network.

8. A deep reinforcement learning vehicle edge computing unloading system based on temporal state perception, characterized in that: include: The network construction module is used to embed the LSTM network in the actor network and the feedforward neural network of the critic network of the DDPG network to construct a deep reinforcement learning network; A state space module, used to define the state space of the deep reinforcement learning network; wherein each state vector in the state space is used to indicate the task queue length, vehicle coordinate position, actual processed task size, vehicle priority value and remaining computing resources of a single vehicle in time slot t; correspondingly, the LSTM network is used to receive a real-time state vector and integrate the real-time state vector with a historical state vector; An action space module, used to define the action space of the deep reinforcement learning network; wherein each action in the action space is used to indicate the task offloading ratio of the vehicle selected and the selected vehicle at time slot t; A reward module, used to define a reward function of the deep reinforcement learning network as a function that is negatively correlated with a system cost; wherein the system cost includes a total delay and a total energy consumption spent on completing a computing task; A training module, used to train the deep reinforcement learning network using sample data in the experience buffer to obtain a vehicle edge computing offloading model; The unloading calculation module is used to obtain the real-time state vector input into the vehicle edge computing unloading model to obtain the real-time unloading decision and resource allocation results.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the method according to any one of claims 1 to 7 is implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images