Internet of vehicles heterogeneous hierarchical task unloading and resource allocation optimization method based on MADDPG

Abstract

The invention discloses an Internet of Vehicles heterogeneous hierarchical task unloading and resource allocation optimization method based on MADDPG, and the method comprises the steps: carrying out the modeling of an intelligent Internet of Vehicles system model, the intelligent Internet of Vehicles system model comprises a plurality of vehicles and a plurality of edge servers, and each edge server is provided with a base station and a road side unit, each vehicle is connected with the base station and the road side unit through a V2I link, and each V2I link is allocated with an orthogonal frequency sub-band; building an optimization problem of low-delay task unloading and resource allocation based on an intelligent car networking system model; the optimization problem is converted into a Markov decision process model; and solving the Markov decision process model by adopting an MADDPG algorithm, outputting the proportion of unloading the task to the edge server and calculating the resource allocation proportion, and realizing heterogeneous and hierarchical task unloading and resource allocation optimization of the Internet of Vehicles. According to the invention, the average time delay of each task can be reduced, and the overall resource utilization rate of the system is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of intelligent vehicle networking, and particularly relates to a heterogeneous layered task offloading and resource allocation optimization method based on MADDPG for vehicle networking. BACKGROUND

[0002] With the rapid development of the Internet of Things, more Internet of Things devices are integrated into vehicle systems. At the same time, the intelligentization of vehicle networking devices and the rise of emerging businesses such as autonomous driving have led to an exponential increase in the amount of data generated by vehicles, placing higher demands on computing resources. Traditional cloud computing, which is centered on large server clusters, can provide sufficient computing power support, but its physical location is usually far from the actual deployment area of vehicles, making it difficult to meet the stringent requirements of low latency and high reliability of smart Internet of Vehicles (SIoV).

[0003] To address the above challenges, Mobile Edge Computing (MEC) has emerged as a new computing paradigm. By deploying computing resources on edge nodes close to vehicle networking users, edge computing can support vehicles to offload part of the computing tasks to local edge servers, thereby effectively reducing the communication latency caused by long-distance data transmission. However, the computing resources of edge servers are still limited compared to the cloud, and it is difficult to support the growing demand for computing-intensive and delay-sensitive tasks in SIoV networks. This contradiction between resources and demand can lead to system congestion, and in severe cases, even cause traffic paralysis or safety accidents. Therefore, how to design an efficient task offloading mechanism and resource allocation strategy to optimize the use of edge resources and ensure the real-time and stability of the system has become an important problem to be solved. SUMMARY

[0004] The present application aims to at least partially solve the technical problems in the related art. To this end, the first object of the present application is to provide a heterogeneous layered task offloading and resource allocation optimization method based on MADDPG for vehicle networking, which can reduce the average latency of each task and improve the overall resource utilization of the system.

[0005] The second object of the present application is to provide a computer readable storage medium.

[0006] To achieve the above-mentioned objects, the present application realizes the following technical solutions:

[0007] A heterogeneous layered task offloading and resource allocation optimization method based on MADDPG for vehicle networking, comprising:

[0008] Modeling an intelligent vehicle networking system model, the intelligent vehicle networking system model comprising a plurality of vehicles and a plurality of edge servers, each edge server being configured with a base station and a road side unit, each vehicle connecting the base station and the road side unit through a V2I link, wherein each V2I link is assigned an orthogonal frequency subband;

[0009] Based on the intelligent vehicle networking system model, an optimization problem of low-latency task offloading and resource allocation is constructed, wherein the task is a main task generated by the vehicle and a subtask having a dependency relationship with the main task;

[0010] The optimization problem is converted into a Markov decision process model;

[0011] The MADDPG algorithm is used to solve the Markov decision process model, and the proportion of task offloading to the edge server and the proportion of computing resource allocation are output, so as to realize the optimization of vehicle networking heterogeneous layered task offloading and resource allocation.

[0012] In a possible implementation, constructing the optimization problem comprises:

[0013] Based on the channel gain between the vehicle and the base station, the communication rate between the vehicle and the base station is calculated;

[0014] According to the communication rate, the task upload delay is determined, and according to the vehicle local processing delay, the task upload delay and the edge computing delay, the total task delay is determined;

[0015] A task dependency matrix is determined, and the task completion time is determined according to the task dependency matrix and the total task delay;

[0016] A task priority vector is determined, and the task completion time weight is determined according to the task priority vector;

[0017] The weighted sum of the task completion time is calculated based on the task completion time weight, and the objective function of the optimization problem is constructed with the goal of minimizing the sum of the weighted completion times of all tasks.

[0018] In a possible implementation, the constraint conditions of the optimization problem comprise first to fourth constraint conditions; wherein,

[0019] The first constraint condition is that the subtask in the subtask is completed after the parent task;

[0020] The second constraint condition is that the main task completion time does not exceed the deadline of the main task;

[0021] The third constraint condition is that the transmission power of the vehicle does not exceed the maximum transmission power of the vehicle;

[0022] The fourth constraint condition is that if the priority of the first main task is higher than the second main task, then any subtask belonging to the first main task must be started no later than any subtask belonging to the second main task.

[0023] In a possible implementation, determining the task dependency matrix comprises:

[0024] The task hypergraph is constructed, and based on the task hypergraph, it is determined whether the task belongs to the parent task set or the child task set in the hyperedge, so as to determine the task dependency matrix.

[0025] In a possible implementation, determining the task completion time according to the task dependency matrix and the total task delay comprises:

[0026] According to the task dependency matrix, the time required for completing a previous historical task and the maximum completion time of the parent task in the parent task set before executing the current task are determined when the task is calculated on the same device, and the larger one of the two is obtained, and the task total delay is summed to obtain the task completion time.

[0027] In a possible implementation, determining the task priority vector comprises:

[0028] Determining factors for determining the task priority, the factors including a task data size, a required CPU cycle number and a deadline;

[0029] Determining a task weight corresponding to each task for each factor, so as to construct a task weight matrix;

[0030] Importance comparison is performed between the factors two by two, a factor importance score between the factors is obtained, and a task priority analysis hierarchical matrix is constructed according to the factor importance score;

[0031] According to the factor importance score, a factor weight feature vector of the task priority analysis hierarchical matrix is determined;

[0032] The task weight matrix and the factor weight feature vector are subjected to matrix multiplication operation to obtain the task priority vector.

[0033] In a possible implementation, the Markov decision process model comprises a state space, an action space and a reward function; wherein,

[0034] The state space comprises computing resources available to the vehicle itself, computing resources available to the edge server and vehicle task characteristics, the vehicle task characteristics including a task data volume, a deadline and a task dependency relationship;

[0035] The action space comprises a task offloading ratio to the edge server and a computing resource allocation ratio;

[0036] The reward function is composed of a time delay reward, a resource utilization reward, and a time delay violation penalty term.

[0037] The optimization objective of the Markov decision process model is to minimize the overall task time delay and maximize the resource utilization of the vehicle or the edge server.

[0038] In a possible implementation, solving the Markov decision process model by using the MADDPG algorithm includes:

[0039] Each vehicle is modeled as an agent, and each agent is trained by using the MADDPG algorithm, and each agent is equipped with an Actor network and a Critic network.

[0040] Each agent outputs an action policy according to the state space information of the local observation input by using the Actor network.

[0041] Each agent further evaluates a value function of a state-action pair by using the Critic network, so that the Actor network adjusts the action policy according to the value evaluation.

[0042] In a possible implementation, when the Actor network outputs the action policy, a deterministic policy with exploration noise is used for action selection.

[0043] To achieve the above object, the second aspect of the present application provides a computer readable storage medium having a computer program stored thereon, wherein the computer program is executed by a processor to implement the method described above.

[0044] The present application has at least the following technical effects:

[0045] (1) The present application proposes a task priority-based classification method, which can cope with the differentiated needs of different vehicle tasks in the SIoV system in terms of time delay sensitivity. The method can determine the priority and optimize the scheduling according to the task characteristics, thereby ensuring the timely processing of high-priority tasks.

[0046] (2) The present application establishes a hypergraph-based task dependency model to address the problem of complex modeling of task hierarchical dependency relationships in the system. Compared with the traditional directed acyclic graph modeling method, this method effectively reduces the modeling complexity and improves the flexibility and scalability of task dependency representation.

[0047] (3) To achieve low time delay and efficient resource utilization of the SIoV system, the present application uses a multi-agent reinforcement learning architecture with centralized training and distributed execution, and proposes an improved MADDPG algorithm. Experimental results show that this method can effectively reduce the average task time delay and improve the system resource utilization.

[0048] Additional aspects and advantages of the application will be set forth in part in the description which follows, and in part will become apparent to those skilled in the art upon examination of the following and the attendant drawings or can be learned by practice of the application. BRIEF DESCRIPTION OF DRAWINGS

[0049] Figure 1 is a flow chart of the MADDPG-based task offloading and resource allocation optimization method for the Internet of Vehicles of the embodiment of the application.

[0050] Figure 2 is a schematic diagram of the intelligent Internet of Vehicles system model of the embodiment of the application.

[0051] Figure 3 is a hypergraph model schematic diagram based on task dependency of the embodiment of the application.

[0052] Figure 4 is a task scheduling diagram of the embodiment of the application.

[0053] Figure 5 is a MADDPG algorithm framework diagram based on task relationship of the embodiment of the application.

[0054] Figure 6 is a convergence comparison diagram under different learning rates of the embodiment of the application.

[0055] Figure 7 is a task average delay comparison diagram under different vehicle numbers of the embodiment of the application.

[0056] Figure 8 is a resource utilization rate comparison diagram under different vehicle numbers of the embodiment of the application. DETAILED DESCRIPTION

[0057] The embodiments are described in detail below with reference to the accompanying drawings, examples of which are shown in the drawings, wherein the same or similar notations represent the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to explain the present application, and cannot be understood as limiting the present application.

[0058] The MADDPG-based task offloading and resource allocation optimization method for the Internet of Vehicles of the embodiment is described below with reference to the accompanying drawings.

[0059] Figure 1 is a flow chart of the MADDPG-based task offloading and resource allocation optimization method for the Internet of Vehicles of the embodiment of the application. As shown in Figure 1 , the method comprises:

[0060] Step S101: modeling an intelligent vehicle networking system model, the intelligent vehicle networking system model comprising a plurality of vehicles and a plurality of edge servers, each edge server being configured with a base station and a road side unit, each vehicle being connected to the base station and the road side unit through a V2I link, wherein each V2I link is assigned an orthogonal frequency subband.

[0061] The intelligent vehicle networking system model is as shown in Figure 2 As shown in the figure, there are a plurality of ordinary vehicles, a plurality of unmanned vehicles, a plurality of edge servers in the system, each edge server being equipped with a single base station and a road side unit (RSU), the base station providing spectrum resources to a communication device, i.e., a vehicle, through the edge server to realize data transmission, i.e., vehicle to infrastructure (V2I) communication. The RSU can collect terminal user information in real time, including latency requirements, vehicle speed, and importance levels of tasks generated by vehicles, and upload these information to the edge server. In each time slot, each vehicle will generate U main tasks, and the edge server makes intelligent decisions on the task execution location, determines whether the task should be calculated locally or offloaded to the edge side for processing, to optimize the overall performance of the system. A main task is composed of a plurality of subtasks with dependency relationship, and the main task offloaded to the edge server is executed in different priority order according to the importance weight given by the RSU. Within the main task, the subtasks are divided into parent tasks and child tasks according to the functional logic, wherein the parent tasks need to be strictly executed in priority, and the child tasks can only start after the parent tasks they depend on are completed. Only when all the subtasks of the current main task are completed, the system will start the next main task, thereby ensuring the timing and dependency of task scheduling. Finally, the edge server returns the completed tasks to the vehicle user.

[0062] It should be noted that in this intelligent vehicle networking system model, orthogonal frequency division multiple access can be applied to N V2I links, i.e., N V2I links are pre-assigned to orthogonal frequency subbands, wherein the N V2I links occupy N subbands. Each subband can be independently assigned to different users or links, thereby avoiding interference between different users.

[0063] Further, for how to realize low-latency task offloading and resource allocation optimization, the following methods can be used to construct the related optimization problem.

[0064] Step S102: constructing an optimization problem of low-latency task offloading and resource allocation based on the intelligent vehicle networking system model.

[0065] In an embodiment of the present application, the construction of the optimization problem comprises: calculating the communication rate between the vehicle and the base station based on the channel gain between the vehicle and the base station; determining the task upload delay according to the communication rate, determining the total task delay according to the vehicle local processing delay, the task upload delay and the edge computing delay; determining the task dependency matrix, determining the task completion time according to the task dependency matrix and the total task delay; determining the task priority vector, determining the task completion time weight according to the task priority vector; calculating the weighted sum of the task completion time based on the task completion time weight, and constructing the objective function of the optimization problem with the minimum sum of the weighted completion time of all tasks as the target.

[0066] Specifically, the communication rate between the nth vehicle and the base station can be expressed as:

[0067]

[0068] wherein, is the channel bandwidth of the uplink transmission, is the transmission power of the nth vehicle, is the noise power, is the channel gain between the nth vehicle and the base station at the tth time slot, which is expressed as:

[0069]

[0070] wherein, is the small-scale path loss, is the large-scale path loss, and the calculation formula of the distance path loss model is α+βlog 10 (d), α=42.6 is the fixed loss, β=26 is the path loss exponent, and d is the distance between the nth vehicle and the base station. General fading is adopted, that is:

[0071]

[0072] wherein, obeys the standard complex Gaussian distribution, is the correlation coefficient, is the small-scale path loss at the t-1 time slot, is the position of the nth vehicle in the tth time slot, and p is the position of the base station, and η is the path loss exponent.

[0073] Further, let denote the task set generated by the nth vehicle, denote the Kth task generated by the nth vehicle, and for each task, it is expressed as wherein, is the size of the i-th task generated by the n-th vehicle, is the computational complexity of the i-th task, is the latency constraint of the i-th task. For the computational processing of the i-th task of the n-th vehicle, when , it indicates that the task is processed locally; when , it indicates that the task is partially offloaded; when , it indicates that the task is completely offloaded to the edge server, wherein represents the task execution location parameter. The vehicle local processing delay can be represented as:

[0074]

[0075] wherein represents the vehicle local processing delay, F n is the computing resource of the n-th vehicle itself, if the i-th task of the n-th vehicle exists partial or complete offloading, it is necessary to first transmit the offloaded part to the base station equipped with the edge server through wireless communication. The cloud system estimates and allocates the computing resource required by the task, and returns the computing result to the vehicle. Since the size of the computing result is much smaller than that of the uploaded part, and the downlink communication rate is high, the delay caused by returning the vehicle with the computing output result is not considered. Therefore, the delay generated by completing the task can be represented as:

[0076]

[0077] wherein is the data volume of the task offloading, is the task uploading delay, is the edge computing delay, is the computing resource allocated by the edge server for the i-th task of the n-th vehicle, and the total task delay can be represented as:

[0078]

[0079] In one possible implementation, determining the task dependency matrix comprises: constructing a task hypergraph, determining whether the task belongs to the parent task set or the child task set in the hyperedge based on the task hypergraph, to determine the task dependency matrix.

[0080] DAG (Directed Acyclic Graph) has been widely used to describe the precedence constraints between parent-child tasks. However, in intelligent vehicle networking, DAG relies on bidirectional edges to depict one-to-one task dependencies, which leads to an explosive growth in the size of the edge set and significantly increases the complexity of topology storage and scheduling algorithms as the number of vehicles and concurrent tasks increases exponentially. To overcome this bottleneck, the present embodiment proposes a hypergraph modeling framework. Unlike DAG, a hyperedge in a hypergraph can contain multiple points, i.e., task nodes. Each hyperedge is regarded as a many-to-many dependency set between a group of tasks, thereby enabling the compressed expression of complex dependency structures with a linear level of edge numbers, which can effectively reduce edge redundancy and simplify calculations.

[0081] A hypergraph is defined as The set of points in the hypergraph can be represented as v i represents the i-th task node in the hypergraph, and the hyperedge set can be represented as where e j represents a hyperedge, is the set of parent tasks in the task, is the set of child tasks in the task, and the points in the same hyperedge represent a task dependency relationship between them. A task dependency matrix H is constructed to depict this dependency relationship, where the element H[i,j] in the task dependency matrix H is represented as:

[0082]

[0083] In essence, a hyperedge is an unordered set representing parallel and concurrent dependencies, while the chain dependency in DAG is an ordered structure. Simplifying the chain dependency into a hyperedge will lose the order information, so it cannot be simply expressed by connecting all tasks with a set. For example, Figure 3 v1,v2 are parent tasks and v3 is a child task in hyperedge e1. At the same time, v3 is a parent task of v4,v5,v6 in hyperedge e3, and v6 is a parent task of v7,v8 in hyperedge e2. Thus, the task dependency matrix can be constructed according to the task dependency relationship in Figure 3

[0084] where if the i-th task is a child task of hyperedge e j = (P j ,i), then the start time T i s needs to satisfy:

[0085]

[0086] where v k ​denotes the kth task node, the time of completion of the parent task in the parent task set.

[0087] In a possible implementation, determining the task completion time according to the task dependency matrix and the total task delay includes: determining, according to the task dependency matrix, the time required for completion of a previous historical task when the task is calculated on the same device, and the maximum completion time of the parent task in the parent task set required for completion before execution of the current task, and obtaining the larger one of the two, summing the total task delay to obtain the task completion time.

[0088] According to the task dependency matrix, the dependency relationship between tasks can be determined. At the same time, whether the tasks are executed sequentially or in parallel, multiple sub-tasks in a main task all depend on successful completion of all previous tasks, as shown in Figure 4 Therefore, the joint scheduling delay of the main task cannot be simply obtained by adding its transmission time and calculation time. In order to ensure accurate completion of a single task, the task must start the sub-task after completion of the parent task in its corresponding sub-task, and the start time of a given scheduled task depends on the maximum completion time of its parent task, thereby causing dependency. Therefore, the task completion time T i cu can be defined as

[0089]

[0090] wherein P(j) is specifically a set of parent tasks in the task, and here specifically represents a parent task set required for completion before execution of the current task, is the task completion time of the jth task, denotes the maximum completion time of the parent task in the parent task set required for completion before execution of the current task, and T i S' denotes the time required for completion of a previous historical task when the task is calculated on the same device, that is, the maximum value of the completion time in the historical task allocated to the same resource (the same resource as i), which can be expressed as:

[0091]

[0092] wherein, is the time of completion of the k1th task, and z i , z k1 respectively represent the places where the ith task and the k1th task are offloaded.

[0093] In a possible implementation, the determining of the task priority vector comprises: determining factors for determining the task priority, the factors comprising a task data size, a required CPU cycle number, and a deadline; determining a task weight of each factor corresponding to each task to construct a task weight matrix; performing importance comparison between the factors to obtain a factor importance score between the factors, and constructing a task priority analysis hierarchical matrix according to the factor importance score; determining a factor weight feature vector of the task priority analysis hierarchical matrix according to the factor importance score; and performing matrix multiplication operation on the task weight matrix and the factor weight feature vector to obtain the task priority vector.

[0094] In view of the complexity of multi-target decision-making in the mobile device task scheduling scenario, the embodiment proposes a main task priority determination model based on the analytic hierarchy process. The model realizes quantitative ordering of the main task priority by constructing a multi-criteria pair comparison matrix, calculating a feature vector weight, and normalizing task attributes. To determine the priority of the task, key factors, that is, the factors, such as a task data size, a required CPU cycle number, and a deadline, are comprehensively considered. In the embodiment, the deadline is given the highest weight to more accurately reflect the priority requirement of the timeliness of the task. First, the factors of the same level are compared and used to construct a task priority analysis hierarchical matrix wherein, is a factor importance score between the ith1factor and the jth1factor, is represented as:

[0095]

[0096] Table 1 shows different factor importance levels and their weights.

[0097] Table 1 Factor importance level

[0098] factor i1 compared to factor j1 Weight Equal importance 1 More importance 5 Absolute importance 9 Other 2, 4, 6, 8, etc.

[0099] The task weight matrix of all tasks is constructed as wherein N1 represents a task quantity, is a task weight of an rth task based on an mth factor, that is:

[0100]

[0101] wherein, f r m is an importance weight of the rth task. Finally, the task priority vector of all tasks is generated as:

[0102]

[0103] Ψ = Λ * Ψ, wherein Ψ is a task priority vector, Λ is a factor weight eigenvector of a task priority analysis hierarchical matrix, and represents a characteristic vector according to a factor weight of the task priority analysis hierarchical matrix , and an eigenvalue of the characteristic vector can be represented as λ m , λ m is represented as:

[0104]

[0105] wherein m is a factor number, is a factor importance score between a k1th factor and a j1th factor.

[0106] The embodiment focuses on the task offloading and resource allocation problem in the intelligent vehicle networking scene. Through optimizing scheduling and allocating offloading strategies, the system can reasonably formulate a task offloading and resource allocation scheme, thereby reducing the overall network delay, improving resource utilization efficiency, solving the priority and dependency problems of multiple tasks in the system, and improving the timely completion rate of tasks. Therefore, the optimization problem of low-latency task offloading and resource allocation in intelligent vehicle networking is described as:

[0107]

[0108] wherein, ω i is a task completion time weight of an i th task, Ψ i is a task priority score of the i th task, and the higher the task priority score, the higher weight is given to the task completion time, and the scheduler will prioritize optimizing the completion delay of them. The first constraint condition C1 ensures that the subtask in the task is completed after the parent task; the second constraint condition C2 ensures that the main task completion time does not exceed its deadline The third constraint condition C3 ensures that the transmission power p n of the n th vehicle cannot exceed its maximum transmission power The fourth constraint condition C4 ensures that high-priority tasks are executed first, indicating that if the priority of the first main task T a is higher than that of the second main task T b , then any subtask belonging to the first main task T a must be started no later than any subtask belonging to the second main task T b , wherein M is a given constant large enough, δ ab = 1, Ψ a > Ψ b otherwise δ ab = 0, wherein, represents the time when the k th task starts, and δ ab represents a logical variable for judging the task priority, v idenotes the i-th task node, which is a subtask belonging to the first main task, v k denotes the k-th task node, which is a subtask belonging to the second main task.

[0109] Step S103: converting the optimization problem into a Markov decision process model.

[0110] In a possible implementation, the Markov decision process model includes a state space, an action space, and a reward function; wherein the state space includes computing resources available to the vehicle itself, computing resources available to the edge server, and vehicle task characteristics, the vehicle task characteristics including task data volume, deadline, and task dependency relationship; the action space includes a task offloading ratio to the edge server and a computing resource allocation ratio; and the reward function is composed of a time delay reward, a resource utilization rate reward, and a time delay violation penalty term.

[0111] The optimization objective of the Markov decision process model is to minimize the overall task time delay and maximize the resource utilization rate of the vehicle or the edge server.

[0112] To achieve the above objective, i.e., formula (15), a Markov decision process (MDP) model is constructed for intelligent Internet of Vehicles, and a reward function of the model is designed according to the objective of the optimization problem. However, in the intelligent Internet of Vehicles scenario, the environment exhibits high variability, and the current state of the agent is converted to other states under a partially known transition probability. To cope with this challenge, deep reinforcement learning (DRL) is used to iteratively train the agent to optimize the decision-making strategy. Within the DRL framework, a centralized decision-making method is adopted, in which the DRL agent is implemented through a network control module. This module directly solves the actual problem by utilizing its function of collecting comprehensive system state information, thereby facilitating optimal decision-making for all requests. Since the mutual influence of resource allocation among tasks cannot be solved in polynomial time by using traditional methods, the embodiment uses a multi-agent method to convert the problem.

[0113] Due to the high dynamicity of the intelligent Internet of Vehicles environment (such as vehicle movement and task burst), it is difficult for traditional optimization methods to solve in real time. Therefore, the task offloading and resource allocation optimization problem is modeled as an MDP model. At time slot t, the vehicle is regarded as an agent, the agent obtains the current state of the state space through local observation, and uses an action value function Q(s t ,a t ) to obtain a strategy, and the agent selects a corresponding action according to the strategy. After all agents complete the strategy selection, the environment will be converted to a new state s t+1 , and the agent will obtain a corresponding reward r. The specific composition of the MDP model is as follows:

[0114] The state space is represented as follows:

[0115] In time slot t, the agent observes the current communication environment and collects a set of parameters that define its state.

[0116] 1)R v : Indicates the computing resources available to the vehicle itself.

[0117] 2)R b : Indicates the computing resources available to the edge server.

[0118] 3)U n Vehicle task characteristics (such as task data volume, deadline, and dependencies).

[0119] The state space is represented by S, and the state of the system in time slot t is represented by s. t ∈S, where s t A vector belonging to set S is defined as:

[0120]

[0121] in, This represents the computational resources available to the nth vehicle in time slot t. Indicates the bth time slot t n The computing resources available to edge servers This represents the task data generated by the nth vehicle in time slot t.

[0122] The action space is represented as follows:

[0123] For each agent, the action is a continuous vector, represented as:

[0124]

[0125] Among them, a t Let be the system's action strategy in time slot t, where Indicates the Nth t The percentage of tasks uninstalled. Indicates that the edge server is the Nth t The proportion of total resources allocated to each task This is a set of element values ​​representing the ratio of task unloading to resource allocation.

[0126] Then the action space A can be represented as Where N t This represents the maximum number of subtasks that a vehicle can consider simultaneously, τ represents the task unloading ratio, and φ represents the resource allocation ratio. N represents the total number of N. tThe task offloading ratio and the resource allocation ratio are both between 0 and 1. N represents the total number of tasks t The resource allocation ratio is between 0 and 1.

[0127] The reward function is represented as follows:

[0128] The reward function r t The overall reward function is composed of a delay reward, a resource utilization reward, and a delay violation penalty, and is represented as:

[0129]

[0130] where κ1 is the delay reward weight, κ2 is the resource utilization reward weight, κ1, κ2 ∈ (0, 1), R d is the delay reward, R u is the reward value brought by different resource utilization, m1 is a task delay parameter, is an indicator function, which takes 1 if the condition is met, and 0 otherwise, where is the task execution delay, is the maximum task delay limit.

[0131]

[0132] The long-term cumulative reward of the agent J can be represented as:

[0133]

[0134] where r J (t0) represents the reward value of the agent J at time slot t0, γ ∈ (0, 1) is a discount factor, and the optimal strategy for task offloading and resource allocation is obtained by maximizing the cumulative reward of each agent.

[0135] Step S104: The MADDPG algorithm is used to solve the Markov decision process model, and the task offloading ratio to the edge server and the computing resource allocation ratio are outputted, so as to realize the optimization of the Internet of Vehicles heterogeneous layered task offloading and resource allocation.

[0136] In a possible implementation, the improved MADDPG algorithm (BR-MADDPG, multi-agent deep deterministic policy gradient algorithm) is used to solve the Markov decision process model, which includes: each vehicle is modeled as an agent, each agent is trained by the MADDPG algorithm, and each agent is equipped with an Actor (action) network and a Critic (evaluation) network; each agent outputs an action policy through the Actor network according to the state space information of the local observation input; wherein each agent also evaluates the value function of the state-action pair through the Critic network, so that the Actor network adjusts the action policy according to the value evaluation.

[0137] When the Actor network outputs the action policy, a deterministic policy with exploration noise is used for action selection.

[0138] Specifically, the embodiment proposes a BR-MADDPG algorithm based on task relationship in a SIoV network environment, which is used for efficient task offloading decision. In the algorithm, each agent is equipped with a set of independent Actor networks and Critic networks, as well as corresponding target networks (Target Network). Among them, the Actor network is used to generate offloading actions based on local observations, and the Critic network evaluates the value function of the state-action pair. In order to improve the training stability and global optimality, the BR-MADDPG algorithm adopts the strategy of centralized training and distributed execution: in the training stage, the Critic network of the agent can access the state and action information of all agents, so as to more accurately estimate the global Q value; in the execution stage, each agent only relies on the local observation input to make decisions independently by its Actor network.

[0139] Figure 5 The overall training framework of the proposed BR-MADDPG algorithm is shown. In this framework, vehicles are modeled as agents, each agent first perceives the current network state, then selects the corresponding action to interact with the environment, which causes the state of the environment to shift and obtains reward feedback. Through continuous interaction with the environment, the agent continuously collects experience data (state, action, reward, next state), and constructs training samples through the replay mechanism. The edge server acts as a centralized control unit, responsible for managing the key network parameters of each agent, and updating each Critic network uniformly using global state-action information, so as to optimize the policy and achieve the goal of maximizing the global Q value.

[0140] The embodiment uses a deterministic policy with exploration noise for action selection, and the action selection is:

[0141]

[0142] Among them, is the policy network output of the agent J, W t is the exploration noise with training decay at time slot t, with decay rate γ N , P denotes probability, ε t is the random exploration probability with training decay at time slot t, with decay rate γ ε , W min is the minimum value of the exploration noise with training decay, ε min is the minimum value of the random exploration probability with training decay, W t-1 is the exploration noise with training decay at time slot t-1, ε t-1 is the random exploration probability with training decay at time slot t-1. By decaying ε t , the deterministic policy with exploration noise can introduce randomness to better explore the environment and learn more useful information. In the early stage, ε t is large, the agent will take actions randomly and explore more possible actions due to lack of experience. When the experience is sufficient, the agent will take actions to maximize the return. For the agent J, the way to update the Critic network Q is:

[0143]

[0144] wherein, denotes the Q value estimate of the Critic network of the agent J for the current state-action pair, denotes the parameters of the estimated Critic network, r J (t) denotes the immediate reward obtained by the agent J, γ J denotes the discount factor, denotes the Q value estimate of the target Critic network for the next time state-action pair, s t+1 denotes the state information at time slot t+1, a t+1 denotes the action policy at time slot t+1, denotes the parameters of the target Critic network.

[0145] The Critic network is controlled by parameters To obtain the optimal parameters, the loss function must be determined. The loss function of the Critic network can be expressed as:

[0146]

[0147] wherein, denotes the loss function of the Critic network, E denotes expectation.

[0148] The Critic network is then updated by stochastic gradient descent to minimize the loss function:

[0149]

[0150] wherein, denotes the gradient operator, δ denotes the time difference error.

[0151] In the training process, in order to ensure the stability of the algorithm, the parameters of the target Actor network and the target Critic network are updated by soft update, that is:

[0152] θ' J ←τ1*θ J +(1-τ1)*θ' J (26)

[0153] φ' J ←τ1*φ J +(1-τ1)*φ' J (27)

[0154] wherein, θ J , φ J denote the current Actor and Critic network parameters, θ' J , φ' J denote the target Actor and Critic network parameters, and τ1 is a soft update coefficient to control the update rate.

[0155] The effectiveness of the proposed BR-MADDPG algorithm is verified by simulation, and Table 2 lists the parameters of the simulation experiment.

[0156] Table 2 Experimental parameter settings

[0157]

[0158]

[0159] In order to evaluate the proposed algorithm, the random proportion task offloading and resource allocation algorithm is used as the baseline algorithm (Baseline), and compared with the improved MADDQN algorithm to compare the simulation performance.

[0160] Figure 6 The convergence performance of the proposed BR-MADDPG algorithm under different learning rate settings is shown. The number of vehicles is fixed at 8 during the training process. From the overall trend, as the training goes deeper, the cumulative reward of the algorithm gradually increases and tends to be stable. When the learning rate of the Actor network is set to 1 × 10 -5 , the learning rate of the Critic network is set to 1 × 10 -3When the learning rate is too high, the parameter update process is unstable, and it is difficult to achieve effective convergence. When the learning rate is too low, the algorithm may fall into a local optimum, limiting the full play of its performance. Therefore, in the subsequent experiments, the learning rate of the Actor network is 1 x 10 -5 , and the learning rate of the Critic network is 1 x 10 -3 , so as to achieve a good balance between convergence speed and policy stability.

[0161] Figure 7 The influence of the number of vehicles on the average time delay of the system under different algorithms is shown. As the number of vehicles increases, the number of tasks to be processed in the system also increases, resulting in an increase in resource allocation pressure under limited vehicle computing resources and edge server resources, and the resource share of a single task decreases, thereby causing the average time delay of the task to increase. Although the three comparative algorithms all show an upward trend in the average time delay as the number of vehicles increases, the results in Table 4 show that the BR-MADDPG algorithm proposed in this paper can effectively alleviate the time delay growth problem caused by resource bottlenecks when facing increasing resource competition, and its performance is more robust to changes in the number of vehicles, with a smaller increase in time delay, showing better task scheduling capability. Compared with the baseline algorithm, the average time delay is reduced by 31.6% under different numbers of vehicles, and compared with the MADQN (Multi-Agent Deep Q-Network) algorithm, the average time delay is reduced by 10.5%. Figure 7

[0162] Further, the influence of the number of vehicles on the overall resource utilization of the system is shown. As the number of vehicles increases, the number of tasks increases significantly, causing more computing resources in the system to be activated for task processing, thereby improving the overall resource utilization. The comparison results show that the three algorithms can all improve the resource utilization as the number of vehicles increases, and the BR-MADDPG algorithm shows the highest resource utilization efficiency in all experimental settings. This indicates that the algorithm can more fully schedule and configure computing resources under the premise of meeting the time delay constraint and task relationship constraint, achieving better coordination and optimization between resource utilization and task processing efficiency. Compared with the baseline algorithm, the average resource utilization is improved by 19.67% under different numbers of vehicles, and compared with the MADQN algorithm, the average resource utilization is improved by 4.54%. Figure 8

[0163] ​To solve the scheduling challenge caused by task heterogeneity and latency sensitivity in smart internet of vehicles (SIoV), a joint optimization scheme of task offloading and resource allocation based on multi-agent deep deterministic policy gradient (MADDPG) algorithm is proposed. A hypergraph model of task dependency is constructed to effectively depict the hierarchical relationship between tasks and reduce the complexity of system modeling. Meanwhile, a deep reinforcement learning mechanism is introduced to realize adaptive decision-making strategy for task latency constraint and dependency constraint by combining with dynamic characteristics of the environment. Simulation results show that the proposed method exhibits better task response speed and resource scheduling efficiency under various vehicle scales, verifying its practicability and robustness in SIoV scenarios.

[0164] Further, the application also provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to realize the method.

[0165] The device embodiments described above are only schematic, wherein the units described as separate components may or may not be physically separate, and the components displayed as units may or may not be physical units, i.e., may be located in one place, or may be distributed on a plurality of network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the embodiments. Those skilled in the art can understand and implement without creative labor.

[0166] From the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be realized by means of software plus necessary universal hardware platforms, and of course can also be realized by hardware. Based on such understanding, the above technical solutions can be embodied in the form of software product, which can be stored in a computer readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes a plurality of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute the methods described in each embodiment or some parts of the embodiments.

[0167] Finally, it should be noted that the above embodiments are the preferred embodiments of the present application, but the embodiments of the present application are not limited to the above embodiments, and any changes, modifications, substitutions, combinations, simplifications made without departing from the spirit and principles of the present application shall be equivalent replacement modes, and all shall be included in the protection scope of the present application.

Claims

1. A method for MADDPG-based task offloading and resource allocation optimization in a heterogeneous hierarchical task in a vehicle-to-everything network, the method comprising: The application relates to a method for constructing an optimization problem of low-latency task offloading and resource allocation in an intelligent vehicle networking system. The method comprises the following steps: modeling an intelligent vehicle networking system model, the intelligent vehicle networking system model comprising a plurality of vehicles and a plurality of edge servers, each edge server being configured with a base station and a road side unit, and each vehicle being connected to the base station and the road side unit through a V2I link, wherein each V2I link is assigned an orthogonal frequency subband; constructing the optimization problem of low-latency task offloading and resource allocation based on the intelligent vehicle networking system model, wherein the tasks are main tasks generated by the vehicles and subtasks dependent on the main tasks; transforming the optimization problem into a Markov decision process model; 2. The method of claim 1, wherein, solving the Markov decision process model by using a MADDPG algorithm to output the proportion of task offloading to the edge servers and the proportion of computing resource allocation, and realizing the optimization of vehicle networking heterogeneous layered task offloading and resource allocation. The method for constructing the optimization problem comprises the following steps: calculating the communication rate between the vehicles and the base stations based on the channel gain between the vehicles and the base stations; determining the task uploading delay according to the communication rate, and determining the total task delay according to the local processing delay of the vehicle, the task uploading delay and the edge computing delay; determining a task dependency matrix, and determining the task completion time according to the task dependency matrix and the total task delay; determining a task priority vector, and determining the task completion time weight according to the task priority vector; 3. The method of claim 1, wherein, performing weighted summation calculation on the task completion time based on the task completion time weight, and constructing an objective function of the optimization problem with the minimum sum of the weighted completion times of all tasks as the target. The constraint conditions of the optimization problem comprise first to fourth constraint conditions; wherein the first constraint condition is that the subtask in the subtask set is completed after the parent task; the second constraint condition is that the main task completion time does not exceed the deadline of the main task; the third constraint condition is that the transmission power of the vehicle does not exceed the maximum transmission power of the vehicle; 4. The method of claim 2, wherein, the fourth constraint condition is that if the priority of the first main task is higher than that of the second main task, any subtask belonging to the first main task must be started no later than any subtask belonging to the second main task. The method for determining the task dependency matrix comprises the following steps:

5. The method of claim 2, wherein, constructing a task hypergraph, and determining whether the task belongs to the parent task set or the subtask set in the hyperedge based on the task hypergraph to determine the task dependency matrix. The method for determining the task completion time according to the task dependency matrix and the total task delay comprises the following steps:

6. The method of claim 2, wherein, determining the time required by the previous historical task completion and the maximum completion time of the parent task set in the parent task set required before the execution of the current task according to the task dependency matrix when the task is calculated on the same device, and obtaining the larger one of the two, and summing the larger one with the total task delay to obtain the task completion time. The method for determining the task priority vector comprises the following steps: determining factors for judging the task priority, wherein the factors comprise the task data size, the required CPU cycle number and the deadline; determining the task weight of each task corresponding to each factor to construct a task weight matrix; performing importance comparison between the factors to obtain a factor importance score between the factors, and constructing a task priority analysis hierarchical matrix according to the factor importance score; determining a factor weight feature vector of the task priority analysis hierarchical matrix according to the factor importance score. The task priority vector is obtained by performing matrix multiplication operation on the task weight matrix and the factor weight feature vector.

7. The method of claim 1, wherein, The Markov decision process model comprises a state space, an action space and a reward function, wherein The state space comprises computing resources available to the vehicle itself, computing resources available to the edge server and vehicle task characteristics, the vehicle task characteristics comprising task data volume, deadline and task dependency relationship; The action space comprises a task offloading ratio to the edge server and a computing resource allocation ratio; The reward function is composed of a time delay reward, a resource utilization reward and a time delay violation penalty term; The optimization objective of the Markov decision process model is to minimize the overall time delay of the tasks and maximize the resource utilization of the vehicle or the edge server.

8. The method according to any one of claims 1 to 7, characterized in that, Solving the Markov decision process model by using the MADDPG algorithm comprises: Each vehicle is modeled as an agent, each agent is trained by using the MADDPG algorithm, and each agent is equipped with an Actor network and a Critic network; Each agent outputs an action policy according to the state space information input by local observation by using the Actor network; Each agent further evaluates a value function of a state-action pair by using the Critic network, so that the Actor network adjusts the action policy according to the value evaluation.

9. The method of claim 8, wherein, When the Actor network outputs the action policy, a deterministic policy with exploration noise is used for action selection.

10. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program, when executed by a processor, implements the method according to any one of claims 1-9.

Images