Federated multi-agent Actor-Critic learning intelligent logistics task offloading and resource allocation system and medium

Abstract

The present invention discloses a federated multi-agent Actor-Critic learning intelligent logistics task unloading and resource allocation system and medium, the system includes a plurality of logistics vehicles, a plurality of assistant warehouses, and a plurality of assistant vehicles; the on-board server of the logistics vehicle stores a delay target calculation model, an energy consumption target calculation model, a task unloading and resource allocation optimization model, and an optimization model calculation framework; the on-board server of the logistics vehicle stores a delay target T m and energy consumption target E m The input is input into the task offloading and resource allocation optimization model, and the offloading strategy of the task sequence #imgabs0# is solved; the onboard server of the logistics vehicle executes the task locally or offloads the task to the server of the assistant vehicle or assistant warehouse for execution according to the offloading strategy. The present invention proposes a multi-agent deep deterministic policy gradient task offloading strategy. The logistics vehicle agent takes action independently, but improves its strategy through collaborative exploration of the environment, thereby determining the task offloading path and resource allocation, and improving the overall performance of the system.

Description

Technical Field

[0001] The present invention relates to the field of logistics, and in particular to a federated multi-agent Actor-Critic learning intelligent logistics task unloading and resource allocation system and medium. Background Art

[0002] Logistics supply chain is an important driving force for national and corporate competition and plays a vital role in economic growth. Smart logistics enabled by artificial intelligence has become an inevitable trend in the development of modern logistics, so convenient and efficient logistics systems have attracted widespread attention.

[0003] At present, with the continuous development of modern advanced information and communication technology (ICT), the Industrial Internet of Things (IIoT) has changed the operation mode and system structure of the logistics system, and smart logistics has become an inevitable trend in the development of modern logistics.

[0004] IIoT analyzes and processes massive amounts of logistics data and information, and combines cloud computing, big data, artificial intelligence and other advanced technologies to achieve intelligent control of logistics objects. Among them, artificial intelligence (AI) as an important technology has been applied to many fields of smart logistics, including logistics transportation, warehousing, loading and unloading, distribution processing, information services, etc., which helps save time and costs, greatly improves logistics transportation efficiency, and promotes the development of smart logistics.

[0005] As the number of sensor nodes deployed in smart logistics continues to increase, the amount of data is also growing exponentially. However, due to the limited computing and communication resources of logistics vehicles, it is difficult to meet the Quality of Service (QoS) requirements of computationally intensive and delay-sensitive tasks in smart logistics.

[0006] Therefore, how to use AI to perform computationally intensive applications on resource-constrained logistics vehicles remains a huge challenge.

[0007] To solve the above problems, logistics vehicles can reduce the load of sensor nodes by optimizing task offloading and obtaining assistance from other computing paradigms, such as Mobile Edge Computing (MEC), which transfers computing to the edge of the network close to the vehicle by placing edge servers closer to the vehicle to provide much higher computing power than the vehicle, and can obtain lower communication latency. In addition, multi-level collaborative task offloading is also often considered. Multi-level collaborative task offloading usually refers to different collaborative processing nodes jointly processing offloaded tasks, including end-edge, edge-cloud, and end-edge-cloud collaboration. Based on the different QoS requirements of applications and the application scenarios of different collaborative processing nodes, latency-sensitive computing-intensive applications are offloaded from logistics vehicles to collaborative processing nodes with additional computing resources, which largely makes up for the problem of insufficient computing power of logistics vehicles.

[0008] Based on the above analysis, how to offload tasks and which tasks to offload become one of the key issues that need to be solved. The task offloading problem of the logistics system in IIoT usually inevitably involves the allocation of computing and transmission resources. Therefore, this problem can be easily transformed into a limited resource allocation problem.

[0009] Among them, task modeling is an important prerequisite for exploring this optimization problem. However, smart logistics will change accordingly according to actual technologies and applications. Identity resolution provides a real-time traceability environment for tasks, and multiple tasks usually have dependencies. Therefore, the above methods are prone to waste resources and are not in line with actual conditions. Considering the dynamics and heterogeneity of logistics systems in the current IIoT, most traditional methods that use one-time optimization may not achieve stable long-term optimization performance. Summary of the invention

[0010] The purpose of the present invention is to provide a federated multi-agent Actor-Critic learning intelligent logistics task unloading and resource allocation system, including a number of logistics vehicles, a number of assistant warehouses, and a number of assistant vehicles;

[0011] The assistant warehouse is equipped with a warehouse server;

[0012] The logistics vehicles and assistant vehicles are both equipped with an on-board server;

[0013] Among them, the onboard server of the mth logistics vehicle generates a task sequence and transmitted to the warehouse server of the assistant warehouse; U is the total number of tasks;

[0014] The onboard server of the logistics vehicle stores a delay target calculation model, an energy consumption target calculation model, a task offloading and resource allocation optimization model, and an optimization model calculation framework;

[0015] The onboard server of the logistics vehicle calculates the task sequence using the delay target calculation model and the energy consumption target calculation model respectively. The delay target T m and energy consumption target E m ;

[0016] The onboard server of the logistics vehicle sends the task sequence The delay target T m and energy consumption target E m Input into the task offloading and resource allocation optimization model, and use the optimization model calculation framework to solve the task sequence Uninstall strategy;

[0017] The onboard server of the logistics vehicle executes the task locally or offloads the task to the server of the assistant vehicle or assistant warehouse for execution according to the unloading strategy.

[0018] Furthermore, the task sequence is a linear sequence; where the u-th subtask T m,u ={d m,u ,c m,u ,τ m,u};d m,u Represents task T m,u The input data size, c m,u Indicates completion of task T m,u The total CPU cycles required, τ m,u Represents task T m,u Tolerance delay.

[0019] Furthermore, the delay target T m As shown below:

[0020]

[0021] In the formula, is the u-th subtask T calculated by the warehouse server using the delay calculation model m,u The computational delay of For task T m,u The transmission delay from unloading at the mth logistics vehicle to the bth assistant warehouse or the nth assistant vehicle.

[0022] Furthermore, the warehouse server calculates the u-th subtask T using the delay calculation model m,u The computational delay As shown below:

[0023]

[0024] In the formula, α m,u , β m,u, γ m,u is the uninstall indicator variable; α m,u =1 indicates task T m,u Executed locally on the mth logistics vehicle, α m,u =0 indicates task T m,u Not executed locally on the mth logistics vehicle; β m,u,b =1 indicates task T m,u Unload to the bth assistant warehouse for execution; β m,u,b =0 indicates task T m,u Not executed in the b-th assistant warehouse; γ m,u,n =1 indicates task T m,u Unload to the nth assistant vehicle for execution; γ m,u,n =0 indicates task T m,u Not executed on the nth helper vehicle;

[0025] Among them, task T m,u Computation latency performed locally on the mth logistics vehicle Task T m,u Computation latency of offloading to the bth assistant warehouse Task T m,u Computation latency offloaded to the nth helper vehicle They are as follows:

[0026]

[0027]

[0028]

[0029] In the formula, and They represent the mth logistics vehicle, bth assistant warehouse and nth assistant vehicle assigned to task T respectively. m,u computing resources; c m,u Indicates completion of task T m,u Total CPU cycles required.

[0030] Further, Task T m,u Transmission delay from unloading the mth logistics vehicle to the bth assistant warehouse or the nth assistant vehicle As shown below:

[0031]

[0032] Wherein, vehicle o represents the assistant vehicle that performs the u-1th subtask of the mth logistics vehicle; α m,u , γ m,u-1,n , γ m,u,o is the uninstall indicator variable;

[0033] Among them, task T m,u Transfer rate from unloading of the mth logistics vehicle to the bth assistant warehouse Task T m,u Transfer rate from unloading of the mth logistics vehicle to the nth assistant vehicle They are as follows:

[0034]

[0035]

[0036] Where w is the bandwidth of the logistics vehicle; is the transmission power of LVm; 2 is the Gaussian white noise power. Represents task T m,u Path loss from unloading at the mth logistics vehicle to the bth assistant warehouse; Represents task T m,u The path loss from unloading the mth logistics vehicle to the nth assistant vehicle; h m is the channel fading coefficient.

[0037] Furthermore, the energy consumption target E m As shown below:

[0038]

[0039] Among them, local computing energy consumption Task T m,u Energy consumption from unloading from the mth logistics vehicle to the bth assistant warehouse Task T m,u Energy consumption from unloading from the mth logistics vehicle to the nth assistant vehicle They are as follows:

[0040]

[0041]

[0042]

[0043] Where k represents the computational energy efficiency coefficient associated with the processor chip of the vehicle server.

[0044] Furthermore, the task offloading and resource allocation optimization model min O is as follows:

[0045]

[0046] In the formula, the unloading indicator variables α, β, and γ are binary variables; the allocation of computing resources is a real number. Respectively represent the computing power of the assistant warehouse server and the computing power of the assistant vehicle server; α m,u , β m,u,b , γ m,u,n is the unloading indicator variable; ω m is the weighting factor; is the upper limit of energy consumption; τ m,u The upper limit of the tolerable delay; They represent assistant warehouse set, logistics vehicle set, assistant vehicle set, and task set respectively.

[0047] Furthermore, the steps of establishing the optimization model calculation framework include:

[0048] 1) Local training of Actor network parameters and Critic network parameters The steps include:

[0049] 1.1) Each logistics vehicle is modeled as a DDPG agent, including an Actor network Actor and a Critic network Critic; the input of the Actor network is the local state obtained by the LV agent observing the network environment of the logistics system, and the output is the task offloading action; the Critic network takes the local state and the selected action as input and outputs the estimated value of the current state; the Actor network parameters and Critic network parameters of the mth logistics vehicle are expressed as and

[0050] 1.2) Every t cycles, use the experience replay strategy to minimize the loss function To update the Critic network;

[0051] Loss Function As shown below:

[0052]

[0053] In the formula, represents the mini-batch sample size of the mth logistics vehicle; Indicates local state and task execution actions; Q m is the action value;

[0054] Among them, the target value generated by the Critic network As shown below:

[0055]

[0056] In the formula, Q' m is the action value; Represents local state and task execution actions; represents the reward; γ is the calculation coefficient;

[0057] 1.3) Calculate the expected reward gradient update of the Actor network of the mth logistics vehicle Right now:

[0058]

[0059] Where D m is the number of batch samples; is the gradient; is the action value; Represents local state and task execution actions; is the Critic network parameter; Perform actions for tasks;

[0060] Among them, the parameters of the Actor network Updated as follows:

[0061]

[0062] Where δ is the update coefficient;

[0063] 1.4) Update Actor network parameters and Critic network parameters Right now:

[0064]

[0065]

[0066] In the formula, τ is the weight parameter;

[0067] 2) Each logistics vehicle agent will be trained locally to obtain the Actor network parameters and Critic network parameters Upload to the warehouse server to perform optimization model calculation framework aggregation;

[0068] Among them, the weight update of the global optimization model calculation framework in round j is:

[0069]

[0070] In the formula, θ j and They are the parameters of the optimization model calculation framework of the jth round and the parameters of the local optimization model calculation framework at the logistics vehicle agent m; is the total batch sample size of all logistics vehicle agents;

[0071] 3) The warehouse server distributes the aggregated global model to all logistics vehicle agents to update the local optimization model calculation framework.

[0072] Furthermore, the optimization model calculation framework includes a state space, an action space and a reward function;

[0073] The state space stores the local state S of the logistics vehicle agent m (t), that is:

[0074] S m (t) = {S m,1+N (t),S m,B (t)} (21)

[0075] In the formula, S m,1+N (t) = {Task m (t),F m,1+N (t),I m (t),Y m,1+N (t)} represents the information set of the mth logistics vehicle and N assistant vehicles; S m,B (t) = {F B (t),Y B (t)} represents the information set of B assistant warehouses; Task m (t) represents the task of the mth logistics vehicle, F m,1+N (t) includes the computing power of the mth logistics vehicle and the computing power of N assistant vehicles I m (t) includes the channel gain and transmission power of the mth logistics vehicle; Y m,1+N (t) includes the real-time geographic location of the mth logistics vehicle m and N assistant vehicles; F B (t) represents the computing power of the warehouse server of the assistant warehouse Y B (t) represents the geographical location of the assistant warehouse;

[0076] The action space stores the offloading strategy A of the task m (t), that is:

[0077] A m (t) = {K m (t),F m,B (t)} (22)

[0078] In the formula, K m (t) = [K m,u (t)] determines the subtask computing node in the mth logistics vehicle; represents the computing resources allocated by the warehouse server of the assistant warehouse to the mth logistics vehicle; K m,u (t) represents the binary operation set of each subtask when it is executed locally, unloaded to the assistant vehicle, and unloaded to the assistant warehouse; Indicates that the b-th assistant warehouse is assigned to task T m,u computing resources;

[0079] Reward function R m (t) is as follows:

[0080]

[0081] In the formula, O m (t) is the objective function of the task offloading and resource allocation optimization model.

[0082] Furthermore, the logistics vehicle set is recorded as The assistant warehouse collection is recorded as The assistant vehicle set is denoted as M, B, and N represent the number of logistics vehicles, assistant warehouses, and assistant vehicles in a logistics scheduling area, respectively.

[0083] A computer-readable medium storing a computer program of the above-mentioned federated multi-agent Actor-Critic learning intelligent logistics task unloading and resource allocation system;

[0084] The computer program is used to generate a federated multi-agent Actor-Critic learning smart logistics task unloading and resource allocation solution;

[0085] When the computer program is executed by a processor, the following steps are implemented:

[0086] 1) The onboard server of the mth logistics vehicle generates a task sequence and transmitted to the warehouse server of the assistant warehouse; U is the total number of tasks;

[0087] 2) The onboard server of the logistics vehicle calculates the task sequence using the delay target calculation model and the energy consumption target calculation model respectively The delay target T m and energy consumption target E m ;

[0088] 3) The onboard server of the logistics vehicle sends the task sequence The delay target T m and energy consumption target E m Input into the task offloading and resource allocation optimization model, and use the optimization model calculation framework to solve the task sequence Uninstall strategy;

[0089] 4) The onboard server of the logistics vehicle executes the task locally or offloads the task to the server of the assistant vehicle or assistant warehouse for execution according to the offloading strategy.

[0090] The technical effect of the present invention is unquestionable. The present invention is aimed at dependent applications in smart logistics, considers the dependencies between tasks, and studies the dynamic task offloading strategy of end-edge collaboration to meet the service quality requirements (Quality of Service, QoS) of logistics vehicles for delay and energy consumption. First, the dependent application ARCore is modeled and converted into a model with a linear execution sequence. Secondly, using the ARCore model, a joint optimization problem of task offloading and resource allocation is established, and a multi-agent deep deterministic policy gradient (Multi-Agent Deep Deterministic Policy Gradient, MADDPG) task offloading strategy is proposed, the goal of which is to ensure that each logistics vehicle carrying a task minimizes the total system cost of delay and energy consumption while meeting QoS. Finally, in order to reduce the computational complexity and signaling overhead of the multi-agent training process, a federated learning-assisted MADDPG learning architecture is designed, which only requires each agent to share its model parameters without sharing local training data. Numerical results show that compared with the baseline strategy, the proposed strategy has obvious superiority in terms of total system cost, and its average system cost of delay and energy consumption is reduced by at least 9.63%.

[0091] In order to meet the differentiated requirements of dependent applications in IIoT in terms of latency and energy consumption, the design of the task offloading strategy should combine the QoS requirements of logistics vehicles, the internal dependency structure of the application, and the advantages and disadvantages of the computing paradigms at all levels, and jointly manage and allocate task offloading decisions and computing resources at all levels, and then associate tasks with collaborative processing nodes. This paper proposes an intelligent collaborative offloading strategy for dependent applications. This strategy first offloads computing-intensive applications with dependency constraints to the end-edge collaborative computing architecture, and formulates the task offloading problem as a system total cost minimization problem consisting of latency and energy consumption. Then, the multi-agent DRL algorithm is used to select collaborative processing nodes and allocate computing resources based on the dependencies between tasks and the computing resources at all levels of the end-edge.

[0092] The present invention is mainly aimed at dependent logistics applications, which are composed of several fixed modules and cannot be arbitrarily partitioned. Taking the ARCore application model as an example, it can be decomposed and modeled as a task model with linear sequence execution, including multiple modules, and the current submodule depends on the output of the previous submodule.

[0093] This paper models the joint task offloading and resource allocation problem of the smart logistics system as a multi-agent DRL problem to minimize the system's latency and energy consumption. A multi-agent deep deterministic policy gradient (MADDPG) task offloading strategy is proposed, in which logistics vehicle agents act independently based on their local observations, but improve their strategies by collaboratively exploring the environment, thereby determining the task offloading path and resource allocation, and improving the overall system performance.

[0094] In order to reduce the signaling overhead caused by the exchange of local information and action values ​​between agents, the present invention designs a federated learning-assisted multi-agent DRL architecture, which only requires each task vehicle agent to share its model parameters to the warehouse without sharing local training data. BRIEF DESCRIPTION OF THE DRAWINGS

[0095] Figure 1 is the system model;

[0096] Figure 2 For the task model;

[0097] Figure 3 An example of scheduling a single logistics vehicle task;

[0098] Figure 4 Learning framework for F-MADDPG.

[0099] Figure 5 It is the convergence diagram of F-MADDPG model;

[0100] Figure 6 is the average cost of different task data sizes;

[0101] Figure 7 Calculate the average cost for different logistics vehicles under different capabilities;

[0102] Figure 8 is the average cost under different MEC server computing capabilities;

[0103] Fig. 9 is the average cost under different numbers of logistics vehicles. DETAILED DESCRIPTION

[0104] The present invention is further described below in conjunction with the embodiments, but it should not be understood that the above subject matter of the present invention is limited to the following embodiments. Without departing from the above technical ideas of the present invention, various substitutions and changes are made according to the common technical knowledge and customary means in the art, which should all be included in the protection scope of the present invention.

[0105] Embodiment 1:

[0106] See also Figures 1 to 9 , a federated multi-agent Actor-Critic learning intelligent logistics task unloading and resource allocation system, including several logistics vehicles, several assistant warehouses, and several assistant vehicles;

[0107] The assistant warehouse is equipped with a warehouse server;

[0108] The logistics vehicles and assistant vehicles are both equipped with an on-board server;

[0109] Among them, the onboard server of the mth logistics vehicle generates a task sequence and transmitted to the warehouse server of the assistant warehouse; U is the total number of tasks;

[0110] The onboard server of the logistics vehicle stores a delay target calculation model, an energy consumption target calculation model, a task offloading and resource allocation optimization model, and an optimization model calculation framework;

[0111] The onboard server of the logistics vehicle calculates the task sequence using the delay target calculation model and the energy consumption target calculation model respectively. The delay target T m and energy consumption target E m ;

[0112] The onboard server of the logistics vehicle sends the task sequence The delay target T m and energy consumption target E m Input into the task offloading and resource allocation optimization model, and use the optimization model calculation framework to solve the task sequence Uninstall strategy;

[0113] The onboard server of the logistics vehicle executes the task locally or offloads the task to the server of the assistant vehicle or assistant warehouse for execution according to the unloading strategy.

[0114] The task sequence is a linear sequence; where the u-th subtask T m,u ={d m,u ,c m,u ,τ m,u};d m,u Represents task T m,u The input data size, c m,u Indicates completion of task T m,u The total CPU cycles required, τ m,u Represents task T m,u Tolerance delay.

[0115] The delay target T m As shown below:

[0116]

[0117] In the formula, is the u-th subtask T calculated by the warehouse server using the delay calculation model m,u The computational delay of For task T m,u The transmission delay from unloading at the mth logistics vehicle to the bth assistant warehouse or the nth assistant vehicle.

[0118] The warehouse server calculates the u-th subtask T using the delay calculation model m,u The computational delay As shown below:

[0119]

[0120] In the formula, α m,u , β m,u , γ m,u is the uninstall indicator variable; α m,u =1 indicates task T m,u Executed locally on the mth logistics vehicle, α m,u =0 indicates task T m,u Not executed locally on the mth logistics vehicle; β m,u,b =1 indicates task T m,u Unload to the bth assistant warehouse for execution; β m,u,b =0 indicates task T m,u Not executed in the b-th assistant warehouse; γ m,u,n =1 indicates task T m,u Unload to the nth assistant vehicle for execution; γ m,u,n =0 indicates task T m,u Not executed on the nth helper vehicle;

[0121] Among them, task T m,u Computation latency performed locally on the mth logistics vehicle Task T m,u Computation latency of offloading to the bth assistant warehouse Task T m,u Computation latency offloaded to the nth helper vehicle They are as follows:

[0122]

[0123]

[0124]

[0125] In the formula, and They represent the mth logistics vehicle, bth assistant warehouse and nth assistant vehicle assigned to task T respectively. m,u computing resources; c m,u Indicates completion of task T m,u Total CPU cycles required.

[0126] Task T m,u Transmission delay from unloading the mth logistics vehicle to the bth assistant warehouse or the nth assistant vehicle As shown below:

[0127]

[0128] Wherein, vehicle o represents the assistant vehicle that performs the u-1th subtask of the mth logistics vehicle; α m,u , γ m,u-1,n , γ m,u,o is the uninstall indicator variable;

[0129] Among them, task T m,u Transfer rate from unloading of the mth logistics vehicle to the bth assistant warehouse Task T m,u Transfer rate from unloading of the mth logistics vehicle to the nth assistant vehicle They are as follows:

[0130]

[0131]

[0132] Where w is the bandwidth of the logistics vehicle; is the transmission power of LVm; 2 is the Gaussian white noise power. Represents task T m,u Path loss from unloading at the mth logistics vehicle to the bth assistant warehouse; Represents task T m,u The path loss from unloading the mth logistics vehicle to the nth assistant vehicle; h m is the channel fading coefficient.

[0133] Energy consumption target E m As shown below:

[0134]

[0135] Among them, local computing energy consumption Task T m,u Energy consumption from unloading from the mth logistics vehicle to the bth assistant warehouse Task T m,uEnergy consumption from unloading from the mth logistics vehicle to the nth assistant vehicle They are as follows:

[0136]

[0137]

[0138]

[0139] Where k represents the computational energy efficiency coefficient associated with the processor chip of the vehicle server.

[0140] The task offloading and resource allocation optimization model min O is as follows:

[0141]

[0142] In the formula, the unloading indicator variables α, β, and γ are binary variables; the allocation of computing resources is a real number. Respectively represent the computing power of the assistant warehouse server and the computing power of the assistant vehicle server; α m,u , β m,u,b , γ m,u,n is the unloading indicator variable; ω m is the weighting factor; is the upper limit of energy consumption; τ m,u The upper limit of the tolerable delay; They represent assistant warehouse set, logistics vehicle set, assistant vehicle set, and task set respectively.

[0143] The steps to establish the optimization model calculation framework include:

[0144] 1) Local training of Actor network parameters and Critic network parameters The steps include:

[0145] 1.1) Each logistics vehicle is modeled as a DDPG agent, including an Actor network Actor and a Critic network Critic; the input of the Actor network is the local state obtained by the LV agent observing the network environment of the logistics system, and the output is the task offloading action; the Critic network takes the local state and the selected action as input and outputs the estimated value of the current state; the Actor network parameters and Critic network parameters of the mth logistics vehicle are expressed as and

[0146] 1.2) Every t cycles, use the experience replay strategy to minimize the loss function To update the Critic network;

[0147] Loss Function As shown below:

[0148]

[0149] In the formula, represents the mini-batch sample size of the mth logistics vehicle; Indicates local state and task execution actions; Q m is the action value;

[0150] Among them, the target value generated by the Critic network As shown below:

[0151]

[0152] In the formula, Q' m is the action value; Represents local state and task execution actions; represents the reward; γ is the calculation coefficient;

[0153] 1.3) Calculate the expected reward gradient update of the Actor network of the mth logistics vehicle Right now:

[0154]

[0155] Where D m is the number of batch samples; is the gradient; is the action value; Represents local state and task execution actions; is the Critic network parameter; Perform actions for tasks;

[0156] Among them, the parameters of the Actor network Updated as follows:

[0157]

[0158] Where δ is the update coefficient;

[0159] 1.4) Update Actor network parameters and Critic network parameters Right now:

[0160]

[0161]

[0162] In the formula, τ is the weight parameter;

[0163] 2) Each logistics vehicle agent will be trained locally to obtain the Actor network parameters and Critic network parameters Upload to the warehouse server to perform optimization model calculation framework aggregation;

[0164] Among them, the weight update of the global optimization model calculation framework in round j is:

[0165]

[0166] In the formula, θ j and They are the parameters of the optimization model calculation framework of the jth round and the parameters of the local optimization model calculation framework at the logistics vehicle agent m; is the total batch sample size of all logistics vehicle agents;

[0167] 3) The warehouse server distributes the aggregated global model to all logistics vehicle agents to update the local optimization model calculation framework.

[0168] The optimization model calculation framework includes a state space, an action space and a reward function;

[0169] The state space stores the local state S of the logistics vehicle agent m (t), that is:

[0170] S m (t) = {S m,1+N (t),S m,B (t)} (21)

[0171] In the formula, S m,1+N (t) = {Task m (t),F m,1+N (t),I m (t),Y m,1+N (t)} represents the information set of the mth logistics vehicle and N assistant vehicles; S m,B (t) = {F B (t),Y B (t)} represents the information set of B assistant warehouses; Task m (t) represents the task of the mth logistics vehicle, F m,1+N (t) includes the computing power of the mth logistics vehicle and the computing power of N assistant vehicles I m (t) includes the channel gain and transmission power of the mth logistics vehicle; Y m,1+N(t) includes the real-time geographic location of the mth logistics vehicle m and N assistant vehicles; F B (t) represents the computing power of the warehouse server of the assistant warehouse Y B (t) represents the geographical location of the assistant warehouse;

[0172] The action space stores the task offloading strategy A m (t), that is:

[0173] A m (t) = {K m (t),F m,B (t)} (22)

[0174] In the formula, K m (t) = [K m,u (t)] determines the subtask computing node in the mth logistics vehicle; represents the computing resources allocated by the warehouse server of the assistant warehouse to the mth logistics vehicle; K m,u (t) represents the binary operation set of each subtask when it is executed locally, unloaded to the assistant vehicle, and unloaded to the assistant warehouse; Indicates that the b-th assistant warehouse is assigned to task T m,u computing resources;

[0175] Reward function R m (t) is as follows:

[0176]

[0177] In the formula, O m (t) is the objective function of the task offloading and resource allocation optimization model.

[0178] The logistics vehicle set is denoted as The assistant warehouse collection is recorded as The assistant vehicle set is denoted as M, B, and N represent the number of logistics vehicles, assistant warehouses, and assistant vehicles in a logistics scheduling area, respectively.

[0179] A computer-readable medium storing a computer program of the above-mentioned federated multi-agent Actor-Critic learning intelligent logistics task unloading and resource allocation system;

[0180] The computer program is used to generate a federated multi-agent Actor-Critic learning smart logistics task unloading and resource allocation solution;

[0181] When the computer program is executed by a processor, the following steps are implemented:

[0182] 1) The onboard server of the mth logistics vehicle generates a task sequence and transmitted to the warehouse server of the assistant warehouse; U is the total number of tasks;

[0183] 2) The onboard server of the logistics vehicle calculates the task sequence using the delay target calculation model and the energy consumption target calculation model respectively The delay target T m and energy consumption target E m ;

[0184] 3) The onboard server of the logistics vehicle sends the task sequence The delay target T m and energy consumption target E m Input into the task offloading and resource allocation optimization model, and use the optimization model calculation framework to solve the task sequence Uninstall strategy;

[0185] 4) The onboard server of the logistics vehicle executes the task locally or offloads the task to the server of the assistant vehicle or assistant warehouse for execution according to the offloading strategy.

[0186] Embodiment 2:

[0187] A federated multi-agent Actor-Critic learning intelligent logistics task unloading and resource allocation system, including several logistics vehicles, several assistant warehouses, and several assistant vehicles;

[0188] The assistant warehouse is equipped with a warehouse server;

[0189] The logistics vehicles and assistant vehicles are both equipped with an on-board server;

[0190] Among them, the onboard server of the mth logistics vehicle generates a task sequence and transmitted to the warehouse server of the assistant warehouse; U is the total number of tasks;

[0191] The onboard server of the logistics vehicle stores a delay target calculation model, an energy consumption target calculation model, a task offloading and resource allocation optimization model, and an optimization model calculation framework;

[0192] The onboard server of the logistics vehicle calculates the task sequence using the delay target calculation model and the energy consumption target calculation model respectively. The delay target T m and energy consumption target E m ;

[0193] The onboard server of the logistics vehicle sends the task sequence The delay target T m and energy consumption target E mInput into the task offloading and resource allocation optimization model, and use the optimization model calculation framework to solve the task sequence Uninstall strategy;

[0194] The onboard server of the logistics vehicle executes the task locally or offloads the task to the server of the assistant vehicle or assistant warehouse for execution according to the unloading strategy.

[0195] Embodiment 3:

[0196] Federated multi-agent Actor-Critic learning intelligent logistics task unloading and resource allocation system, the main content is shown in Example 2, wherein the task sequence is a linear sequence; where the u-th subtask T m,u ={d m,u ,c m,u ,τ m,u};d m,u Represents task T m,u The input data size, c m,u Indicates completion of task T m,u The total CPU cycles required, τ m,u Represents task T m,u Tolerance delay.

[0197] Embodiment 4:

[0198] Federated multi-agent Actor-Critic learning intelligent logistics task unloading and resource allocation system, the main content is shown in Example 2, wherein the delay target T m As shown below:

[0199]

[0200] In the formula, is the u-th subtask T calculated by the warehouse server using the delay calculation model m,u The computational delay of For task T m,u The transmission delay from unloading at the mth logistics vehicle to the bth assistant warehouse or the nth assistant vehicle.

[0201] Embodiment 5:

[0202] The federated multi-agent Actor-Critic learning intelligent logistics task unloading and resource allocation system, the main content of which is shown in Example 2, wherein the warehouse server uses the time delay calculation model to calculate the u-th subtask T m,u The computational delay As shown below:

[0203]

[0204] In the formula, α m,u , β m,u , γ m,u is the uninstall indicator variable; α m,u =1 indicates task T m,u Executed locally on the mth logistics vehicle, α m,u =0 indicates task T m,u Not executed locally on the mth logistics vehicle; β m,u,b =1 indicates task T m,u Unload to the bth assistant warehouse for execution; β m,u,b =0 indicates task T m,u Not executed in the b-th assistant warehouse; γ m,u,n =1 indicates task T m,u Unload to the nth assistant vehicle for execution; γ m,u,n =0 indicates task T m,u Not executed on the nth helper vehicle;

[0205] Among them, task T m,u Computation latency performed locally on the mth logistics vehicle Task T m,u Computation latency of offloading to the bth assistant warehouse Task T m,u Computation latency offloaded to the nth helper vehicle They are as follows:

[0206]

[0207]

[0208]

[0209] In the formula, and They represent the mth logistics vehicle, bth assistant warehouse and nth assistant vehicle assigned to task T respectively. m,u computing resources; c m,u Indicates completion of task T m,u Total CPU cycles required.

[0210] Embodiment 6:

[0211] The federated multi-agent Actor-Critic learning intelligent logistics task unloading and resource allocation system, the main content of which is shown in Example 2, where task T m,u Transmission delay from unloading the mth logistics vehicle to the bth assistant warehouse or the nth assistant vehicle As shown below:

[0212]

[0213] In the formula, vehicle o represents the assistant vehicle that performs the u-1th subtask of the mth logistics vehicle;

[0214] Among them, task T m,u The transfer rate r from the mth logistics vehicle to the bth assistant warehouse m,u,b 、Task T m,u Transfer rate r from unloading from the mth logistics vehicle to the nth assistant vehicle m,u,n They are as follows:

[0215]

[0216]

[0217] Where w is the bandwidth of the logistics vehicle; is the transmission power of LVm; 2 is the Gaussian white noise power.

[0218] Embodiment 7:

[0219] Federated multi-agent Actor-Critic learning intelligent logistics task unloading and resource allocation system, the main content is shown in Example 2, where the energy consumption target E m As shown below:

[0220]

[0221] Among them, local computing energy consumption Task T m,u Energy consumption from unloading from the mth logistics vehicle to the bth assistant warehouse Task T m,u Energy consumption from unloading from the mth logistics vehicle to the nth assistant vehicle They are as follows:

[0222]

[0223]

[0224]

[0225] Where k represents the computational energy efficiency coefficient associated with the processor chip of the vehicle server.

[0226] Embodiment 8:

[0227] The federated multi-agent Actor-Critic learning intelligent logistics task offloading and resource allocation system, the main content of which is shown in Example 2, where the task offloading and resource allocation optimization model is as follows:

[0228]

[0229] In the formula, the unloading indicator variables α, β, and γ are binary variables; the allocation of computing resources is a real number.

[0230] Embodiment 9:

[0231] The federated multi-agent Actor-Critic learning intelligent logistics task unloading and resource allocation system, the main content of which is shown in Example 2, wherein the steps of establishing the optimization model calculation framework include:

[0232] 1) Local training of Actor network parameters and Critic network parameters The steps include:

[0233] 1.1) Each logistics vehicle is modeled as a DDPG agent, including an Actor network Actor and a Critic network Critic; the input of the Actor network is the local state obtained by the LV agent observing the network environment of the logistics system, and the output is the task offloading action; the Critic network takes the local state and the selected action as input and outputs the estimated value of the current state; the Actor network parameters and Critic network parameters of the mth logistics vehicle are expressed as and

[0234] 1.2) Every t cycles, use the experience replay strategy to minimize the loss function To update the Critic network;

[0235] Loss Function As shown below:

[0236]

[0237] In the formula, represents the mini-batch sample size of the mth logistics vehicle;

[0238] Among them, the target value generated by the Critic network As shown below:

[0239]

[0240] In the formula, Q m is the action value function;

[0241] 1.3) Calculate the expected reward gradient update of the Actor network of the mth logistics vehicle Right now:

[0242]

[0243] Among them, the parameters of the Actor network Updated as follows:

[0244]

[0245] 1.4) Update Actor network parameters and Critic network parameters Right now:

[0246]

[0247]

[0248] 2) Each logistics vehicle agent will be trained locally to obtain the Actor network parameters and Critic network parameters Upload to the warehouse server to perform optimization model calculation framework aggregation;

[0249] Among them, the weight update of the global optimization model calculation framework in round j is:

[0250]

[0251] In the formula, θ j and They are the parameters of the optimization model calculation framework of the jth round and the parameters of the local optimization model calculation framework at the logistics vehicle agent m; is the total batch sample size of all logistics vehicle agents;

[0252] 3) The warehouse server distributes the aggregated global model to all logistics vehicle agents to update the local optimization model calculation framework.

[0253] Embodiment 10:

[0254] A federated multi-agent Actor-Critic learning intelligent logistics task unloading and resource allocation system, the main content of which is shown in Example 2, wherein the optimization model calculation framework includes a state space, an action space, and a reward function;

[0255] The state space stores the local state S of the logistics vehicle agent m (t), that is:

[0256] S m (t) = {S m,1+N (t),S m,B (t)} (21)

[0257] In the formula, S m,1+N(t) = {Task m (t),F m,1+N (t),I m (t),Y m,1+N (t)} represents the information set of the mth logistics vehicle and N assistant vehicles; S m,B (t) = {F B (t),Y B (t)} represents the information set of B assistant warehouses; Task m (t) represents the task of the mth logistics vehicle, F m,1+N (t) includes the computing power of the mth logistics vehicle and the computing power of N assistant vehicles; I m (t) includes the channel gain and transmission power of the mth logistics vehicle; Y m,1+N (t) includes the real-time geographic location of the mth logistics vehicle m and N assistant vehicles; F B (t) represents the computing power of the warehouse server of the assistant warehouse; Y B (t) represents the geographical location of the assistant warehouse;

[0258] The action space stores the offloading strategy of the task, namely:

[0259] A m (t) = {K m (t),F m,B (t)} (22)

[0260] In the formula, K m (t) = [K m,u (t)] determines the subtask computing node in the mth logistics vehicle; Determine the computing resources allocated by the warehouse server of the assistant warehouse to the mth logistics vehicle; K m,u (t) represents the binary operation set of each subtask when it is executed locally, unloaded to the assistant vehicle, and unloaded to the assistant warehouse; represents the computing resources allocated by the assistant warehouse to each subtask of the mth logistics vehicle;

[0261] Reward function R m (t) is as follows:

[0262]

[0263] In the formula, O m (t) is the objective function of the task offloading and resource allocation optimization model.

[0264] Embodiment 11:

[0265] The federated multi-agent Actor-Critic learning intelligent logistics task unloading and resource allocation system, the main content of which is shown in Example 2, where the logistics vehicle set is denoted as The assistant warehouse collection is recorded as The assistant vehicle set is denoted as M, B, and N represent the number of logistics vehicles, assistant warehouses, and assistant vehicles in a logistics scheduling area, respectively.

[0266] Embodiment 12:

[0267] Federated multi-agent Actor-Critic learning intelligent logistics task unloading and resource allocation system, including:

[0268] 1 Network Model

[0269] This embodiment considers the closed logistics park scenario, such as Figure 1 As shown in the figure, it consists of logistics vehicles (LVs) and logistics collaboration assistants. The logistics collaboration assistants are divided into two categories, namely helper warehouses (HWs) and helper vehicles (HVs). Both HWs and HVs can perform communication and computing operations at the same time. A group of HWs Evenly distributed on both sides, each assistant warehouse is equipped with a server to exchange information with LVs and HVs. The collection of LVs and HVs is respectively and All LVs and HVs travel along the road, and the speed of vehicle i is represented by v i LVs can generate a series of computing tasks with different requirements. We assume that each LV generates at most one task during a single time slot. Tasks can be computed locally in LVs or offloaded to HW or HV for computation.

[0270] 2. Task Model

[0271] Compared with the complete offloading application model, the partial offloading application model provides greater flexibility and can better match the smart logistics application scenarios. Generally speaking, cloud service applications can be divided into the following categories: data partitioning-oriented applications, code partitioning-oriented applications, and continuously executed applications. Since complex applications are composed of several fixed components and cannot be partitioned arbitrarily, we focus on logistics applications oriented to code partitioning. In recent years, IIoT technologies have been widely used in manufacturing systems, such as radio frequency identification (RFID), wireless, mobile, and sensor devices. This embodiment simplifies the complex application module dependency system into a linear sequence execution module, such as Figure 2As shown. Taking the emerging ARCore application model as an example, the application can be represented as a linear sequence execution architecture. The application contains multiple modules, and the current submodule depends on the output of the previous submodule. In addition, this embodiment is not limited to ARCore applications. When extended to other applications with the above-mentioned linear sequence execution, the proposed model is still applicable.

[0272] As mentioned above, this embodiment models the logistics application as a For the u-th subtask of logistics vehicle m, that is, T m,u , can be described by three variables: T m,u ={d m,u ,c m,u ,τ m,u}. Among them, d m,u Represents task T m,u The input data size, c m,u Indicates completion of task T m,u The total CPU cycles required, τ m,u Represents task T m,u As long as the data size of the first subtask is known, according to the ratio and Then we can know the data size of all subtasks. m,u It can be executed on LVm, i.e., executed locally, or offloaded to HWb / HVn for execution. Assume that task T m,u The uninstall indicator variable is K m,u ={α m,u ,β m,u,b ,γ m,u,n}, where α m,u ,β m,u ,γ m,u ∈(0,1), and α m,u +β m,u +γ m,u = 1. If task T m,u Executed locally on LVm, then α m,u =1, otherwise α m,u =0; if LVm sets task T m,u Unload to HWb for execution, then β m,u,b =1, otherwise β m,u,b =0; Similarly, if LVm sets task T m,u If it is unloaded to HVn for execution, then γ m,u,n =1, otherwise γ m,u,n =0.

[0273] 3 Delay Model

[0274] exist Figure 3 In the example of task scheduling in a certain time slot of LVm, the five subtasks of the ARCore application are scheduled to the corresponding computing nodes for calculation. Each subtask is executed after the previous subtask is completed. The execution result of the previous subtask needs to be passed to the next subtask. Therefore, task T m,u The computation offloading decision K m,u Affected by the dependencies between tasks and task T m,u Waiting for its previous subtask T m,u-1 The impact of execution time. This embodiment takes this impact into consideration and m,u Specifically, the completion delay of each task consists of two parts, namely, data transmission delay and task calculation delay.

[0275] (1) Calculation delay

[0276] Assume that LVm, HWb, and HVn are all equipped with processors with constant computing power, which can be expressed as and in, and Respectively represent LV m, HW b and HV n assigned to task T m,u Since multiple vehicles can offload tasks to the same HW or HV, are highly correlated with the idle computing resources on HW b and HV n, respectively. If vehicle m processes task T locally m,u , then calculate the delay It can be expressed as:

[0277]

[0278] When the task T m,u When offloaded to HW b, the calculation delay for:

[0279]

[0280] If vehicle m takes task T m,u When offloading to HV n for execution, the calculation delay for:

[0281]

[0282] Therefore, task T m,u The computational delay can be expressed as:

[0283]

[0284] (2) Transmission delay

[0285] In order to improve spectrum utilization, we assume that the link between the logistics vehicle and the assistant vehicle reuses the spectrum resources of the link between the logistics vehicle and the assistant warehouse. This embodiment models the calculation offloading link and expresses the path loss as X θ , where X and θ represent the distance from LV m to the offloading computing node and the path loss index, respectively. In addition, the channel fading coefficient is expressed as h m Represents, using complex Gaussian random variables for modeling. When the task T m,u When offloading from LV m to HW b / HV n, the transmission rates are:

[0286]

[0287]

[0288] Among them, w is the bandwidth allocated by the server to the logistics vehicle. Here it is assumed that the server evenly allocates bandwidth to each vehicle; is the transmission power of LV m; δ 2 is the Gaussian white noise power.

[0289] If two adjacent subtasks of LV m are calculated on the same computing node, there is no need to send the output of the previous subtask to another node, and the transmission delay is recorded as 0. Since the output data size of the computing task on the HW server is much larger than the input data size, the delay overhead of the return connection can be ignored. Assume that the transmission delay from the u-1th subtask to the uth subtask of LV m is It can be expressed as:

[0290]

[0291] The transmission delay is divided into four cases. The arrows represent the conversion of task computing nodes. m,u,m→b It means that module u-1 is processed locally and module u is offloaded to HW b. Therefore, formula (7) can be divided into the following four cases for calculation.

[0292] Case 1: Module u-1 is processed locally, and module u is executed on HW b. Its transmission delay can be expressed as:

[0293]

[0294] Case 2: Module u-1 is processed locally, and module u is executed on assistant HV n. Its transmission delay can be expressed as:

[0295]

[0296] Case 3: Module u-1 is executed on HV n, and module u is executed on LV m. The transmission delay can be expressed as:

[0297]

[0298] Case 4: Module u-1 is executed on HV n, and module u is executed on HW b. The transmission delay can be expressed as:

[0299]

[0300] Therefore, the transmission delay from the u-1th subtask to the uth subtask of LV m in formula (7) is can be re-expressed as:

[0301]

[0302] Among them, vehicle o represents the HV of the u-1th subtask of LV m.

[0303] 4 Energy consumption model:

[0304] In the logistics system, the energy consumption of LV m includes two parts: one is the energy consumption of LV m when it performs tasks locally. m,u energy consumption; second, LV m will be task T m,u Energy consumption sent to HW b or HV n.

[0305] 1) Energy consumption of local computing

[0306] The computing power of the known LV m server The energy consumption of the processor is modeled as (Joules per second), where k represents the computational energy efficiency factor associated with the processor chip of the vehicle server. Taking into account the computational latency (1), the energy consumption of local computation is given by:

[0307]

[0308] 2) Energy consumption of offloading tasks

[0309] When LV m is offloaded to task T on HW b or HV n m,u , its energy consumption mainly comes from the uplink transmission from LV m to HW b or HV n. m,u When offloading to HW b, its V2I uplink transmission energy consumption can be expressed as:

[0310]

[0311] When LV m chooses to transfer task T m,uWhen unloading to HV n, its V2V uplink transmission energy consumption can be expressed as:

[0312]

[0313] 5. Question Formulation

[0314] In the logistics system, it is necessary to optimize both system delay and energy consumption. m , this embodiment takes into account the impact of calculation delay and transmission delay. Delay target T m , which can be expressed as:

[0315]

[0316] For the energy consumption target E of LV m m , mainly considering the energy consumption of local computing tasks and the energy consumption of sending tasks during offloading. Energy consumption target E m , which can be expressed as:

[0317]

[0318] This embodiment aims to solve the trade-off between energy consumption and latency cost of code partitioned logistics application execution. In order to construct the objective function of this embodiment, a weighting factor ω is introduced m , adjust the weights of energy consumption and delay cost according to the task requirement preferences of logistics vehicle users. The goal of this embodiment is to determine task offloading decisions and resource allocation to minimize the number of logistics vehicles. The execution delay and energy consumption weighted cost of the code partitioning application. Therefore, the task offloading decision and resource allocation problem can be formulated as the following optimization problem:

[0319]

[0320] st

[0321]

[0322] C2:α m,u ,β m,u,b ,γ m,u,n ∈{0,1}

[0323]

[0324]

[0325]

[0326]

[0327] Among them, constraints C1 and C2 ensure that a subtask can only be processed on one computing node, that is, locally, or offloaded to a HW or a nearby HV; C3 ensures that the total computing resources allocated in the HW server do not exceed its computing capacity; C4 ensures that the computing resources allocated in the HV must be less than its computing capacity; C5 ensures that a subtask must be completed within its tolerable delay; C6 ensures that the total energy consumption of the LV should not exceed its own total energy. The optimization problem in formula (18) is a mixed integer nonlinear programming problem (MINLP), in which the offloading indicator variables α, β, γ are binary variables, and the allocated computing resources is a real number. In addition, in the actual logistics system, as time goes by, the scale of the logistics transportation network continues to expand, and its network topology is also changing, which leads to a significant increase in the complexity of the optimization problem in this embodiment. In addition, DRL has been recognized as an effective method for finding the optimal strategy in a complex dynamic network. Based on the above reasons, this embodiment attempts to use the DRL solution to solve the optimization problem.

[0328] 6F-MADDPG task offloading and resource allocation strategy

[0329] In this section, we first describe the multi-agent environment, mainly defining the state space, action space, and reward function of the multi-agent. Then, we give a learning framework for the F-MADDPG task offloading and resource allocation strategy, which is the core of implementing the multi-agent model training. Finally, based on this framework, we describe in detail the implementation steps of the algorithm proposed in this embodiment.

[0330] 6.1 Multi-agent Environment Description

[0331] In this section, we describe the optimization problem (18) using a Markov decision process (MDP). Then, based on the multi-user MDP formula, we use a multi-agent reinforcement learning method to solve the MDP problem. Figure 1In the smart logistics scenario shown, each LV determines its own computational offloading and resource allocation scheme based on its local environment and the total resources of each time slot. The decisions of different LVs influence each other, resulting in tight coupling of the performance of different LV areas. Therefore, it can be modeled as a multi-agent reinforcement learning problem, in which each LV acts as an agent and interacts with the environment to gain experience to improve its computational offloading and resource allocation strategies. Consider introducing a strategy approach, namely Deep Deterministic Policy Gradient (DDPG), which uses dual neural networks to estimate actions and Q values ​​respectively. In this scenario, the MADDPG algorithm is used to solve the optimization problem, which is an improvement on the DDPG algorithm to adapt to multi-agent environments. In the system described in this embodiment, the state space, action space, and reward function are defined as follows.

[0332] 1) State space:

[0333] The state is the specific and direct situation that the agent observes itself. Its setting should fully reflect the network environment of the logistics system, including the tasks, computing power, communication resources and real-time location status of logistics vehicles, as well as the computing power and real-time location of HW. The local state of agent m is defined as a set of variables, which can be expressed as:

[0334] S m (t) = {S m,1+N (t),S m,B (t)} (19)

[0335] Where S m,1+N (t) represents the information set of LVm and N HVs, S m,B (t) represents the information set of B HWs. Specifically, S m,1+N (t) = {Task m (t),F m,1+N (t),I m (t),Y m,1+N (t)}, where Task m (t) represents the task of LVm, F m,1+N (t) Including LVm computing capabilities and the computing power of N HVs I m (t) includes the channel gain and transmit power of LVm, Y m,1+N (t) Includes the real-time geographic locations of LVm and N HVs. m,B (t) = {F B (t),Y B (t)}, where F B (t) represents the computing power of the HW server Y B (t) represents the geographical location of HW.

[0336] 2) Action Space:

[0337] An action is the set of all possible actions that an agent can make. In a logistics system, an LVm agent must decide whether to perform a task locally, to which HV or HWS to offload the task, and how much computing resources to allocate for the computational task. m (t) can be expressed as:

[0338] A m (t) = {K m (t),F m,B (t)} (20)

[0339] Where K m (t) Determine the computing nodes of the subtasks in LVm, F m,B (t) Determine the computing resources that the HW server allocates to LVm. Specifically, K m (t) = [K m,u (t)], Where K m,u (t) represents the set of binary operations of each subtask executed locally, offloaded to HV, and offloaded to HW, as defined in Section 2.2. Indicates the computing resources allocated by HW to each subtask of LVm.

[0340] 3) Rewards and punishments:

[0341] Reward is a feedback that measures the success or failure of the agent's behavior in a given state. Reward setting plays a key role in training neural networks. The main goal of formula (18) is to minimize the weighted sum of system task execution delay and energy consumption, while the reward function aims to maximize the reward obtained. Therefore, the reward should be inversely proportional to the objective function. Therefore, this embodiment considers the following as the reward function:

[0342]

[0343] 6.2 Federated Learning-Assisted Multi-Agent Learning Architecture

[0344] Each LV agent takes independent actions based on its local state and user needs, and through collaborative exploration of the environment, refines its task offloading and resource allocation strategies, thereby improving the overall latency and energy consumption performance of the system. In a multi-agent environment, global information about the states and actions of all LV agents is required to train each agent. However, LV agents need to exchange local information and action values ​​with each other, which may result in a large amount of signaling overhead. The lack of training data may also pose a major challenge to the training of accurate DRL models for each agent. In order to solve this problem with low overhead, distributed federated learning can be used to improve the training performance of a single local DRL model without the need for centralized training data. In response to the above problems, this embodiment proposes a federated learning-assisted multi-agent deep deterministic policy gradient (Federated Multi-Agent Deep Deterministic Policy Gradient, F-MADDPG) framework. As Figure 4 As shown in the figure, the F-MADDPG framework mainly consists of three parts: local training, model aggregation, and model distribution. The purpose of local training is to use the DRL algorithm of MADDPG to derive local model parameters suitable for each agent. Then, each agent uploads the trained model parameters to the server, performs model aggregation, and generates a global model suitable for data training. Finally, the server distributes the aggregated global model to each agent.

[0345] 1) Local training

[0346] exist Figure 4 In the above figure, each LV is modeled as a DDPG agent, which consists of two parts: the actor network and the critic network. The input of the actor network is the local state obtained by the LV agent observing the network environment of the logistics system, and the output is the action it chooses. The critic network takes the local state and the selected action as input and outputs the estimated value of the current state. For LV agent m, its actor and critic network parameters are expressed as and In addition, to ensure the non-correlation of training data, LV agent m uses the experience replay area to store the transition unit {s m (t),a m (t),R m (t),s' m (t)}. Every time period, the Actor and Critic networks can be updated by uniformly sampling a small batch of samples from the experience replay area. Specifically, the experience replay strategy is used to minimize the loss function To update the Critic network, it is defined as:

[0347]

[0348] in, represents the mini-batch sample size of LV agent m. is the target value generated by the Critic target network and can be expressed by (23).

[0349] Action-value function Q m Taking its own local state and action as input, the Critic network can evaluate the quality of the selected action based on its local state.

[0350]

[0351] On the other hand, the expected reward gradient update of the Actor network of LV agent m is given by:

[0352]

[0353] Accordingly, the parameters of the Actor network Updated to:

[0354]

[0355] On the other hand, the parameters of the target network are slowly updated by the main network in each time period. and Updated to:

[0356]

[0357]

[0358] 2) Model aggregation

[0359] In multi-agent learning scenarios, interactive information is needed to share strategies of different agents. However, the transmission and processing of observation space data consumes too much communication and computing resources. Therefore, to overcome these difficulties, inspired by the concept of federated learning, all LV agents share their network parameters and perform joint updates. Each LV agent uploads the parameters of its local model to the server through a dedicated backhaul control link to perform model aggregation. Specifically, a small batch stochastic gradient descent method is used for joint averaging, where the global model weights of the jth round are updated as:

[0360]

[0361] where θ j and are the global model parameters of the jth round and the parameters of the local model at LV agent m, respectively. is the total batch sample size of all LV agents. The server then distributes the aggregated global model to all LV agents to update their local models accordingly.

[0362] 6.3 F-MADDPG Online Learning Algorithm

[0363] According to the proposed F-MADDPG learning framework, Algorithm 1 gives the F-MADDPG learning algorithm, which includes four steps: (1)

[0364] Lines 1 to 4 are the initialization process; lines 6 to 11 are the exploration behavior process, in which the agent chooses whether to behave randomly or follow the actor network strategy; (2) lines 12 to 17 are the network replay training process; (3) lines 18 to 20 are the periodic target network update process; (4) lines 21 to 23 are the joint update process. See Algorithm 1 for details.

[0365]

[0366] Embodiment 13:

[0367] The verification experiment of the federated multi-agent Actor-Critic learning intelligent logistics task offloading and resource allocation system includes:

[0368] The simulation environment of this experiment is built based on Python 3.7. The distribution of logistics vehicles in the smart logistics system conforms to the Poisson distribution. The specific settings of each parameter are shown in Table 1. In order to analyze the performance of the task offloading strategy of F-MADDPG proposed above, this strategy is compared with other task offloading strategies, as described below:

[0369] (1) Joint Computation Offloading and Task Migration Optimization (JCOTM): A multi-agent deep Q-network JCOTM algorithm is proposed to solve the joint optimization problem of minimizing system latency and energy consumption.

[0370] (2) Multi-agent Separate Learning (MASL): The vehicle user agent is an independent learner that does not require any collaboration. Each agent learns strategies based on its own observations and interactions with the environment. There is no model sharing or information exchange between agents.

[0371] (3) All Local Computing Strategy (ALCS): Under this strategy, all users’ computing tasks are performed on their local vehicles, that is, no computing offloading is performed. The user’s task offloading cost consists of two parts: the computing latency of executing tasks locally and the device energy consumption generated by the CPU executing tasks.

[0372] (4) Random Offloading Computing Strategy (ROCS): In this strategy, tasks are randomly executed at the MEC layer or the local computing layer.

[0373] (5) All MEC Computing Strategy (AMCS): In this strategy, vehicle users offload all tasks to the MEC server for execution.

[0374] Table 1 Simulation parameters

[0375]

[0376] Figure 5 The convergence of the F-MADDPG algorithm proposed in this example during the model training process was evaluated. The simulation setting model training process has 1000 rounds, each with 100 time steps. Figure 5 In the figure, the x-axis represents the number of training rounds, and the y-axis represents the cumulative reward during the agent training process, that is, the total cost of each LV regarding latency and energy consumption. It can be seen that after 150 rounds of iterations, the cumulative reward begins to stabilize and gradually approaches zero loss. Therefore, the training of the F-MADDPG algorithm model gradually converges, and the model is considered to be trained.

[0377] Figure 6 The cost changes corresponding to different data sizes of logistics vehicle tasks are compared. Figure 6As shown in the figure, as the amount of task input data increases, the total cost of each offloading strategy will increase. This is because the larger the task data size, the greater the computational delay, transmission delay and energy consumption. From the comparison in the figure, the total cost corresponding to the proposed F-MADDPG task offloading strategy is lower than that of other strategies, because F-MADDPG optimizes the task offloading strategy, and the optimization performance is better than JCOTM and MASL. When the task data size is small, the logistics vehicle can perform the task locally, and the cost is lower than offloading all tasks to the edge server; when the task data size is larger, most of the tasks will be offloaded to the edge server for execution, because the rich computing resources of the edge server are sufficient to perform these large data tasks. When the data size of the task increases from 150Kbit to 200Kbit, the total cost of the proposed F-MADDPG strategy increases by 27.74%, while the total costs of the ALCS, AMCS, ROCS, MASL and JCOTM strategies increase by at least 36.63%, 29.65%, 26.32%, 24.59% and 26.31%, respectively. This means that the total cost of the offloading strategy increases with the increase of network load, and the total cost of F-MADDPG always remains at a low level.

[0378] Changing trend of capacity improvement: It can be seen that as the computing power of the logistics vehicle increases, the total cost corresponding to this curve of the full MEC execution strategy remains unchanged. This is because the change in the computing power of the logistics vehicle has no effect on the process of full MEC execution of the offloading task. The total cost of the F-MADDPG, JCOTM, MASL, ROCS, and AMCS offloading strategies all decreases as the computing power of the logistics vehicle increases. For example, when the computing power of the logistics vehicle is 1.4GHz, Figure 7 The total costs are 612, 647, 675, 802, 1176 and 1242 respectively, from low to high; and when the computing power of the logistics vehicle is 1.6GHz, the total costs of the F-MADDPG, AMCS and ALCS offloading strategies to complete the task are 576, 1242 and 1080 respectively. This is because the increase in the computing power of the logistics vehicle enables the LV to have more computing resources, thereby reducing the computing resources that the logistics vehicle obtains from the MEC server, and the delay in executing tasks on the logistics vehicle is also reduced. Compared with the computing delay, the increase in the computing power of the logistics vehicle has a relatively small impact on the computing energy consumption. Therefore, the total cost of the proposed task offloading strategy shows a downward trend.

[0379] Figure 8 The figure shows the impact of the computing power of the MEC server on the total cost of the logistics system. Figure 8In the figure, the total cost of the task offloading strategy of the multi-agent DDPG proposed in this embodiment is lower than that of other offloading strategies and shows a downward trend. It can be seen that the total cost of the proposed strategy decreases with the increase of the computing power of the MEC server. This is because the size of the computing power of the MEC server can save the execution delay of the corresponding offloading task. When the computing power of the MEC server increases, the MEC server can use more computing resources to execute tasks unloaded from logistics vehicles, thereby executing the unloading tasks faster. When the computing power of the MEC server is 20GHz, the total cost of the proposed F-MADDPG strategy to complete the task is 777, and the total costs of the ALCS, AMCS, ROCS, MASL and JCOTM offloading strategies are 1432, 1222, 973, 847, and 807, respectively. Correspondingly, the total costs are reduced by 45.74%, 36.42%, 20.14%, 8.26% and 3.72%, respectively. In addition, the curve of the ALCS strategy will not change with the increase of the computing power of the MEC server, because the strategy does not use the computing power of the edge server. Fig. 9 In this paper, we compared the average system unloading costs of six unloading strategies under different numbers of vehicles. It can be seen that the system cost of each unloading strategy gradually increases with the increase in the number of logistics vehicles. Obviously, more logistics vehicles mean more tasks to be handled, which increases the time and energy costs. Fig. 9 It can be seen that compared with other offloading strategies, the proposed F-MADDPG has the lowest offloading cost, and the offloading performance of JCOTM is second. In addition, the performance of MASL is better than ROCS, and the performance of ROCS is better than AMCS and ALCS. In addition, when the number of logistics vehicles gradually increases, the offloading cost of all local calculations is always greater than the edge computing strategy, but the growth rate of AMCS is faster than ALCS, and the difference in the average system cost between the two is gradually decreasing. The reason is that when a large number of tasks are offloaded to the same MEC server, the computing resources that can be allocated to each logistics vehicle will be reduced, resulting in an increase in computing costs.

[0380] This embodiment studies the joint optimization problem of task offloading and resource allocation in a smart logistics system, in which logistics vehicles carrying delay-sensitive dependent tasks are energy-constrained. To address this problem, this patent minimizes the total system cost of latency and energy consumption while ensuring QoS requirements. In addition, a federated learning-assisted DRL framework is designed to reduce the computational complexity and signaling overhead caused by the training process. Through a large number of simulation experiments, the effectiveness of this strategy has been verified under different task input data sizes, edge server computing power, and the number of logistics vehicles. Because the road scenes and traffic conditions in real life are more complex and changeable.

Claims

1. A federated multi-agent Actor-Critic learning intelligent logistics task unloading and resource allocation system, characterized by: Including several logistics vehicles, several assistant warehouses, and several assistant vehicles; The assistant warehouse is equipped with a warehouse server; The logistics vehicles and assistant vehicles are both equipped with an on-board server; Among them, the onboard server of the mth logistics vehicle generates a task sequence and transmitted to the warehouse server of the assistant warehouse; U is the total number of tasks; The onboard server of the logistics vehicle stores a delay target calculation model, an energy consumption target calculation model, a task offloading and resource allocation optimization model, and an optimization model calculation framework; The onboard server of the logistics vehicle calculates the task sequence using the delay target calculation model and the energy consumption target calculation model respectively. The delay target T m and energy consumption target E m ; The onboard server of the logistics vehicle sends the task sequence The delay target T m and energy consumption target E m Input into the task offloading and resource allocation optimization model, and use the optimization model calculation framework to solve the task sequence Uninstall strategy; The onboard server of the logistics vehicle executes the task locally or offloads the task to the server of the assistant vehicle or assistant warehouse for execution according to the unloading strategy.

2. The federated multi-agent Actor-Critic learning intelligent logistics task offloading and resource allocation system according to claim 1 is characterized by: The task sequence is a linear sequence; where the u-th subtask T m,u ={d m,u ,c m,u ,τ m,u };d m,u Represents task T m,u The input data size, c m,u Indicates completion of task T m,u The total CPU cycles required, τ m,u Represents task T m,u Tolerance delay.

3. The federated multi-agent Actor-Critic learning intelligent logistics task unloading and resource allocation system according to claim 1 is characterized in that: The delay target T m As shown below: In the formula, is the u-th subtask T calculated by the warehouse server using the delay calculation model m,u The computational delay of For task T m,u The transmission delay from unloading at the mth logistics vehicle to the bth assistant warehouse or the nth assistant vehicle.

4. The federated multi-agent Actor-Critic learning intelligent logistics task unloading and resource allocation system according to claim 3 is characterized in that: The warehouse server calculates the u-th subtask T using the delay calculation model m,u The computational delay As shown below: In the formula, α m,u , β m,u , γ m,u is the uninstall indicator variable; α m,u =1 indicates task T m,u Executed locally on the mth logistics vehicle, α m,u =0 indicates task T m,u It is not executed locally on the mth logistics vehicle; β m,u,b =1 indicates task T m,u Unload to the bth assistant warehouse for execution; β m,u,b =0 indicates task T m,u Not executed in the b-th assistant warehouse; γ m,u,n =1 indicates task T m,u Unload to the nth assistant vehicle for execution; γ m,u,n =0 indicates task T m,u Not executed on the nth helper vehicle; Among them, task T m,u Computation latency performed locally on the mth logistics vehicle Task T m,u Computation latency of offloading to the bth assistant warehouse Task T m,u Computation latency offloaded to the nth helper vehicle They are as follows: In the formula, and They represent the mth logistics vehicle, bth assistant warehouse and nth assistant vehicle assigned to task T respectively. m,u computing resources; c m,u Indicates completion of task T m,u Total CPU cycles required.

5. The federated multi-agent Actor-Critic learning intelligent logistics task unloading and resource allocation system according to claim 3 is characterized in that: Task T m,u Transmission delay from unloading the mth logistics vehicle to the bth assistant warehouse or the nth assistant vehicle As shown below: Wherein, vehicle o represents the assistant vehicle that performs the u-1th subtask of the mth logistics vehicle; α m,u , γ m,u-1,n , γ m,u,o is the uninstall indicator variable; d m,u Represents task T m,u The input data size; β m,u,b =1 indicates task T m,u Unload to the bth assistant warehouse for execution; β m,u,b =0 indicates task T m,u Not executed in the b-th assistant warehouse; γ m,u,n =1 indicates task T m,u Unload to the nth assistant vehicle for execution; γ m,u,n =0 indicates task T m,u Not executed on the nth helper vehicle; Among them, task T m,u Transfer rate from unloading of the mth logistics vehicle to the bth assistant warehouse Task T m,u Transfer rate from unloading of the mth logistics vehicle to the nth assistant vehicle They are as follows: Where w is the bandwidth of the logistics vehicle; is the transmission power of LVm; 2 is the Gaussian white noise power; Represents task T m,u Path loss from unloading at the mth logistics vehicle to the bth assistant warehouse; Represents task T m,u The path loss from unloading the mth logistics vehicle to the nth assistant vehicle; h m is the channel fading coefficient.

6. The federated multi-agent Actor-Critic learning intelligent logistics task unloading and resource allocation system according to claim 1 is characterized in that: Energy consumption target E m As shown below: In the formula, α m,u , β m,u , γ m,u is the uninstall indicator variable; Among them, local computing energy consumption Task T m,u Energy consumption from unloading from the mth logistics vehicle to the bth assistant warehouse Task T m,u Energy consumption from unloading from the mth logistics vehicle to the nth assistant vehicle They are as follows: Where k represents the computational energy efficiency coefficient associated with the processor chip of the vehicle server; Indicates that the mth logistics vehicle is assigned to task T m,u computing resources; Represents task T m,u The computational latency executed locally on the mth logistics vehicle; c m,u Indicates completion of task T m,u Total CPU cycles required; is the transmit power of LV m; For task T m,u The transfer rate from the mth logistics vehicle to the bth assistant warehouse; For task T m,u Transfer rate from unloading at the mth logistics vehicle to the nth helper vehicle.

7. The federated multi-agent Actor-Critic learning intelligent logistics task unloading and resource allocation system according to claim 1 is characterized in that: The task offloading and resource allocation optimization model min O is as follows: In the formula, the unloading indicator variables α, β, and γ are binary variables; the allocation of computing resources is a real number; Respectively represent the computing power of the assistant warehouse server and the computing power of the assistant vehicle server; α m,u , β m,u,b , γ m,u,n is the unloading indicator variable; ω m is the weighting factor; is the upper limit of energy consumption; τ m,u The upper limit of the tolerable delay; They represent assistant warehouse set, logistics vehicle set, assistant vehicle set, and task set respectively; is the u-th subtask T calculated by the warehouse server using the delay calculation model m,u The computational delay of For task T m,u The transmission delay from unloading at the mth logistics vehicle to the bth assistant warehouse or the nth assistant vehicle.

8. The federated multi-agent Actor-Critic learning intelligent logistics task unloading and resource allocation system according to claim 1 is characterized in that: The steps to establish the optimization model calculation framework include: 1) Local training of Actor network parameters and Critic network parameters The steps include: 1.1) Each logistics vehicle is modeled as a DDPG agent, including an Actor network Actor and a Critic network Critic; the input of the Actor network is the local state obtained by the LV agent observing the network environment of the logistics system, and the output is the task offloading action; the Critic network takes the local state and the selected action as input and outputs the estimated value of the current state; the Actor network parameters and Critic network parameters of the mth logistics vehicle are expressed as and 1.2) Every t cycles, use the experience replay strategy to minimize the loss function To update the Critic network; Loss Function As shown below: In the formula, represents the mini-batch sample size of the mth logistics vehicle; Indicates local state and task execution actions; Q m is the action value; Among them, the target value generated by the Critic network As shown below: In the formula, Q' m is the action value; Represents local state and task execution actions; represents the reward; γ is the calculation coefficient; 1.3) Calculate the expected reward gradient update of the Actor network of the mth logistics vehicle Right now: Where D m is the number of batch samples; is the gradient; is the action value; Represents local state and task execution actions; is the Critic network parameter; Perform actions for tasks; Among them, the parameters of the Actor network Updated as follows: Where δ is the update coefficient; 1.4) Update Actor network parameters and Critic network parameters Right now: In the formula, τ is the weight parameter; 2) Each logistics vehicle agent will be trained locally to obtain the Actor network parameters and Critic network parameters Upload to the warehouse server to perform optimization model calculation framework aggregation; Among them, the weight update of the global optimization model calculation framework in round j is: In the formula, θ j and They are the parameters of the optimization model calculation framework of the jth round and the parameters of the local optimization model calculation framework at the logistics vehicle agent m; is the total batch sample size of all logistics vehicle agents; 3) The warehouse server distributes the aggregated global model to all logistics vehicle agents to update the local optimization model calculation framework.

9. The federated multi-agent Actor-Critic learning intelligent logistics task unloading and resource allocation system according to claim 1 is characterized in that: The optimization model calculation framework includes a state space, an action space and a reward function; The state space stores the local state S of the logistics vehicle agent m (t), that is: S m (t)={S m,1+N (t),S m,B (t)} (21) In the formula, S m,1+N (t) = {Task m (t),F m,1+N (t),I m (t),Y m,1+N (t)} represents the information set of the mth logistics vehicle and N assistant vehicles; S m,B (t) = {F B (t),Y B (t)} represents the information set of B assistant warehouses; Task m (t) represents the task of the mth logistics vehicle, F m,1+N (t) includes the computing power of the mth logistics vehicle and the computing power of N assistant vehicles I m (t) includes the channel gain and transmission power of the mth logistics vehicle; Y m,1+N (t) includes the real-time geographic location of the mth logistics vehicle m and N assistant vehicles; F B (t) represents the computing power of the warehouse server of the assistant warehouse Y B (t) represents the geographical location of the assistant warehouse; The action space stores the task offloading strategy A m (t), that is: A m (t)={K m (t),F m,B (t)} (22) In the formula, K m (t) = [K m,u (t)] determines the subtask computing node in the mth logistics vehicle; represents the computing resources allocated by the warehouse server of the assistant warehouse to the mth logistics vehicle; K m,u (t) represents the binary operation set of each subtask when it is executed locally, unloaded to the assistant vehicle, and unloaded to the assistant warehouse; Indicates that the b-th assistant warehouse is assigned to task T m,u computing resources; Reward function R m (t) is as follows: In the formula, O m (t) is the objective function of the task offloading and resource allocation optimization model.

10. A computer readable medium, characterized in that: The computer-readable medium stores a computer program of the federated multi-agent Actor-Critic learning intelligent logistics task unloading and resource allocation system according to any one of claims 1 to 9; The computer program is used to generate a federated multi-agent Actor-Critic learning smart logistics task unloading and resource allocation solution; When the computer program is executed by a processor, the following steps are implemented: 1) The onboard server of the mth logistics vehicle generates a task sequence And transmit to the warehouse server of the assistant warehouse; U is the total number of tasks; 2) The onboard server of the logistics vehicle calculates the task sequence using the delay target calculation model and the energy consumption target calculation model respectively The delay target T m and energy consumption target E m ; 3) The onboard server of the logistics vehicle sends the task sequence The delay target T m and energy consumption target E m Input into the task offloading and resource allocation optimization model, and use the optimization model calculation framework to solve the task sequence Uninstall strategy; 4) The onboard server of the logistics vehicle executes the task locally or offloads the task to the server of the assistant vehicle or assistant warehouse for execution according to the offloading strategy.

Images