Multi-unmanned aerial vehicle-truck collaborative distribution path optimization method, system and equipment and medium

Abstract

The invention discloses a multi-unmanned aerial vehicle-truck cooperative distribution path optimization method, system, device and medium, and the method comprises the steps: obtaining the input feature phi truck, t of a superior DQN and the input feature phi drone, t of a lower MADQN when a cooperative distribution task request is received; and inputting the input feature [phi] truck, t of the upper-level DQN and the input feature [phi] drone, t of the lower-level MADQN into the trained hierarchical MADQN model to obtain an optimal distribution path of multi-unmanned aerial vehicle-truck collaboration, and the method, system, device and medium can optimize the path in unmanned aerial vehicle-truck collaboration distribution.

Description

Technical Field

[0001] The present invention belongs to the technical field of collaborative distribution logistics, and relates to a method, system, device and medium for optimizing the collaborative distribution path of multiple unmanned aerial vehicles - trucks. Background Art

[0002] The collaborative distribution of unmanned aerial vehicles and trucks (MDTCD), as an innovative model in smart city logistics distribution, provides a new idea for solving the "last mile" problem of modern logistics by combining the flexibility of unmanned aerial vehicles with the load - carrying and endurance advantages of trucks. Unmanned aerial vehicles can efficiently complete short - distance distribution tasks, while trucks serve as long - distance transportation tools and mobile supply stations for unmanned aerial vehicles. The synergy between the two significantly improves the distribution efficiency, optimizes the path planning, and reduces energy consumption and costs. With the increasing complexity of urban distribution requirements and the continuous improvement of environmental protection requirements, the importance of the collaborative distribution model in urban logistics has gradually emerged. How to effectively coordinate the distribution tasks of unmanned aerial vehicles and trucks, optimize the path planning, and improve the system efficiency has become the focus of research.

[0003] However, existing research still has deficiencies in the collaborative path optimization of multiple unmanned aerial vehicles, task allocation, and environmental adaptability. Existing path planning methods mainly rely on traditional heuristic algorithms and exact optimization algorithms. For example, Schermer (2018) et al. proposed a path planning method based on tabu search, and Yurek and Ozmutlu (2018) studied an iterative heuristic algorithm based on decomposition to optimize the unmanned aerial vehicle path. These methods perform well in small - scale static environments, but their adaptability and efficiency significantly decline when facing complex large - scale scenarios. In addition, most existing research focuses on the optimization of a single unmanned aerial vehicle's single task or truck path planning. Although some work has explored the possibility of multi - agent collaboration, it has not fully considered issues such as information sharing and path conflicts among multiple unmanned aerial vehicles. In the collaborative distribution problem involving multiple distribution units (such as unmanned aerial vehicles and trucks), the behaviors of each intelligent agent affect each other, and traditional algorithms may not be able to efficiently coordinate the behaviors of multiple intelligent agents, thus limiting the improvement of the overall system efficiency.

[0004] With the continuous development of the application of reinforcement learning in optimization problems, related research has begun to explore how to use reinforcement learning to solve the path optimization problem in the collaborative distribution of drones and trucks. Reinforcement learning has shown significant advantages especially in dynamic and complex environments. For example, Vinyals (2015) et al. proposed a path planning algorithm based on pointer networks to solve the TSP problem. Nazari (2018) et al. used the policy gradient algorithm to solve the VRP problem with multiple customers. However, these methods mostly focus on single-agent path optimization scenarios and have limited support for the complex problems of multi-drone and truck collaboration. On the other hand, multi-agent reinforcement learning (MARL) provides a new idea for solving the problem of multi-agent collaborative distribution. By coordinating the collaborative behaviors of drones and trucks, the efficiency and flexibility of the system are improved. For example, Sykora (2021) et al. proposed a multi-agent path planning method based on graph neural networks to solve the dynamic path conflict problem in complex environments, but the multi-task collaboration and resource optimization problems in the collaborative distribution of drones and trucks have not been fully solved. To sum up, there is still a large research space for the MDTCD path planning problem in terms of multi-task and multi-agent collaboration at the present stage. Summary of the Invention

[0005] The purpose of the present invention is to overcome the above-mentioned disadvantages of the prior art and provide a multi-drone-truck collaborative distribution path optimization method, system, device and medium, which can optimize the path in the collaborative distribution of drones and trucks.

[0006] To achieve the above purpose, the present invention discloses a multi-drone-truck collaborative distribution path optimization method, including:

[0007] When receiving a collaborative distribution task request, obtain the input feature φ of the upper-level DQN truck,t and the input feature φ of the lower-level MADQN drone,t ;

[0008] Input the input feature φ of the upper-level DQN truck,t and the input feature φ of the lower-level MADQN drone,t into the trained hierarchical MADQN model to obtain the optimal distribution path for multi-drone-truck collaboration.

[0009] Further, the input feature φ of the upper-level DQN truck,t is:

[0010]

[0011] Among them, represents the position information of the current truck, represents the current truck task status, For the overall revenue of the current upper-layer truck module, is the set of visited stop points.

[0012] Furthermore, the input feature φ of the lower-layer MADQN drone,t is:

[0013]

[0014] where represents the position information of the current drone, is 0 or 10, represents the current task status of the drone, represents the total travel distance of the drone at the current moment, represents the load capacity of the drone at the current moment, is the battery energy consumption of the drone, is the overall revenue of the current drone, represents the status information of other drones that each drone can observe.

[0015] Furthermore, the hierarchical MADQN model includes an upper-layer DQN and a lower-layer MADQN.

[0016] Furthermore, the reward function corresponding to the upper-layer DQN in the training process of the hierarchical MADQN model is:

[0017] r truck = w1r tpath + w2r visit

[0018] where r tpath is the shortest distance reward, r visit is the successful visit reward for the stop point, and w1 and w2 represent weight coefficients;

[0019] The reward function corresponding to the lower-layer MADQN in the training process of the hierarchical MADQN model is:

[0020] r drone = r1r dpath + r2r task + r3r energy + r4r crash + r5r all

[0021] where r dpath is the shortest distance reward for the drone to execute the delivery task, r task is the task completion reward, r energy is the drone energy consumption reward, r crash represents the penalty term when the distance between drones is less than the safety threshold, r allFor the overall additional reward after visiting all customer points in the stop area, r1, r2, r3, r4, and r5 represent weight coefficients.

[0022] Furthermore, the hierarchical MADQN model is trained using the gradient descent method.

[0023] Furthermore, the implementation strategy π of the upper-level DQN truck is:

[0024] π truck :(φ truck ; θ truck ) → k i

[0025] where θ truck represents the model parameters of the upper-level DQN, and k i ∈C represents the temporary target stop point selected by the truck, and C represents the set of temporary target stop points;

[0026] The control strategy π of the lower-level MADQN drone is:

[0027] π drone :(φ drone ; θ drone ) → a u

[0028] where θ drone represents the model parameters of the lower-level MADQN, and a u ∈PR u represents the selected path planning strategy, and PR u represents the set of planned paths of the UAV.

[0029] The present invention discloses a multi-UAV-truck collaborative distribution path optimization system, including:

[0030] An acquisition module, configured to acquire the input feature φ of the upper-level DQN when receiving a collaborative distribution task request truck,t and the input feature φ of the lower-level MADQN drone,t ;

[0031] A prediction module, configured to input the input feature φ of the upper-level DQN truck,t and the input feature φ of the lower-level MADQN drone,t into the trained hierarchical MADQN model to obtain the optimal distribution path for multi-UAV-truck collaboration.

[0032] The present invention discloses a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the multi-UAV-truck collaborative delivery path optimization method are implemented.

[0033] The present invention discloses a computer-readable storage medium storing a computer program, and when the computer program is executed by a processor, the steps of the multi-UAV-truck collaborative delivery path optimization method are implemented.

[0034] The present invention has the following beneficial effects:

[0035] When the multi-UAV-truck collaborative delivery path optimization method, system, device, and medium of the present invention are specifically operated, the input feature φ of the upper-level DQN truck,t and the input feature φ of the lower-level MADQN drone,t are input into the trained hierarchical MADQN model to obtain the optimal delivery path for multi-UAV-truck collaboration. Among them, considering both UAV delivery and truck delivery, through a hierarchical multi-agent reinforcement learning framework, combining global path planning and local task allocation, it not only optimizes the path planning of the truck as a whole but also efficiently schedules the multi-UAV collaborative tasks locally, overcoming the deficiency in the prior art that only focuses on a single optimization level.

[0036] Furthermore, through the experience replay mechanism in reinforcement learning, historical data can be fully utilized to improve the model performance and at the same time reduce the algorithm complexity, solving the problem of high computational complexity of traditional algorithms in large-scale problems. In addition, fully considering the resource competition and path conflict problems in multi-UAV collaborative delivery, through state sharing and collaborative scheduling strategies, the collaboration efficiency among multi-UAVs is optimized, effectively overcoming the defect of insufficient multi-agent scheduling ability in the prior art. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] The accompanying drawings forming a part of the present invention are used to provide a further understanding of the present invention. The schematic embodiments and descriptions thereof of the present invention are used to explain the present invention and do not constitute an improper limitation to the present invention. In the drawings:

[0038] Figure 1 is a flowchart of the method of the present invention;

[0039] Figure 2 is a schematic diagram of the collaborative delivery scenario used in the present invention;

[0040] Figure 3 is a double-layer path planning framework diagram based on hierarchical multi-agent reinforcement learning of the present invention;

[0041] Figure 4This is the overall framework diagram of the multi-UAV truck collaborative distribution model based on hierarchical multi-agent reinforcement learning of the present invention;

[0042] Figure 5 This is the schematic diagram of the path planning algorithm strategy of the present invention;

[0043] Figure 6 This is the schematic diagram of the hierarchical MADQN training algorithm of the present invention;

[0044] Figure 7 This is the specific operation schematic diagram of the path planning algorithm based on hierarchical MADQN of the present invention. Detailed implementation manners

[0045] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all of the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0046] In the description of the present invention, it should be understood that the terms "include" and "comprise" indicate the presence of the described features, wholes, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or their combinations.

[0047] It should also be understood that the terms used in the specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the specification of the present invention and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include the plural forms.

[0048] It should also be further understood that the term " / and" used in the specification of the present invention and the appended claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes these combinations. For example, A and / or B can represent: the case of A alone, the case of A and B existing simultaneously, and the case of B alone. In addition, the character " / " in the present invention generally represents an "or" relationship between the front and rear associated objects.

[0049] It should be understood that although the terms first, second, third, etc. may be used in the embodiments of the present invention to describe preset ranges, etc., these preset ranges should not be limited to these terms. These terms are only used to distinguish the preset ranges from each other. For example, without departing from the scope of the embodiments of the present invention, the first preset range may also be referred to as the second preset range, and similarly, the second preset range may also be referred to as the first preset range.

[0050] Depending on the context, as used herein, the word "if" can be interpreted as "when" or "while" or "in response to determining" or "in response to detecting". Similarly, depending on the context, the phrase "if determined" or "if detecting (stated condition or event)" can be interpreted as "when determined" or "in response to determining" or "when detecting (stated condition or event)" or "in response to detecting (stated condition or event)".

[0051] To make the objectives, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are some, but not all, of the embodiments of the present invention. Generally, the components described and shown in the accompanying drawings herein can be arranged and designed in a variety of different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the claimed invention, but merely represents selected embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0052] Schematic diagrams of various structures according to the disclosed embodiments of the present invention are shown in the drawings. These figures are not drawn to scale, where for the purpose of clear expression, some details are enlarged and some details may be omitted. The shapes of various regions and layers shown in the figures and their relative sizes and positional relationships are merely exemplary, and may actually deviate due to manufacturing tolerances or technical limitations, and those skilled in the art can additionally design regions / layers with different shapes, sizes and relative positions according to actual needs.

[0053] Embodiment 1

[0054] Refer to Figure 1 , the multi-UAV-truck collaborative distribution path optimization method of the present invention includes the following steps:

[0055] 1) Determine network parameters and generate training and test data;

[0056] Refer to Figure 2 , the operation process of step 1) is:

[0057] 1a) Determine the parameters of the multi-UAV-truck collaborative distribution task, and the parameters of the multi-UAV-truck collaborative distribution task are: {0} / {m + 1} represents the warehouse location, {0} represents the starting point, and {m + 1} represents the ending point; C = {c1, c2,..., c m ,..., c M} represents the set of temporary stopping points of the truck, cm denotes the \(m\)-th temporary stop, and \(M\) denotes the number of temporary stops; \(C_0=\{0\}\cup C\) represents the set of nodes from which the truck can depart; \(C_1 = \{m + 1\}\cup C\) represents the set of nodes that the truck can reach; \(P=\{p_1,p_2,\cdots,p k ,\cdots,p K \}\) represents the set of customer points, and \(p k \) represents the \(k\)-th type of customer point, and \(K\) represents the number of types of customer points; \(V=\{v_1,v_2,\cdots,v u ,\cdots,v U \}\) represents the set of drones, and \(v u \) represents the \(u\)-th drone, and \(U\) represents the number of drones; \(q i \) represents the demand of the \(i\)-th customer point; \(Q\) represents the maximum load capacity of the drone; \(d ij \) represents the distance between node \(i\) and node \(j\); \(E battery \) represents the maximum energy consumption of the drone; \(L\) represents the maximum single flight distance of the drone; \(E truck \), \(E drone \) represent the unit travel costs of the truck and the drone respectively;

[0058] 1b) Construct a path planning task model for truck-multi-drone collaborative distribution;

[0059] Model the path planning task as a joint representation of truck path planning and multi-drone path planning. The truck path planning problem can be expressed as:

[0060]

[0061] where \(x ij \) is a decision variable. When the truck travels from node \(i\) to node \(j\), \(x ij = 1\); otherwise, \(x ij = 0\). \(Z\) is the total travel distance of the truck. Usually, the objective is to minimize the total distance to solve the shortest path. The multi-drone path planning problem can be modeled as a capacitated vehicle routing problem, that is:

[0062]

[0063] where the decision variable \) represents that when the drone \(v\) travels from node \(i\) to node \(j\), \(x ijk = 1\); otherwise, \(x ijk = 0\). The entire collaborative distribution problem is jointly represented, and the optimization objective is to minimize the total travel cost of collaborative distribution, that is:

[0064] minF a = E truck \cdot Z+E drone \cdot Y

[0065] 1c) Generate training data;

[0066] Randomly generate 100 sets of delivery task sets to train the hierarchical MADQN model. The number of truck stop points (cities) in each task set is 10, the number of customers is 60, the number of trucks is 1, and the number of drones is defaulted to 3. Among them, 70 sets are used as training samples and 30 sets are used as test samples. The number of training rounds is 5000.

[0067] 2) Build a two-layer path planning framework;

[0068] Reference Figure 3 , the specific process of step 2) is as follows:

[0069] 2a) Build a hierarchical path planning framework. The hierarchical path planning framework includes an upper-layer truck path planning module and a lower-layer multi-drone path planning module. Among them, the upper-layer controller is implemented by a DQN, and the lower-layer controller is implemented by a MADQN. The specific principle is:

[0070] 2a1) The implementation strategy of the upper-layer truck path planning module is represented by π truck as:

[0071] π truck :(φ truck ; θ truck )→k i

[0072] where φ truck represents the input features of the upper-layer truck path planning module; θ truck represents the parameters of the upper-layer truck DQN model; c i ∈C represents the temporary target stop point selected by the truck, and C represents the set of temporary target stop points.

[0073] 2a2) The control strategy of the lower-layer drone path planning module is represented by π drone as:

[0074] π drone :(φ drone ; θ drone )→a u

[0075] where φ drone represents the input features of the lower-layer multi-drone path planning module; θ drone represents the parameters of the lower-layer multi-drone path planning model, and a u ∈PR u represents the selected path planning strategy, and PR u represents the set of planned paths of the drones;

[0076] 2b) Determine the input feature s of the hierarchical MADQN model t =(φ truck , φ drone ), where φ truck is the input feature of the upper-level truck path planning module, and φ drone is the input feature of the lower-level multi-UAV path planning module. The input feature φ truck of the upper-level truck path planning module is as follows:

[0077]

[0078] Among them, represents the position information of the current truck; represents the current truck task status. When , it means waiting for the UAV at the docking point, and when , it means going to the next docking point; is the overall revenue of the current upper-level truck module; is the set of visited docking points. The input feature φ drone of the lower-level multi-UAV path planning module is as follows:

[0079]

[0080] Among them, represents the position information of the current UAV, is fixed at 0 or 10; represents the current task status of the UAV. When , it means waiting on the truck, and when , it means the UAV is performing a delivery task; represents the total travel distance of the UAV at the current moment; represents the payload of the UAV at the current moment; is the battery energy consumption of the UAV; is the overall revenue of the current UAV; φ' drone,t represents the status information of other UAVs that each UAV can observe, which is used to coordinate task allocation and avoid conflicts.

[0081] 2c) Determine the temporary target docking point;

[0082] Cluster the customer points through the K-means clustering algorithm to generate a set of candidate docking points; the goal of the upper-level DQN module is to maximize its Q-value function, and select the truck docking point that maximizes the Q-value as the current temporary target docking point;

[0083] 2d) Determine the feasible path planning range;

[0084] Refer to Figure 4, the specific process of step 2d) is as follows:

[0085] Let PR u represent the set of feasible planning ranges of paths where represents the h-th feasible path of the u-th UAV. For any it needs to satisfy the following conditions:

[0086] 2d1) where U represents the number of UAVs and N is the number of customer points;

[0087] 2d2) where E total is the total energy consumption of each path. Among them, P base is the basic energy consumption of the UAV without load, and E battery is the battery capacity of the UAV;

[0088] 2d3) ||L u (t) - L v (t)|| ≥ d safe where d safe represents the minimum safe space distance between the u-th and v-th UAVs at time t;

[0089] 2d4) means that each customer point i can be visited by only one UAV u;

[0090] 2d5) represents the starting point and the ending point of each path must be the truck stop point k l ∈ C.

[0091] 2e) Determine the path planning strategy;

[0092] Referring to Figure 5 , the specific process of step 2e) is as follows:

[0093] 2e1) Design a planning strategy with the goal of maximizing the total reward function of the current distribution task:

[0094] 2e1-1) Initialize PR u ;

[0095] 2e1-2) When then go to step 2e1-3), otherwise go to step 2e1-6);

[0096] 2e1-3) Calculate the set C of temporary target stop points within the feasible path planning range i (t);

[0097] 2e1 - 4) In the set C of temporary target docking points i (t), select the one that satisfies Truck temporary docking point;

[0098] 2e1 - 5) Select the customer points that satisfy for delivery;

[0099] 2e1 - 6) The path planning task fails.

[0100] 3) Train the hierarchical MADQN model;

[0101] Refer to Figure 6 , the specific process of step 3) is as follows:

[0102] 3a) Determine the reward function;

[0103] The process of step 3a) is as follows:

[0104] 3a1) Based on the truck state characteristics, customer point locations, and demand characteristics, the upper - layer DQN makes decisions on temporary target docking points, and design the reward function r truck as:

[0105] r truck = w1r tpath + w2r visit

[0106] where r tpath is the shortest - distance reward, which is used to encourage the upper - layer truck to select a shorter access path; r visit is the successful - access reward for the docking point, which encourages the truck to complete the task more effectively.

[0107] 3a2) According to the characteristics of the lower - layer MADQN making decisions based on the upper - layer truck's temporary target docking points, design the reward function r drone as:

[0108] r drone = r1r dpath + r2r task + r3r energy + r4r crash + r5r all

[0109] where r dpath is the shortest - distance reward for the drone to execute the delivery task, which encourages each drone to reduce the flight path; r task is the task - completion reward, which encourages the drone to complete the delivery task as soon as possible and avoid task failure; r energy is the drone energy - consumption reward, which is used to encourage the drone to select a path with lower energy consumption; r crashA penalty term indicating that the distance between drones is less than the safety threshold, which is used to reduce the probability of multiple drones colliding; r all is the overall additional reward after visiting all customer points in the area of the docking point, encouraging the drone to complete all tasks within the area.

[0110] 3b) Train the hierarchical MADQN;

[0111] The specific process of step 3b) is as follows:

[0112] 3b1) Initialize the upper and lower DQN parameters θ truck 、θ drone and the data buffer D truck and D drone ;

[0113] 3b2) Let Define the update frequencies M1 and M2 to control the sampling of the experience replay pool and the update frequency of the target network;

[0114] 3b3) Initialize the number of training episodes episode and the time step t, set episode = 1, t = 1, and done = False;

[0115] 3b4) Set the total number of training episodes L. When episode < L and done = False, start a new round of training;

[0116] 3b5) Initialize the network feature s0 and determine the input feature φ of the upper DQN truck,t , φ truck,t includes:

[0117] The geographical coordinates of the current truck are initialized to the warehouse coordinates [2, 2];

[0118] The current truck task status indicates the current truck task status and is initialized to 1;

[0119] The total system revenue of the upper module in the time interval [0, t] is initialized to 0;

[0120] The set of visited docking points

[0121] 3b6) Initialize the temporary target docking point k of the truck i = ε - greedy(φ truck,t ; θ truck );

[0122] 3b7) Let r truck,t = 0, r drone,t = 0;

[0123] 3b8) Determine the truck action with k t = ε-greedy(φ truck,t ; θ truck );

[0124] 3b9) Execute the truck action k t , observe and calculate the immediate reward r truck,t , update the truck state and obtain the input feature φ of the upper Q network truck,t ;

[0125] 3b10) Store the upper-level truck experience sample (φ truck,t , k t , r truck,t , φ truck,t+1 ) into the experience pool D truck ;

[0126] 3b11) When done = False, go to step 3b12), otherwise, go to step (3b25);

[0127] 3b12) Determine the current network feature s t , and determine the input features φ drone,t , φ drone,t of the lower-level MADQN as:

[0128]

[0129] where, is the position information of the current drone, initialized to [2, 2, 0], indicating that it is initially on the truck; is the current task state of drone u, initialized to 0; is the total travel distance of drone u at the current time, initialized to 0; is the payload z of the drone at the current time; is the battery energy consumption of drone u at the current time; is the total system revenue of the upper-level module in the time interval [0, t], initialized to 0; φ' drone,t represents the state information of other drones that each drone can observe, which is used to coordinate task allocation and avoid conflicts;

[0130] The battery energy consumption of drone u at the current time is:

[0131]

[0132] where, P base is the basic energy consumption (watts) of the drone without load, k is the power consumption coefficient (watts / kg), d uj$d_j$ is the distance for the UAV to fly to the $j$-th customer, and $v$ is the flight speed of the UAV.

[0133] The payload of the UAV at the current moment is:

[0134]

[0135] where represents the total customer demand to be served by the UAV $u$ at the $k$-th type of customer points, and $q$ m , $m\in P$ k is the demand of the customer point $m$ of the $k$-th type of customers, and $q$ j , $j\in P$ k represents the demand of the served customer point $j$;

[0136] 3b13) Use the $\epsilon$-greedy strategy to determine the UAV action $a$ u $=( \varphi$ drone,t ; $\theta$ drone );

[0137] 3b14) Execute the UAV action $a$ u , observe and calculate the reward $r$ obtained by the UAV drone,t ;

[0138] 3b15) Update the state of the UAV $u$ and extract the next feature $\varphi$ of the UAV drone,t+1 ;

[0139] 3b16) Store the experience sample ($\varphi$ drone,t , $a$ u , $r$ drone,t , $\varphi$ drone,t+1 ) of the UAV into $D$ drone ;

[0140] 3b17) When the data volume of $D$ truck is a multiple of $M1$, randomly draw a sample ($\varphi$ truck , $k$ truck,t , $r$ t , $\varphi$ truck,t ) from the experience pool $D$ truck,t+1 and go to step 3b18), otherwise, go to step 3b20);

[0141] 3b18) Calculate the target value $y$ truck , where $y$ truck $=r$ truck,t $+\gamma\max$ k′ $Q'$ truck $(\varphi$ truck,t , $k'$ t ; $\theta'$ truck );

[0142] 3b19) With respect to y truck Perform gradient descent to update the Q-network parameters θ of the upper-layer truck truck ;

[0143] 3b20) When D drone When the data volume is M2, randomly select some samples (φ drone,t , a u , r drone,t , φ drone,t+1 ) and go to step 3b21), otherwise, go to step 3b23);

[0144] 3b21) Calculate the target value y drone , where y drone = r drone,t + γ max a' Q' drone (φ drone,t , a′ t ; θ' drone );

[0145] 3b22) Perform gradient descent on y drone to update the Q-network parameters θ of the lower-layer UAV drone ;

[0146] 3b23) Update the target network parameters every C steps of training, that is, θ' truck = θ truck , θ' drone = θ drone ;

[0147] 3b24) t = t + 1, if all customers in the k area are served, set done = True. When done = False, go to step 3b11), otherwise, go to step 3b25);

[0148] 3b25) Set episode = episode + 1. If episode <= L, go to step 3b5). If episode > L, the training ends.

[0149] 4) Complete the collaborative distribution task based on the hierarchical MADQN model;

[0150] The process of step 4) is as follows:

[0151] 4a) Initialize the parameters of the upper and lower layer DQN models, and set the number of customers P and the number of docking points M;

[0152] 4b) Receive a collaborative distribution task request;

[0153] 4c) Determine the current state s tand the input feature φ of the upper-level DQN truck,t ;

[0154] 4d) Determine the temporary target stopping point of the truck

[0155] 4e) Determine the input feature φ of the lower-level MADQN drone,t ;

[0156] 4f) Determine the

[0157] 4g) Execute the delivery task for the customer points corresponding to a*;

[0158] 4h) Let p = p + 1. When p < P, go to step 4e). Otherwise, go to step 4i);

[0159] 4i) Let m = m + 1. When m < M, go to step 4c). Otherwise, the collaborative delivery work is completed.

[0160] Embodiment 2

[0161] The multi-UAV-truck collaborative delivery path optimization system described in the present invention includes:

[0162] An acquisition module, configured to acquire the input feature φ of the upper-level DQN truck,t and the input feature φ of the lower-level MADQN drone,t ;

[0163] A prediction module, configured to input the input feature φ of the upper-level DQN truck,t and the input feature φ of the lower-level MADQN drone,t into the trained hierarchical MADQN model to obtain the optimal delivery path for multi-UAV-truck collaboration.

[0164] In the embodiments of the present application, the division of modules is illustrative, merely a logical function division. In actual implementation, there may be other division methods. In addition, in each embodiment of the present application, each functional module may be integrated in a processor, may also exist separately physically, or two or more modules may be integrated in one module. The above integrated modules may be implemented in the form of hardware or in the form of software functional modules.

[0165] Embodiment 3

[0166] A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the multi-UAV-truck collaborative delivery path optimization method are implemented. For example, it includes: when a collaborative delivery task request is received, the input feature φ of the upper-level DQN is obtained truck,t and the input feature φ of the lower-level MADQN drone,t ; the input feature φ of the upper-level DQN truck,t and the input feature φ of the lower-level MADQN drone,t are input into the trained hierarchical MADQN model to obtain the optimal delivery path for multi-UAV-truck collaboration. Among them, the memory may include internal memory, such as high-speed random access memory, and may also include non-volatile memory, such as at least one disk memory, etc.; the processor, network interface, and memory are interconnected through an internal bus, which can be an Industry Standard Architecture bus, a Peripheral Component Interconnect standard bus, an Extended Industry Standard Architecture bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. The memory is used to store programs. Specifically, the program can include program code, and the program code includes computer operation instructions. The memory can include internal memory and non-volatile memory and provide instructions and data to the processor.

[0167] Example 4

[0168] A computer-readable storage medium stores a computer program. When the computer program is executed by a processor, the steps of the multi-UAV-truck collaborative delivery path optimization method are implemented. For example, it includes: when a collaborative delivery task request is received, the input feature φ of the upper-level DQN is obtained truck,t and the input feature φ of the lower-level MADQN drone,t ; the input feature φ of the upper-level DQN truck,t and the input feature φ of the lower-level MADQN drone,t are input into the trained hierarchical MADQN model to obtain the optimal delivery path for multi-UAV-truck collaboration. Specifically, the computer-readable storage medium includes, but is not limited to, for example, volatile memory and / or non-volatile memory. The volatile memory can include random access memory and / or cache memory, etc. The non-volatile memory can include read-only memory, hard disk, flash memory, optical disc, magnetic disk, etc.

[0169] Those skilled in the art should understand that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media containing computer-usable program code.

[0170] The present application is described with reference to the flowcharts and / or block diagrams of methods, apparatuses (systems), and computer program products according to the embodiments of the present application. It should be understood that each flow and / or block in the flowchart and / or block diagram, as well as the combination of flows and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing devices generate means for implementing the functions specified in one Figure 1 one flow or multiple flows and / or blocks Figure 1 one block or multiple blocks.

[0171] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory generate a manufactured article including instruction means, and the instruction means implement the functions specified in one Figure 1 one flow or multiple flows and / or blocks Figure 1 one block or multiple blocks.

[0172] These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operation steps are executed on the computer or other programmable device to generate a computer-implemented process, and thus the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one Figure 1 one flow or multiple flows and / or blocks Figure 1 one block or multiple blocks.

[0173] After considering the specification and the disclosure of the invention, those skilled in the art will readily think of other embodiments of the present invention. The present application is intended to cover any variations, uses, or adaptations of the present invention, which follow the general principles of the present invention and include the common general knowledge or conventional technical means in the technical field not disclosed in the present invention. The specification and the embodiments are only regarded as exemplary, and the true scope and spirit of the present invention are pointed out by the following claims.

[0174] It should be understood that the present invention is not limited to the exact structures described above and shown in the drawings, and various modifications and changes can be made without departing from its scope. The scope of the present invention is only limited by the appended claims.

[0175] The above are only the preferred embodiments of the present invention and do not impose any limitation on the present invention. Any simple modifications, changes, and equivalent structural changes made to the above embodiments based on the technical essence of the present invention still fall within the scope of protection of the technical solution of the present invention.

Claims

1. A method for optimizing the collaborative distribution path of multiple unmanned aerial vehicles and trucks, characterized in that, Including: When a collaborative distribution task request is received, the input feature φ of the upper-level DQN is obtained truck,t and the input feature φ of the lower-level MADQN drone,t ; Input the input feature φ of the upper-level DQN truck,t and the input feature φ of the lower-level MADQN drone,t into the trained hierarchical MADQN model to obtain the optimal distribution path for multi-UAV-truck cooperation.

2. The multi-UAV-truck collaborative distribution path optimization method according to claim 1, wherein The input feature φ of the upper DQN truck,t is as follows: Among them, represents the position information of the current truck, represents the task status of the current truck, is the overall revenue of the current upper-level truck module, is the set of visited stops.

3. The multi-UAV-truck collaborative distribution path optimization method according to claim 1, wherein The input feature φ of the lower-layer MADQN drone,t is as follows: Among them, represents the position information of the current UAV, is 0 or 10, represents the current task status of the UAV, represents the total travel distance of the UAV at the current moment, represents the payload of the UAV at the current moment, is the battery energy consumption of the UAV, is the overall revenue of the current UAV, φ' drone,t represents that each UAV can observe the status information of other UAVs.

4. The multi-UAV-truck collaborative distribution path optimization method according to claim 1, characterized in that The hierarchical MADQN model includes an upper DQN and a lower MADQN.

5. The multi-UAV-truck collaborative distribution path optimization method according to claim 4, wherein During the training process of the hierarchical MADQN model, the reward function corresponding to the upper DQN is: r truck = w1r tpath + w2r visit where r tpath is the shortest distance reward, r visit is the reward for successful visit to the docking point, and w1 and w2 represent weight coefficients; During the training process of the hierarchical MADQN model, the reward function corresponding to the lower MADQN is: r drone ＝r1r dpath +r2r task +r3r energy +r4r crash +r5r all Among them, r dpath is the shortest distance reward for the drone to perform the delivery task, r task is the task completion reward, r energy is the energy consumption reward of the drone, r crash represents the penalty term when the distance between drones is less than the safety threshold, r all is the overall additional reward after visiting all customer points in the area of this docking point. r1, r2, r3, r4, and r5 represent weight coefficients.

6. The multi-UAV-truck collaborative distribution route optimization method according to claim 4, characterized in that The hierarchical MADQN model is trained using the gradient descent method.

7. The multi-UAV-truck collaborative distribution path optimization method according to claim 4, wherein, The implementation strategy π of the upper DQN truck is as follows: π truck :(φ truck ; θ truck ) → k i Among them, θ truck represents the model parameters of the upper DQN, and k i ∈C represents the temporarily selected target parking point of the truck, and C represents the set of temporarily selected target parking points; Control strategy π of the lower-level MADQN drone is as follows: π drone :(φ drone ; θ drone ) → a u Among them, θ drone represents the model parameters of the lower-level MADQN, a u ∈PR u represents the selected path planning strategy, and PR u represents the set of planned paths of the UAVs.

8. A multi-UAV-truck collaborative distribution path optimization system, characterized in that, Including: An acquisition module, configured to acquire the input feature φ of the upper-level DQN when receiving a collaborative distribution task request truck,t and the input feature φ of the lower-level MADQN drone,t ; A prediction module for inputting the input feature φ of the upper-level DQN truck,t and the input feature φ of the lower-level MADQN drone,t into the trained hierarchical MADQN model to obtain the optimal delivery path for multi-UAV-truck collaboration.

9. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the multi-UAV-truck collaborative distribution path optimization method according to any one of claims 1-7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the multi-UAV-truck collaborative distribution path optimization method according to any one of claims 1-7.

Images