A low-altitude logistics multi-UAV collaborative delivery method based on deep reinforcement learning

Abstract

This invention provides a low-altitude logistics multi-UAV collaborative delivery method based on deep reinforcement learning, comprising: acquiring delivery tasks and multi-level dynamic airspace information, and allocating delivery tasks to idle UAVs according to a task allocation mechanism; constructing an airspace scheduling mechanism based on the multi-level dynamic airspace information, wherein the airspace scheduling mechanism includes an airspace scheduling altitude layer selector and a multi-UAV collaborative pathfinding solver; the airspace scheduling altitude layer selector determines the flight altitude layer of the UAVs according to the delivery tasks and multi-level dynamic airspace information, obtaining altitude layer decisions; the multi-UAV collaborative pathfinding solver performs horizontal range collaborative path planning at each altitude layer, obtaining path decisions; and the UAVs execute delivery tasks according to the altitude layer decisions and path decisions. This invention's method achieves high throughput and low latency delivery goals in complex urban scenarios and reduces operational risks caused by airspace conflicts.

Description

Technical Field

[0001] This invention relates to the field of low-altitude logistics technology, and in particular to a multi-UAV collaborative delivery method for low-altitude logistics based on deep reinforcement learning. Background Technology

[0002] With the continuous advancement of modern intelligent transportation technology and the ever-increasing demand for commuting, transportation, and on-demand services in human settlements, the idea of ​​leveraging untapped aerospace resources through aircraft technology to expand the economic functions of traditional ground-based road networks has emerged. As an emerging economic form, the Low-Altitude Economy, driven by various low-altitude flight activities of manned and unmanned aircraft, radiates and promotes the integrated development of related fields in the low-altitude domain, possessing strong economic potential and application prospects. Unmanned aerial vehicle (UAV) logistics is a sub-scenarios with immense economic potential within the Low-Altitude Economy. This service utilizes unmanned aerial vehicles (UAVs) for "last-mile" short-distance delivery in low-altitude airspace in urban or remote areas, leveraging the flexibility, rapid response capabilities, and environmental friendliness of UAVs to provide efficient logistics services. In situations of urban congestion and inconvenient ground transportation, UAVs can quickly deliver packages to customers by air, significantly reducing delivery time. This is particularly suitable for time-sensitive delivery tasks, such as the delivery of medicines and medical supplies. Meanwhile, delivery drones are generally powered by electricity, which reduces emissions of carbon dioxide and other pollutants compared to traditional logistics vehicles.

[0003] In existing research on drone logistics solutions, researchers mainly focus on planning the optimal delivery plan for a single drone within the drone delivery service area, based on different navigation modes, through reasonable path planning schemes and heuristic pathfinding algorithms such as A*. However, a single drone is insufficient to meet the throughput requirements of modern delivery systems and the real-time demands of users. Therefore, increasing research is also focusing on low-altitude logistics delivery solutions based on multiple drones. By having multiple drones simultaneously perform logistics tasks, the performance of the delivery system can be improved. However, implementing multi-drone logistics delivery presents a series of challenges.

[0004] First, there is the complexity of multi-drone three-dimensional airspace operations. The atomic task of multi-drone logistics delivery is to operate multiple drones, each with its assigned task, planning the path from the task's starting point to its destination. This problem can be viewed as a multi-agent path finding (MAPF) problem. However, compared to agents operating in planar environments, such as warehouse robots, drones also need to consider vertical movement. In single-drone scenarios, pathfinding schemes considering three-dimensional movement can be solved with acceptable computational overhead. However, in multi-drone scenarios, simply extending the solver applicable to 2D scenarios to a three-dimensional solution space greatly increases computational overhead, making it difficult to find a feasible solution within an acceptable timeframe.

[0005] Secondly, potential airspace congestion restricts the performance of the delivery system. The nature of drone delivery tasks dictates that drones need to make rapid movements with large initial spans within a certain airspace. With multiple drones operating simultaneously, the convergence of multiple delivery routes on a spatiotemporal scale is inevitable, generating potential traffic congestion and hindering the overall performance of the delivery system. To mitigate the almost unavoidable airspace congestion problem in multi-drone systems, existing solutions employ a hierarchical airspace management approach. Through altitude-level stratification and the design of flight paths at each altitude level, drone flight paths are hierarchically designed within the low-altitude airspace to disperse drones across different altitudes and alleviate airspace congestion. However, current solutions rely solely on the concept of hierarchical airspace division and lack specific airspace scheduling strategies. How to further manage airspace effectively through airspace scheduling schemes remains to be discussed. Summary of the Invention

[0006] In view of this, the present invention provides a low-altitude logistics multi-UAV collaborative delivery method based on deep reinforcement learning to solve the above problems.

[0007] This invention provides a low-altitude logistics multi-UAV collaborative delivery method based on deep reinforcement learning, comprising: acquiring delivery tasks and multi-level dynamic airspace information, and allocating the delivery tasks to idle UAVs according to a task allocation mechanism; constructing an airspace scheduling mechanism based on the multi-level dynamic airspace information, wherein the airspace scheduling mechanism includes an airspace scheduling altitude layer selector and a multi-UAV collaborative pathfinding solver; the airspace scheduling altitude layer selector determines the flight altitude layer of the UAVs according to the delivery tasks and multi-level dynamic airspace information, obtaining an altitude layer decision; the multi-UAV collaborative pathfinding solver performs horizontal range collaborative path planning at each altitude layer, obtaining a path decision; and the UAVs execute the delivery tasks according to the altitude layer decisions and the path decisions.

[0008] In another implementation of the present invention, the dynamic airspace information includes the current location information of the UAV, the layout of obstacles at each layer, the location information of other UAVs distributed in the airspace, the delivery path, and the task execution status.

[0009] In another implementation of the present invention, the airspace scheduling altitude layer selector aggregates the dynamic airspace information through an attention layer and learns the potential dependencies between UAVs.

[0010] In another implementation of the invention, the attention layer calculates the weighted attention of the h-th head using the following formula:

[0011]

[0012] Among them, e h Let d be the input vector. k The dimension of the individual features vectorized from the dynamic spatial information extracted by the encoder. For query matrix, The key matrix, It is a value matrix.

[0013] In another implementation of the present invention, the communication vectors between the UAVs are merged through a bidirectional long short-term memory network, and the attention weights are obtained using the following formula:

[0014]

[0015] Here, i and j represent different drones, FC(·) is a fully connected layer, and Gumble(·) is the Gumbel-Softmax function used to implement backpropagation.

[0016] In another aspect, the present invention provides a low-altitude logistics multi-UAV collaborative delivery system based on deep reinforcement learning, comprising: a task allocation module for acquiring delivery tasks and multi-level dynamic airspace information, and allocating the delivery tasks to idle UAVs according to a task allocation mechanism; an airspace scheduling module for constructing an airspace scheduling mechanism based on the multi-level dynamic airspace information, wherein the airspace scheduling mechanism includes an airspace scheduling altitude layer selector and a multi-UAV collaborative pathfinding solver; the airspace scheduling altitude layer selector determines the flight altitude layer of the UAVs according to the delivery tasks and multi-level dynamic airspace information, obtaining an altitude layer decision; the multi-UAV collaborative pathfinding solver performs horizontal range collaborative path planning at each altitude layer, obtaining a path decision; and a delivery module for the UAVs to execute the delivery tasks according to the altitude layer decision and the path decision.

[0017] This invention presents a low-altitude logistics multi-UAV collaborative delivery method based on deep reinforcement learning. It designs a height-level scheduling strategy for UAV delivery tasks in the vertical dimension, thereby coordinating and planning the airspace distribution of UAVs. By alleviating airspace congestion, it improves the performance of the delivery system. It can reduce the complexity of multi-UAV pathfinding problems, allowing UAVs to solve pathfinding problems without needing to solve a complex three-dimensional action solution space, enabling pathfinding within a feasible timeframe. This achieves an efficient and real-time task allocation mechanism and airspace coordination. Furthermore, it allows each UAV to perform real-time path planning and airspace coordination in a dynamic environment, thus achieving high throughput and low latency delivery goals in complex urban scenarios and reducing operational risks caused by airspace conflicts. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. By reading the detailed description of the embodiments below, the advantages and benefits of the solutions will become clear to those skilled in the art. The accompanying drawings are only for illustrating preferred embodiments and are not intended to limit the present invention.

[0019] In the attached diagram:

[0020] Figure 1 This is a schematic diagram of a low-altitude logistics multi-UAV collaborative delivery method based on deep reinforcement learning, according to an embodiment of the present invention.

[0021] Figure 2 This is a schematic diagram of the overall structure of a multi-drone collaborative delivery framework in a low-altitude logistics scenario according to an embodiment of the present invention.

[0022] Figure 3 This is a schematic diagram of a joint airspace scheduling and multi-UAV cooperative pathfinding algorithm framework based on deep reinforcement learning, according to an embodiment of the present invention.

[0023] Figure 4 This is a schematic diagram of a city simulation scene according to an embodiment of the present invention.

[0024] Figure 5 This is a schematic diagram illustrating the impact of different altitude layer selection strategies on the performance of the UAV cooperative pathfinding module in an experimental simulation of an embodiment of the present invention.

[0025] Figure 6 This is a schematic diagram comparing the average continuous delivery throughput of different delivery systems according to an embodiment of the present invention.

[0026] Figure 7 This is a schematic diagram comparing the average continuous delivery delay rate of different delivery systems according to an embodiment of the present invention. Detailed Implementation

[0027] To enable those skilled in the art to better understand the technical solutions in the embodiments of the present invention, the technical solutions in the embodiments of the present invention will be clearly and thoroughly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art should fall within the protection scope of the present invention.

[0028] Figure 1 This is a schematic diagram of a low-altitude logistics multi-UAV collaborative delivery method based on deep reinforcement learning, provided by an embodiment of the present invention. Figure 1 As shown, this embodiment mainly includes:

[0029] S101. Obtain delivery tasks and multi-level dynamic airspace information, and allocate the delivery tasks to idle drones according to the task allocation mechanism.

[0030] For example, such as Figure 2 As shown, a collaborative delivery framework for drones is constructed. The task allocation module effectively assigns delivery tasks to idle drones. Based on a low-altitude drone collaborative delivery framework with multi-layer airspace division, a real-time task allocation mechanism for multiple drones within a region is realized, thereby improving the overall task throughput in the continuous path planning process.

[0031] Based on the concept of multi-level airspace, the airspace within the delivery range is divided into discrete altitude layers for hierarchical management, and drones are allocated to different altitude layers for delivery through an airspace scheduling mechanism to alleviate air congestion.

[0032] S102. Based on the multi-level dynamic airspace information, an airspace scheduling mechanism is constructed, wherein the airspace scheduling mechanism includes an airspace scheduling altitude layer selector and a multi-UAV cooperative pathfinding solver.

[0033] For example, the algorithm training of the airspace scheduling mechanism is based on deep reinforcement learning methods to make efficient and rapid decisions under dynamic and complex airspace traffic, so that hierarchical scheduling can effectively weigh various factors and make high-level decisions with short delivery distances and low probability of conflict.

[0034] Instead of directly solving for the delivery path in three-dimensional space with high complexity, the vertical and horizontal displacement of UAVs are controlled by an airspace scheduling altitude layer selector and a multi-UAV cooperative pathfinding solver. The airspace scheduling module learns an efficient altitude layer scheduling decision scheme under complex and dynamic airspace traffic conditions through deep reinforcement learning.

[0035] S103. The airspace scheduling altitude layer selector determines the flight altitude layer of the UAV based on the delivery task and multi-level dynamic airspace information, and obtains the altitude layer decision.

[0036] For example, in the constructed low-altitude drone collaborative delivery framework, the drone only performs vertical ascent and descent at the starting point and destination of a single delivery, based on the altitude layer it is scheduled to, thereby reducing the computational complexity of three-dimensional spatial pathfinding.

[0037] S104. The multi-UAV cooperative pathfinding solver performs horizontal collaborative path planning at each altitude level to obtain path decisions.

[0038] For example, during collaborative pathfinding within the altitude layer, the UAV uses a convolutional neural network's perception module to perceive and adapt to changes in the surrounding environment in real time, serving as downstream input for path decision-making. Information is shared through an attention-based multi-agent communication module, selecting agents with a high probability of conflict to interact with, thus reducing the risk of air congestion.

[0039] The multi-drone collaborative delivery solution is solved dynamically in real time. The drones learn real-time pathfinding and information exchange strategies in a dynamic airspace environment through a multi-agent deep reinforcement learning method, rather than solving the delivery path in advance and then performing pathfinding and delivery.

[0040] The training process of the airspace scheduling module jointly considers the influence of the horizontal multi-UAV pathfinding solver. Its training sampling process depends on the pathfinding scheme given by the horizontal multi-UAV pathfinding. The horizontal multi-UAV solver at each altitude layer maximizes the individual benefits of the multi-UAV cooperative pathfinding problem under the framework of centralized training and distributed execution through multi-agent reinforcement learning method, and simultaneously optimizes the global benefits.

[0041] S105. The drone executes the delivery task based on the altitude layer decision and the path decision.

[0042] For example, the delivery completion path of each drone includes flying from the current location to the delivery origin to pick up the goods, and flying from the delivery origin to the delivery destination to release the goods. After completing the current delivery task, the drone will continue to be assigned new delivery tasks.

[0043] The drone delivery process is continuous, meaning that the drone will be continuously assigned delivery tasks, climb to the assigned altitude and navigate horizontally, and finally land at the destination to complete the delivery task. This process involves task allocation mechanisms, airspace coordination, and continuous decision-making and mutual influence of drone navigation.

[0044] This invention presents a low-altitude logistics multi-UAV collaborative delivery method based on deep reinforcement learning. It designs a height-level scheduling strategy for UAV delivery tasks in the vertical dimension, thereby coordinating and planning the airspace distribution of UAVs. By alleviating airspace congestion, it improves the performance of the delivery system. It can reduce the complexity of multi-UAV pathfinding problems, allowing UAVs to solve pathfinding problems without needing to solve a complex three-dimensional action solution space, enabling pathfinding within a feasible timeframe. This achieves an efficient and real-time task allocation mechanism and airspace coordination. Furthermore, it allows each UAV to perform real-time path planning and airspace coordination in a dynamic environment, thus achieving high throughput and low latency delivery goals in complex urban scenarios and reducing operational risks caused by airspace conflicts.

[0045] In another implementation of the present invention, the dynamic airspace information includes the current location information of the UAV, the layout of obstacles at each layer, the location information of other UAVs distributed in the airspace, the delivery path, and the task execution status.

[0046] For example, a deep learning network perception module is designed based on a convolutional neural network architecture and a gated recurrent unit, enabling each UAV to extract state learning information about the UAV, obstacles, and heuristic pathfinding guidance within the observable range during pathfinding.

[0047] In another implementation of the present invention, the airspace scheduling altitude layer selector aggregates the dynamic airspace information through an attention layer and learns the potential dependencies between UAVs.

[0048] For example, considering that both vertical airspace scheduling and horizontal multi-UAV pathfinding require efficient and rapid decision-making in complex and dynamic air traffic, an algorithm based on deep reinforcement learning for joint airspace scheduling and multi-UAV cooperative pathfinding is proposed, such as... Figure 3 As shown, a Markov Decision Process (MDP) is established in the design of the airspace scheduling mechanism. The state input is dynamic airspace information monitored and obtained by the distribution center, including the current delivery task of the UAV, the location information of other UAVs in the airspace, and the task execution status. For high-dimensional, dynamic airspace state information, a multi-head attention (MHA) mechanism is used to aggregate the state information. Based on the attention mechanism, a representation of the high-dimensional airspace state is learned, and the implicit associations between the UAV to be scheduled and other UAVs are extracted.

[0049] Specifically, the input matrix is ​​first processed by a multi-layer perceptron (MLP) encoder to extract vectorized individual features of dimension d. k These embedded features are fed into a system with N hThe attention layer of the device takes a single query vector Q from the current drone, projects it linearly, compares it with a set of key values ​​K of the features described by other drones, and then computes a random attention matrix to aggregate a set of output values ​​V.

[0050] Attention layer network training query matrix Key matrix Sum matrix

[0051]

[0052] In another implementation of the invention, the attention layer calculates the weighted attention of the h-th head using the following formula:

[0053]

[0054] Among them, e h Let d be the input vector. k The dimension of the individual features vectorized from the dynamic spatial information extracted by the encoder. For query matrix, The key matrix, It is a value matrix.

[0055] For example, the outputs of all heads are combined through a linear layer to form a summed weighted attention vector, representing the implicit correlation between the UAV and the airspace state. This vector is added to the output of the autoencoder and serves as the upstream input to the policy and value networks of the deep reinforcement learning. In implementing the reinforcement learning mechanism, the algorithm structure of this invention is based on a discretized Soft Actor-Critic (SAC) method, integrating long-term rewards and low-entropy policy penalties, achieving broad and robust learning through effective exploration and entropy regularization.

[0056] The network of this invention comprises an actor network and two critic networks. The actor network outputs the probability distribution of actions and generates random actions through sampling; the critic networks are used to estimate the Q-value of state-action pairs, reducing overestimation bias. During algorithm training, after obtaining a high-dimensional representation of the dynamic and complex spatial domain through an attention mechanism, the policy network samples actions and interactions with the environment, recording the experience in the replay pool; the Q-network is updated using the minimum Q-value, and the policy network is updated by maximizing the sum of reward and entropy.

[0057] In another implementation of the present invention, the communication vectors between the UAVs are merged through a bidirectional long short-term memory network, and the attention weights are obtained using the following formula:

[0058]

[0059] Here, i and j represent different drones, FC(·) is a fully connected layer, and Gumble(·) is the Gumbel-Softmax function used to implement backpropagation.

[0060] For example, through an airspace scheduling mechanism, drones assigned to tasks but not yet performing delivery pathfinding are scheduled to a specific altitude layer. Multiple drones in the same altitude layer then engage in horizontal collaborative pathfinding, and a multi-agent pathfinding solver solves the action for the next moment based on the current observations.

[0061] To enable drones to accurately interpret observational information and cooperate with surrounding drones to further mitigate potential conflicts, the pathfinding network design for each drone incorporates observation and communication structures such as... Figure 3 As shown, the observation matrix can be represented by a stack of binary vectors as (U, O, E), where U represents the position of the UAV within the observation radius, O represents the distribution of obstacles, and E is composed of four two-dimensional one-hot vectors stacked together. Each binary element indicates whether the distance to the target point is shortened after performing the corresponding action at the corresponding position.

[0062] The features of the observation matrix are first extracted through an observation encoder based on a convolutional neural network (CNN). This encoder consists of a single convolutional layer, three residual convolutional blocks, and a gated recurrent unit (GRU) stacked together. Subsequently, the UAV shares its observation encoding with nearby UAVs within a finite radius, using an attention-based two-layer communication structure shared between each UAV. In the first layer of communication, the UAV identifies other UAVs within its observation range that are most likely to collide with it through a hard attention mechanism.

[0063] In the second phase of communication, a soft attention model was used to further infer the correlation between UAVs. Its mechanism is similar to the attention aggregator in the airspace scheduling mechanism. During the update, the multi-agent pathfinding solver adopts the Counterfactual Multi-Agent Policy Gradient (COMA) multi-agent reinforcement learning method to optimize the update. Under the framework of centralized training and distributed execution, the multi-UAV cooperative pathfinding problem is maximized based on individual benefits, while simultaneously optimizing the global benefits.

[0064] The communication between multiple drones is based on a two-stage attention mechanism, which selects only drones with high relevance (high probability of conflict) within the observable range for cooperative communication, thereby resolving pathfinding conflicts at a low communication cost.

[0065] In another implementation of the present invention, based on the construction and definition of the above algorithm modules, the present invention designs a low-altitude logistics multi-UAV collaborative delivery scheme based on deep reinforcement learning as follows:

[0066] 1) Initialize system parameters and status information based on the current distribution of delivery orders, drone distribution, etc.:

[0067] a) Initialize the drone ensemble D at the central apron and define the number of currently schedulable altitude layers;

[0068] b) Initialize the set of undelivered tasks T U The set of assigned tasks T A ;

[0069] 2) Multi-drone delivery process:

[0070] a) Repeat;

[0071] b) Select the drone D with the current event set as "idle" from the drone collection. E ;

[0072] √ Assign delivery tasks and solve for the drone-task pair S. d,t ,d∈D E ,t∈T U ;

[0073] √ The assigned task is removed from the undelivered task set T U Add to the assigned task set T A ;

[0074] √Utilize a multi-head attention mechanism to aggregate spatial states and generate spatial representation vectors that are then fed into the downstream network;

[0075] √ Use the soft actor-critic model to calculate the operational altitude layer for the drone delivery in the current instance;

[0076] √ Update the current drone destination; the drone enters ascent mode.

[0077] c) Iterate through each drone in the drone collection;

[0078] √ For all drones currently performing horizontal pathfinding:

[0079] 1. Perform state coding based on locally observed locations of neighboring UAVs, obstacles, and targets;

[0080] 2. By utilizing a two-layer soft and hard attention intelligent agent mechanism, state information encoding is transmitted between neighboring UAVs to obtain cooperative information of adjacent UAVs at the same altitude layer;

[0081] 3. Calculate the pathfinding action at the current time step using an MLP decision network;

[0082] 4. Step forward and update the drone's location information;

[0083] √For all drones undergoing vertical displacement:

[0084] 1. If there is no conflict, move up or down one height level;

[0085] 2. In the event of a conflict, perform a time-step hovering in the air;

[0086] 3. For drones that have reached their assigned altitude, set the next time step to enter horizontal pathfinding mode;

[0087] 4. For drones that have arrived at their destination, deliver the goods.

[0088] d) Until all delivery tasks are completed.

[0089] The algorithm proposed in this invention, which is based on deep reinforcement learning for joint airspace scheduling and multi-UAV collaborative pathfinding, allows each UAV to perform real-time path planning and airspace coordination in a dynamic environment. This enables high-throughput, low-latency delivery in complex urban scenarios and reduces operational risks caused by airspace conflicts.

[0090] In another implementation of the present invention, in order to verify the performance of the delivery system of the present invention, the experimental simulation focuses on the urban center of the city. This scenario has high practical significance in drone delivery services. At the same time, high-rise buildings restrict the already limited airspace and increase the complexity of drone scheduling, so the challenges are also quite significant. Figure 4 To verify the performance of this invention, two grid-based 3D city maps were simulated, each covering an area of ​​2500m × 2500m. Multiple drone delivery services were deployed in the airspace below 80m. To study the impact of building density on drone delivery efficiency, the two maps were named sparse and dense, with building coverage rates set to 21% and 38%, respectively. The physical grid size was set to 25m × 25m, and the allowed height layer division was set to 4 layers to stratify randomly generated buildings as obstacles in the height dimension. The number of drones was set to 32 and 64, representing airspace traffic flow during normal and busy periods, respectively.

[0091] In scenarios with varying air traffic pressures, the proposed solution is compared with baselines such as Equal-DistributedAstar (EDA, which distributes UAVs evenly to different altitude layers and then performs A* pathfinding), Nearest-Layer-FirstSearch (NFS, which prioritizes the nearest altitude layer for delivery paths), and Soft Actor-Critic (SAC, which uses a non-attentional mechanism and employs soft actor-critic reinforcement learning for airspace scheduling pathfinding baselines).

[0092] Figure 5 The experiment simulates the impact of different altitude layer selection strategies on the performance of the UAV cooperative pathfinding module. The results show that in various scenarios, the solution of this invention can further understand the attention association of UAVs in complex dynamic airspace in a higher dimension. While ensuring that the altitude layer distribution of UAVs is not overly crowded, it also takes into account the shorter delivery distance, thereby significantly reducing the average delivery step (AS) and maximum delivery step (MS), effectively reducing the average conflict (AC) between UAVs, and avoiding altitude layer congestion.

[0093] Figure 6 and Figure 7 These are comparisons of the average throughput of continuous delivery tasks and the delivery delay rate of different delivery systems. The joint collaboration strategy of this invention can effectively empower delivery systems, achieving the highest delivery throughput and the lowest delivery delay rate in various continuous delivery scenarios. This advantage of coordination is even more pronounced in scenarios with busier air traffic and higher urban building density, because busy air traffic and more limited airspace resources highlight the necessity of overall airspace coordination and drone collaboration.

[0094] Another aspect of the present invention provides a low-altitude logistics multi-UAV collaborative delivery system based on deep reinforcement learning, comprising:

[0095] Task allocation module: used to acquire delivery tasks and multi-level dynamic airspace information, and allocate the delivery tasks to idle drones according to the task allocation mechanism.

[0096] An airspace scheduling module is used to construct an airspace scheduling mechanism based on the multi-level dynamic airspace information. The airspace scheduling mechanism includes an airspace scheduling altitude layer selector and a multi-UAV cooperative pathfinding solver. The airspace scheduling altitude layer selector determines the flight altitude layer of the UAV based on the delivery task and the multi-level dynamic airspace information to obtain an altitude layer decision. The multi-UAV cooperative pathfinding solver performs horizontal range cooperative path planning at each altitude layer to obtain a path decision.

[0097] Delivery module: used by the UAV to execute the delivery task based on the altitude layer decision and the path decision.

[0098] In the context of low-altitude logistics, this solution addresses the challenge posed by the computational complexity of multi-drone three-dimensional airspace pathfinding in multi-drone collaborative delivery. It designs a multi-drone collaborative delivery framework based on the concept of multi-level airspace. The framework includes modules such as dynamic task allocation, airspace scheduling, and multi-drone dynamic pathfinding solvers for each airspace layer. It achieves real-time and continuous drone delivery with low computational complexity, providing a framework foundation for efficient and rapid collaborative scheduling and path planning of multiple drones in urban environments.

[0099] To effectively avoid airspace congestion, the airspace scheduling module is responsible for implementing horizontal collaborative path planning and information sharing among UAVs at the altitude level based on the current UAV status and airspace traffic distribution. This is achieved through a multi-UAV collaborative pathfinding solver, which learns pathfinding state representations via convolutional neural networks, resolves pathfinding conflicts at route intersections through a two-stage attention communication mechanism, and learns multi-UAV collaborative strategies through multi-agent reinforcement learning. This selects altitude levels with low congestion probability and short delivery paths for the current delivery UAV to reduce airspace congestion. Implicit associations between UAVs are extracted through a multi-head attention mechanism, and deep reinforcement learning enables efficient decision-making in complex, high-dimensional airspace, improving the collaborative capabilities of multiple UAVs in dynamic and complex airspace and further enhancing the system's delivery efficiency. The low-altitude delivery scheme proposed in this invention enables the delivery system to continuously complete multi-UAV logistics tasks, exhibiting high task throughput and low delivery delay rates.

[0100] The method of this invention can be applied in industrial control systems in industries such as manufacturing, energy, and transportation, enhancing the system's resistance to attacks by providing customized security services.

[0101] In another aspect of the present invention, the electronic device includes: a processor, a memory, and a communication bus and a communication interface.

[0102] in:

[0103] The processor, memory, and communication interface communicate with each other via a communication bus.

[0104] A communication interface is used to communicate with other electronic devices or servers.

[0105] The processor is used to execute programs, specifically, to execute any of the steps of the low-altitude logistics multi-drone collaborative delivery method based on deep reinforcement learning in the above embodiments.

[0106] Specifically, the program may include program code, which includes computer operation instructions.

[0107] The processor may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of this application. The one or more processors included in the smart device may be processors of the same type, such as one or more CPUs; or they may be processors of different types, such as one or more CPUs and one or more ASICs.

[0108] Memory is used to store programs. Memory may include high-speed RAM, and may also include non-volatile memory, such as at least one disk drive.

[0109] Specifically, the program can be used to cause the processor to execute the steps of any of the deep reinforcement learning-based low-altitude logistics multi-UAV collaborative delivery methods described in the embodiments. The specific implementation of each step in the program can be found in the corresponding descriptions of the steps and units executed in any of the deep reinforcement learning-based low-altitude logistics multi-UAV collaborative delivery methods described above, and will not be repeated here. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the devices and modules described above can be referred to the corresponding process descriptions in the foregoing method embodiments.

[0110] An exemplary embodiment of this application also provides a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause a computer to perform the methods of various embodiments of this application.

[0111] The methods described above according to embodiments of the present invention can be implemented in hardware, firmware, or as software or computer code that can be stored in a recording medium (such as a CD-ROM, RAM, floppy disk, hard disk, or magneto-optical disk), or as computer code originally stored on a remote recording medium or a non-transitory machine-readable medium and subsequently stored on a local recording medium, downloaded via a network. Thus, the methods described herein can be processed by software stored on a recording medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware (such as an ASIC or FPGA). It is understood that the computer, processor, microprocessor controller, or programmable hardware includes storage components (e.g., RAM, ROM, flash memory, etc.) capable of storing or receiving software or computer code, which, when accessed and executed by the computer, processor, or hardware, implements the methods described herein. Furthermore, when a general-purpose computer accesses code used to implement the methods shown herein, the execution of the code transforms the general-purpose computer into a dedicated computer for executing the methods shown herein.

[0112] Specific embodiments of the invention have now been described. Other embodiments are within the scope of the appended claims. In some cases, the actions described in the claims can be performed in a different order and still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing can be advantageous.

[0113] It should be noted that all directional indicators (such as up, down, left, right, back, etc.) in the embodiments of the present invention are only used to explain the relative positional relationship and movement of each component in a certain specific posture (as shown in the figure). If the specific posture changes, the directional indicator will also change accordingly.

[0114] In the description of this invention, the terms "first" and "second" are used only for convenience in describing different components or names, and should not be construed as indicating or implying a sequential relationship, relative importance, or implicitly specifying the number of technical features indicated. Thus, a feature defined with "first" and "second" may explicitly or implicitly include at least one of that feature.

[0115] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art of the present invention. The terms used in this specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention.

[0116] It should be noted that although specific embodiments of the present invention have been described in detail with reference to the accompanying drawings, this should not be construed as limiting the scope of protection of the present invention. Various modifications and variations that can be made by those skilled in the art without inventive effort within the scope described in the claims still fall within the scope of protection of the present invention.

[0117] The examples of the embodiments of the present invention are intended to concisely illustrate the technical features of the embodiments of the present invention, so that those skilled in the art can intuitively understand the technical features of the embodiments of the present invention, and are not intended to be an improper limitation of the embodiments of the present invention.

[0118] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A low-altitude logistics multi-UAV collaborative delivery method based on deep reinforcement learning, characterized in that, include: Acquire delivery tasks and multi-level dynamic airspace information, and allocate the delivery tasks to idle drones according to the task allocation mechanism; Based on the multi-level dynamic airspace information, an airspace scheduling mechanism is constructed, wherein the airspace scheduling mechanism includes an airspace scheduling altitude layer selector and a multi-UAV cooperative pathfinding solver. The airspace scheduling altitude layer selector determines the flight altitude layer of the UAV based on the delivery task and multi-level dynamic airspace information, thus obtaining an altitude layer decision. The airspace scheduling altitude layer selector aggregates the airspace state using a multi-head attention mechanism, generates an airspace representation vector, and inputs it into the downstream network. It also uses a soft actor-critic model to calculate the operational altitude layer of the UAV for the current delivery. The multi-UAV cooperative pathfinding solver performs horizontal collaborative path planning at each altitude level to obtain path decisions; The drone executes the delivery task based on the altitude layer decision and the path decision; During the collaborative pathfinding process within the altitude layer, the UAV perceives and adapts to changes in the surrounding environment in real time through the perception module of the convolutional neural network, which serves as the downstream input for path decision-making. Based on the convolutional neural network architecture and gated recurrent units, a deep learning network perception module is designed, enabling each UAV to extract information about the UAV, obstacles, and heuristic pathfinding guidance within the observable range for state learning during pathfinding. The communication between multiple UAVs is based on a two-stage attention mechanism, which selects only UAVs with high correlation within the observable range for cooperative communication to resolve pathfinding conflicts with low communication costs. During the update, the multi-agent pathfinding solver adopts a multi-agent reinforcement learning method with counterfactual multi-agent policy gradient to optimize the update. Under the framework of centralized training and distributed execution, it maximizes the individual benefits of the multi-UAV cooperative pathfinding problem and optimizes the global benefits at the same time.

2. The method according to claim 1, characterized in that, The dynamic airspace information includes the current location of the UAV, the layout of obstacles at each level, the location information of other UAVs in the airspace, delivery routes, and mission execution status.

3. The method according to claim 2, characterized in that, In the airspace scheduling altitude layer selector, the dynamic airspace information is aggregated through the attention layer, and the potential dependencies between UAVs are learned.

4. The method according to claim 3, characterized in that The attention layer is calculated using the following formula: h Weighted attention based on size: in, For the input vector, The dimension of the individual features vectorized from the dynamic spatial information extracted by the encoder. For query matrix, The key matrix, It is a value matrix.

5. The method according to claim 1, wherein Also includes: The communication vectors between the drones are merged through a bidirectional long short-term memory network, and the attention weights are obtained using the following formula: in, i and j Representing different drones, FC(·) is a fully connected layer, and Gumble(·) is the Gumbel-Softmax function used to implement backpropagation.

6. A low-altitude logistics multi-UAV collaborative delivery system based on deep reinforcement learning, the system being used to implement the method as described in any one of claims 1 to 5, characterized in that, include: Task allocation module: used to acquire delivery tasks and multi-level dynamic airspace information, and allocate the delivery tasks to idle drones according to the task allocation mechanism; An airspace scheduling module is used to construct an airspace scheduling mechanism based on the multi-level dynamic airspace information. This airspace scheduling mechanism includes an airspace scheduling altitude layer selector and a multi-UAV cooperative pathfinding solver. The airspace scheduling altitude layer selector determines the flight altitude layer of the UAV based on the delivery task and the multi-level dynamic airspace information, obtaining an altitude layer decision. The multi-UAV cooperative pathfinding solver performs horizontal range cooperative path planning at each altitude layer, obtaining a path decision. Delivery module: Used by the UAV to execute the delivery task based on the altitude layer decision and the path decision.

Images