Low-altitude logistics multi-unmanned aerial vehicle cooperative distribution method based on deep reinforcement learning

Abstract

The invention provides a low-altitude logistics multi-unmanned aerial vehicle cooperative distribution method based on deep reinforcement learning, and the method comprises the steps: obtaining a distribution task and multi-level dynamic airspace information, and distributing the distribution task to idle unmanned aerial vehicles according to a task distribution mechanism; based on the multi-level dynamic airspace information, an airspace scheduling mechanism is constructed, and the airspace scheduling mechanism comprises an airspace scheduling height level selector and a multi-unmanned aerial vehicle cooperative path-finding solver; the airspace scheduling height level selector determines the flight height level of the unmanned aerial vehicle according to the distribution task and the multi-level dynamic airspace information to obtain a height level decision; the multi-unmanned aerial vehicle cooperative path-finding solver performs cooperative path planning in a horizontal range at each height layer to obtain a path decision; and the unmanned aerial vehicle executes a distribution task according to the height layer decision and the path decision. According to the method provided by the invention, the high-throughput and low-delay delivery target is realized in a complex urban scene, and the operation risk caused by airspace conflicts is reduced.

Description

Technical Field

[0001] The present invention relates to the field of low-altitude logistics technology, and in particular to a low-altitude logistics multi-UAV collaborative delivery method based on deep reinforcement learning. Background Art

[0002] With the continuous progress of modern intelligent transportation technology and the continuous growth of demand for commuting, transportation, and instant services in human settlements, the idea of ​​using aircraft technology to utilize untapped aerospace resources and broaden the economic functions of traditional ground road networks has emerged. As an emerging economic form, the low-altitude economy is driven by various low-altitude flight activities of various manned and unmanned aircraft, radiating and driving the integration and development of related fields in the low-altitude field, and has strong economic potential and application prospects. Drone logistics is a sub-scenario with great economic potential in the field of low-altitude economy. This service uses drones (Unmanned Aerial Vehicles, UAVs) to perform "last mile" short-distance delivery in low-altitude airspace in cities or remote areas, and uses the flexibility, rapid response and environmental friendliness of drones to provide efficient logistics services. In the case of urban congestion and inconvenient ground transportation, drones can quickly deliver packages to customers through the air, greatly reducing delivery time. This is especially suitable for delivery tasks that require timeliness, such as the delivery of medicines and medical supplies. At the same time, delivery drones are generally powered by electricity, which can reduce emissions of carbon dioxide and other pollutants compared to traditional logistics vehicles.

[0003] In the existing research on drone logistics solutions, researchers mainly focus on planning the optimal delivery solution for a single drone within the scope of drone delivery services, based on different navigation modes, through reasonable path planning solutions and heuristic path-finding algorithms such as A*. However, a single drone is difficult to meet the throughput indicators of modern distribution systems for logistics tasks, as well as users' requirements for the immediacy of logistics tasks. Therefore, more and more work has begun to focus on low-altitude logistics delivery solutions based on multiple drones, and improve the performance of the distribution system by having multiple drones perform logistics tasks simultaneously. However, in implementation, there are a series of challenges in considering the logistics delivery of multiple drones.

[0004] One is the complexity of multi-UAV three-dimensional airspace operations. The atomic task of multi-UAV logistics distribution is to control multiple UAVs and plan the path from the task starting point to the task end point under each assigned task. This problem can be regarded as a multi-agent path finding problem (Multi-Agent Path Finding, MAPF). However, compared with intelligent agents such as warehouse robots that operate in a plane, drones also need to consider movement in the height dimension. In a single-UAV scenario, the path finding solution that considers three-dimensional movement can still be solved with an acceptable computational cost, but in a multi-UAV scenario, simply expanding the solver applicable to the 2D scenario into a three-dimensional solution space will greatly increase the computational cost, making it difficult to find a feasible solution within an acceptable time.

[0005] Second, potential airspace congestion restricts the performance of the distribution system. The nature of drone delivery tasks determines that drones need to perform rapid displacement with a large initial span within a certain airspace. Under the premise of multiple drones operating at the same time, it will inevitably lead to the intersection of multiple delivery routes in time and space, resulting in potential traffic congestion and restricting the overall performance of the distribution system. In order to resolve the almost inevitable airspace congestion problem in multi-drone systems, the existing scheme adopts a hierarchical division and management of the airspace. Through the design of altitude stratification and each altitude layer channel, the flight trajectory of the drone is hierarchically designed in the low-altitude airspace to achieve the purpose of dispersing the distribution of drones at various altitudes and eliminating airspace congestion. However, the current scheme only relies on the concept of airspace hierarchical division, and lacks a specific airspace scheduling strategy. How to further effectively manage the airspace through airspace scheduling schemes remains to be discussed. Summary of the invention

[0006] In view of this, the present invention provides a low-altitude logistics multi-UAV collaborative delivery method based on deep reinforcement learning to solve the above problems.

[0007] The present invention provides a low-altitude logistics multi-UAV collaborative delivery method based on deep reinforcement learning, comprising: obtaining delivery tasks and multi-level dynamic airspace information, and assigning the delivery tasks to idle UAVs according to a task allocation mechanism; constructing an airspace scheduling mechanism based on the multi-level dynamic airspace information, wherein the airspace scheduling mechanism includes an airspace scheduling altitude layer selector and a multi-UAV collaborative path finding solver; the airspace scheduling altitude layer selector determines the flight altitude layer of the UAV according to the delivery task and the multi-level dynamic airspace information to obtain an altitude layer decision; the multi-UAV collaborative path finding solver performs collaborative path planning in a horizontal range at each altitude layer to obtain a path decision; the UAV executes the delivery task according to the altitude layer decision and the path decision.

[0008] In another implementation of the present invention, the dynamic airspace information includes the location information of the current drone, the layout of obstacles at each layer, the location information of other drones distributed in the airspace, the delivery path and the task execution status.

[0009] In another implementation of the present invention, in the airspace scheduling altitude layer selector, the dynamic airspace information is aggregated through an attention layer, and the potential dependencies between drones are learned.

[0010] In another implementation of the present invention, the attention layer calculates the weighted attention of the h-th head by the following formula:

[0011]

[0012] Among them, e h is the input vector, d k The dimension of the individual features of the dynamic spatial information vectorized by the encoder, is the query matrix, is the key matrix, is a value matrix.

[0013] In another implementation of the present invention, the communication vectors between the drones are merged through a bidirectional long short-term memory network, and the attention weight is obtained by the following formula:

[0014]

[0015] Here, i and j represent different drones, FC(·) is the fully connected layer, and Gumble(·) is the Gumbel-Softmax function used to implement back-propagation.

[0016] In another aspect of the present invention, a low-altitude logistics multi-UAV collaborative delivery system based on deep reinforcement learning is provided, comprising: a task allocation module: used to obtain delivery tasks and multi-level dynamic airspace information, and assign the delivery tasks to idle UAVs according to the task allocation mechanism; an airspace scheduling module, used to construct an airspace scheduling mechanism based on the multi-level dynamic airspace information, wherein the airspace scheduling mechanism includes an airspace scheduling altitude layer selector and a multi-UAV collaborative path finding solver; the airspace scheduling altitude layer selector determines the flight altitude layer of the UAV according to the delivery task and the multi-level dynamic airspace information, and obtains an altitude layer decision; the multi-UAV collaborative path finding solver performs collaborative path planning in a horizontal range at each altitude layer to obtain a path decision; a delivery module: used for the UAV to execute the delivery task according to the altitude layer decision and the path decision.

[0017] The low-altitude logistics multi-UAV collaborative delivery method based on deep reinforcement learning of the present invention designs a high-level scheduling strategy for UAV delivery tasks in the vertical dimension, and then coordinates and plans the airspace distribution of UAVs, thereby improving the performance of the distribution system by eliminating airspace congestion; it can eliminate the complexity of the multi-UAV pathfinding problem, so that the UAVs do not need to solve the three-dimensional complex action solution space when finding the path, and can perform pathfinding within a feasible time, thereby realizing an efficient and real-time task allocation mechanism and airspace coordination; it allows each UAV to perform real-time path planning and airspace coordination in a dynamic environment, thereby achieving high-throughput and low-latency delivery goals in complex urban scenarios, and reducing the operational risks brought by airspace conflicts. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the prior art descriptions. By reading the detailed description of the following implementations, the advantages and benefits of the solutions become clear to those skilled in the art. The drawings are only used to illustrate the preferred implementations and are not to be considered as limiting the present invention.

[0019] In the attached picture:

[0020] Figure 1 This is a flow chart of a method for collaborative delivery of low-altitude logistics multiple UAVs based on deep reinforcement learning according to an embodiment of the present invention.

[0021] Figure 2 This is a schematic diagram of the overall structure of a multi-UAV collaborative delivery framework in a low-altitude logistics scenario according to an embodiment of the present invention.

[0022] Figure 3 This is a schematic diagram of a joint airspace scheduling and multi-UAV collaborative pathfinding algorithm framework based on deep reinforcement learning according to an embodiment of the present invention.

[0023] Figure 4 The figure is a schematic diagram of a city simulation scene according to an embodiment of the present invention.

[0024] Figure 5 This is a schematic diagram of the impact of different altitude layer selection strategies on the performance of the UAV collaborative pathfinding module in an experimental simulation of an embodiment of the present invention.

[0025] Figure 6 It is a schematic diagram comparing the average task throughput of continuous delivery of different delivery systems according to an embodiment of the present invention.

[0026] Figure 7 It is a schematic diagram comparing the average task delay rates of continuous delivery of different delivery systems according to an embodiment of the present invention. DETAILED DESCRIPTION

[0027] In order to enable those skilled in the art to better understand the technical solutions in the embodiments of the present invention, the technical solutions in the embodiments of the present invention will be described clearly and in detail below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in the field based on the embodiments in the embodiments of the present invention should fall within the scope of protection of the embodiments of the present invention.

[0028] Figure 1 A schematic diagram of a low-altitude logistics multi-UAV collaborative delivery method based on deep reinforcement learning provided by an embodiment of the present invention is shown in FIG. Figure 1 As shown, this embodiment mainly includes:

[0029] S101, obtaining delivery tasks and multi-level dynamic airspace information, and assigning the delivery tasks to idle drones according to a task assignment mechanism.

[0030] For example, Figure 2 As shown in the figure, a UAV collaborative delivery framework is constructed. The task allocation module effectively allocates delivery tasks to idle UAVs. The low-altitude UAV collaborative delivery framework based on multi-layer airspace division realizes the real-time task allocation mechanism of multiple UAVs in the region, thereby improving the overall task throughput in the continuous path planning process.

[0031] Based on the concept of multi-level airspace, the airspace within the delivery range is divided into discrete altitude layers for hierarchical management, and drones are assigned to different altitude layers for delivery through the airspace scheduling mechanism to alleviate air congestion.

[0032] S102. Based on the multi-level dynamic airspace information, an airspace scheduling mechanism is constructed, wherein the airspace scheduling mechanism includes an airspace scheduling altitude layer selector and a multi-UAV collaborative path finding solver.

[0033] Exemplarily, the algorithm training of the airspace scheduling mechanism is based on deep reinforcement learning methods to make efficient and fast decisions under dynamic and complex airspace traffic, so that hierarchical scheduling can effectively weigh various factors and make altitude decisions with short delivery distances and low conflict probabilities.

[0034] The vertical and horizontal displacements of drones are controlled by the airspace scheduling altitude layer selector and the multi-drone collaborative pathfinding solver, rather than directly solving the delivery path of drones in three-dimensional space with high complexity. The airspace scheduling module uses deep reinforcement learning methods to learn efficient altitude layer scheduling decision-making solutions under complex and dynamic airspace traffic conditions.

[0035] S103, the airspace scheduling altitude layer selector determines the flight altitude layer of the UAV according to the delivery task and multi-level dynamic airspace information to obtain an altitude layer decision.

[0036] For example, under the constructed low-altitude UAV collaborative delivery framework, the UAV only performs vertical ascent and descent between the starting point and the destination of a single delivery according to the altitude layer to which it is dispatched, thereby reducing the computational complexity of three-dimensional space pathfinding.

[0037] S104, the multi-UAV collaborative path-finding solver performs collaborative path planning in a horizontal range at each altitude layer to obtain a path decision.

[0038] For example, in the collaborative pathfinding process within the altitude layer, the drone uses the perception module of the convolutional neural network to perceive and adapt to changes in the surrounding environment in real time as the downstream input of the path decision. The multi-agent communication module based on the attention mechanism is used to share information, and the agents with a high probability of conflict with themselves are selected for interaction to reduce the risk of air congestion.

[0039] The multi-UAV collaborative delivery solution is solved dynamically and in real time. The UAVs learn to make real-time path-finding decisions and information exchange strategies in a dynamic airspace environment based on multi-agent deep reinforcement learning methods, rather than solving the delivery path in advance and then performing path-finding and delivery.

[0040] The training process of the airspace scheduling module jointly considers the influence of the horizontal multi-UAV pathfinding solver. Its training sampling process relies on the pathfinding solution given by the horizontal multi-UAV pathfinding. The horizontal multi-UAV solvers at each altitude layer use multi-agent reinforcement learning methods, centralized training and distributed execution to maximize the individual benefits of the multi-UAV collaborative pathfinding problem, and optimize the global benefits at the same time.

[0041] S105. The drone performs the delivery task according to the altitude decision and the path decision.

[0042] Exemplarily, the task delivery completion path of each drone includes flying from the current location to the delivery starting point to pick up the goods, and flying from the delivery starting point to the delivery destination to release the goods. After completing the current delivery task, the drone will continue to assign new delivery tasks.

[0043] The drone delivery process is ongoing, that is, the drone will be continuously assigned delivery tasks, climb to the assigned altitude and find the path horizontally, and finally land at the destination to complete the delivery task. This process involves task allocation mechanism, airspace coordination, and continuous decision-making and mutual influence of drone pathfinding.

[0044] The low-altitude logistics multi-UAV collaborative delivery method based on deep reinforcement learning of the present invention designs a high-level scheduling strategy for UAV delivery tasks in the vertical dimension, and then coordinates and plans the airspace distribution of UAVs, thereby improving the performance of the distribution system by eliminating airspace congestion; it can eliminate the complexity of the multi-UAV pathfinding problem, so that the UAVs do not need to solve the three-dimensional complex action solution space when finding the path, and can perform pathfinding within a feasible time, thereby realizing an efficient and real-time task allocation mechanism and airspace coordination; it allows each UAV to perform real-time path planning and airspace coordination in a dynamic environment, thereby achieving high-throughput and low-latency delivery goals in complex urban scenarios, and reducing the operational risks brought by airspace conflicts.

[0045] In another implementation of the present invention, the dynamic airspace information includes the location information of the current drone, the layout of obstacles at each layer, the location information of other drones distributed in the airspace, the delivery path and the task execution status.

[0046] Exemplarily, a deep learning network perception module is designed based on a convolutional neural network architecture and a gated recurrent unit, so that each drone can extract drones, obstacles, and heuristic path-finding guidance information within the observable range for state learning when finding its way.

[0047] In another implementation of the present invention, in the airspace scheduling altitude layer selector, the dynamic airspace information is aggregated through an attention layer, and the potential dependencies between drones are learned.

[0048] For example, considering that airspace scheduling in the vertical direction of movement and multi-UAV pathfinding in the horizontal direction both require efficient and rapid decision-making in complex and dynamic air traffic, an algorithm for joint airspace scheduling and multi-UAV collaborative pathfinding based on deep reinforcement learning is proposed, such as Figure 3 As shown in the figure, in the design of the airspace scheduling mechanism, a Markov Decision Process (MDP) is established. The state input is the dynamic airspace information obtained by the distribution center monitoring, including the current drone's delivery task, the location information of other drones in the airspace, and the task execution status. For high-dimensional and dynamic airspace state information, the multi-head attention mechanism (MHA) is used to aggregate the state information. Based on the attention mechanism, the representation of high-dimensional airspace state is learned to extract the implicit association between the current drone to be scheduled and other drones.

[0049] Specifically, the input matrix is ​​first passed through a multi-layer perceptron (MLP) encoder to extract vectorized individual features with a dimension of d k , these embedded features are fed into a hThe attention layer accepts a single query vector Q from the current drone, performs a linear projection, compares it with a key-value set K describing features of other drones, and then calculates a random attention matrix to aggregate a set of output values ​​V.

[0050] Attention layer network training query matrix Key Matrix Sum Matrix

[0051]

[0052] In another implementation of the present invention, the attention layer calculates the weighted attention of the h-th head by the following formula:

[0053]

[0054] Among them, e h is the input vector, d k The dimension of the individual features of the dynamic spatial information vectorized by the encoder, is the query matrix, is the key matrix, is a value matrix.

[0055] Exemplarily, the outputs of all heads are combined through a linear layer to form a summarized weighted attention vector, which represents the implicit correlation between the drone and the airspace state. This vector is added to the output of the autoencoder and serves as the upstream input of the policy network and value network of deep reinforcement learning. In the implementation of the reinforcement learning mechanism, the algorithm structure of the present invention is based on the discretized Soft Actor-Critic (SAC) method, which combines long-term rewards and low-entropy policy penalties, and achieves extensive and robust learning through effective exploration and entropy regularization.

[0056] The network of the present invention includes an actor network (Actor) and two Q-value critic network (Critic) structures. The Actor network outputs the probability distribution of actions and generates random actions through sampling; the Critic network is used to estimate the Q value of the state-action pair to reduce the overestimation bias. During algorithm training, after obtaining a high-dimensional representation of a dynamic and complex spatial domain through the attention mechanism, the strategy network samples actions and interacts with the environment, and records the experience into the playback pool; the Q network is updated using the minimum Q value, and the strategy network is updated by maximizing the sum of the reward and entropy.

[0057] In another implementation of the present invention, the communication vectors between the drones are merged through a bidirectional long short-term memory network, and the attention weight is obtained by the following formula:

[0058]

[0059] Here, i and j represent different drones, FC(·) is the fully connected layer, and Gumble(·) is the Gumbel-Softmax function used to implement back-propagation.

[0060] For example, through the airspace scheduling mechanism, after the drones assigned to the task but not yet performing delivery pathfinding are dispatched to a specific altitude layer, multiple drones in the same altitude layer adopt horizontal collaborative pathfinding, and the multi-agent pathfinding solver solves the action at the next moment based on the current observation.

[0061] In order for the drones to accurately interpret the observation information and cooperate with surrounding drones to further mitigate potential conflicts, the observation and communication structure of each drone’s pathfinding network is designed as follows: Figure 3 The observation matrix can be represented by a binary vector stack as (U, O, E), where U represents the position of the drone within the observation radius, O represents the obstacle distribution, and E is composed of four two-dimensional one-hot vectors stacked together. Each binary element indicates whether the distance to the target point is shortened after the corresponding action is performed at the corresponding position.

[0062] The features of the observation matrix are first extracted through an observation encoder based on Convolutional Neural Networks (CNN), which is composed of an independent convolutional layer, three residual convolution blocks and a gated recurrent unit (GRU) unit stacked together. Subsequently, the drone shares its observation code with nearby drones within a limited radius, using a two-layer attention-based communication structure and sharing it between each drone. In the first layer of communication, the drone identifies other drones within the observation range that are most likely to conflict with itself through a hard attention mechanism.

[0063] In the second phase of communication, a soft attention model is used to further infer the correlation between drones. Its mechanism is similar to the attention aggregator in the airspace scheduling mechanism. When updating, the multi-agent pathfinding solver adopts the multi-agent reinforcement learning method of Counterfactual Multi-Agent Policy Gradient (COMA) to optimize the update. Through the framework of centralized training and distributed execution, the multi-UAV collaborative pathfinding problem is maximized based on individuals and the global benefit is optimized at the same time.

[0064] The communication among multiple UAVs is based on a two-stage attention mechanism, which only selects UAVs with high correlation (high probability of conflict) within the observable range for collaborative communication, thus resolving pathfinding conflicts at a low communication cost.

[0065] In another implementation of the present invention, based on the construction and definition of the above algorithm module, the present invention designs a low-altitude logistics multi-UAV collaborative delivery solution based on deep reinforcement learning as follows:

[0066] 1) Initialize system parameters and status information based on the current distribution of delivery orders, drone distribution, etc.:

[0067] a) Initialize the drone set D at the central apron and define the number of currently dispatchable altitude layers;

[0068] b) Initialize the undelivered task set T U , the assigned task set T A ;

[0069] 2) Multi-drone delivery process:

[0070] a) Repeat;

[0071] b) Select the drone D whose current event is "idle" from the drone collection E ;

[0072] √ Carry out distribution task allocation and solve the UAV-task pair S d,t ,d∈D E ,t∈T U ;

[0073] √ The assigned tasks are removed from the undelivered tasks set T U , join the assigned task set T A ;

[0074] √ Use the multi-head attention mechanism to aggregate the spatial state and generate a spatial representation vector to be passed into the downstream network;

[0075] √ Use the soft actor-critic model to calculate the operating altitude layer of the drone for the current delivery;

[0076] √Update the current drone destination and the drone enters the ascending state;

[0077] c) Traverse each drone in the drone collection;

[0078] √ For all drones that are performing horizontal pathfinding:

[0079] 1. Encode the state based on the locally observed positions of neighboring drones, obstacles, and targets;

[0080] 2. Using the agent mechanism with double-layer soft and hard attention, the state information encoding is transmitted between neighboring UAVs to obtain the collaborative information of neighboring UAVs in the same altitude layer;

[0081] 3. Calculate the path-finding action for the current time step through the MLP decision network;

[0082] 4. Step forward and update the drone’s location information;

[0083] √ For all drones that are moving vertically:

[0084] 1. If there is no conflict, move up or down one level;

[0085] 2. In case of conflict, hover in the air for one time step;

[0086] 3. For the drones that have reached the assigned altitude layer, set the next time step to enter the horizontal pathfinding state;

[0087] 4. For drones that have arrived at their destination, deliver the delivery mission;

[0088] d)Until all delivery tasks are completed.

[0089] The algorithm of joint airspace scheduling and multi-UAV collaborative pathfinding based on deep reinforcement learning proposed in the present invention allows each UAV to perform real-time path planning and airspace coordination in a dynamic environment, thereby achieving high-throughput, low-latency delivery goals in complex urban scenarios and reducing operational risks caused by airspace conflicts.

[0090] In another implementation of the present invention, in order to verify the performance of the distribution system of the present invention, the experimental simulation focuses on the urban center of the city. This scenario has high practical significance in drone delivery services. At the same time, high-rise buildings restrict the already limited airspace and increase the complexity of drone scheduling. Therefore, the challenges faced are also more significant. Figure 4 Two grid-based three-dimensional city maps were simulated to verify the performance of the present invention. Each map covers an area of ​​2500 meters × 2500 meters, and multi-drone delivery services are deployed in the airspace below 80 meters. In order to study the impact of building density on drone delivery efficiency, the two maps are named sparse and dense, and the building coverage is set to 21% and 38%, respectively. The physical grid size is set to 25 meters × 25 meters, and the allowed height layer division is set to 4 layers to stratify the randomly generated buildings into obstacles in the height dimension. The number of drones is set to 32 and 64, representing the airspace traffic flow during normal and busy hours, respectively.

[0091] In scenarios with different air traffic pressures, the proposed solution is compared with baselines such as Equal-Distributed Astar (EDA, evenly distributing drones to various altitude layers and then performing A* pathfinding), Nearest-Layer-First Search (NFS, pathfinding with the nearest altitude layer first for the delivery path), and Soft Actor-Critic (SAC, no attention mechanism, using soft actor-critic reinforcement learning for airspace scheduling).

[0092] Figure 5 It is the impact of different altitude layer selection strategies on the performance of the UAV collaborative pathfinding module in the experimental simulation. The results show that in various scenarios, the solution of the present invention can further understand the attention association of UAVs in complex dynamic airspace in a high-dimensional way, and take into account the shorter delivery distance on the basis of ensuring that the altitude layer distribution of UAVs is not overly crowded, thereby significantly reducing the average delivery step (Average Step, AS) and the maximum delivery step (Maximum Step, MS), effectively reducing the conflict (Average Conflict, AC) between UAVs, and avoiding altitude layer congestion.

[0093] Figure 6 and Figure 7 They are the comparison of the average task throughput of continuous delivery of different delivery systems and the delivery delay rate. The joint coordination strategy of the present invention can effectively empower the delivery system, and can achieve the highest delivery throughput and the lowest delivery delay rate in each continuous delivery scenario. The advantage of this coordination is more prominent in scenarios with busier air traffic and denser urban buildings, because busy air traffic and more limited airspace resources highlight the necessity of overall airspace coordination and drone coordination.

[0094] Another aspect of the present invention provides a low-altitude logistics multi-UAV collaborative delivery system based on deep reinforcement learning, comprising:

[0095] Task allocation module: used to obtain delivery tasks and multi-level dynamic airspace information, and allocate the delivery tasks to idle drones according to the task allocation mechanism.

[0096] The airspace scheduling module is used to construct an airspace scheduling mechanism based on the multi-level dynamic airspace information, wherein the airspace scheduling mechanism includes an airspace scheduling altitude layer selector and a multi-UAV collaborative path finding solver; the airspace scheduling altitude layer selector determines the flight altitude layer of the UAV according to the delivery task and the multi-level dynamic airspace information to obtain an altitude layer decision; the multi-UAV collaborative path finding solver performs collaborative path planning in the horizontal range at each altitude layer to obtain a path decision.

[0097] Distribution module: used for the UAV to perform the distribution task according to the altitude decision and the path decision.

[0098] In the scenario of low-altitude logistics, this scheme designs a multi-UAV collaborative delivery framework based on the multi-level airspace concept to address the challenges brought by the computational complexity of multi-UAV three-dimensional airspace pathfinding in multi-UAV collaborative delivery. The framework includes modules such as dynamic task allocation, airspace scheduling, and multi-UAV dynamic pathfinding solvers at each airspace layer. It realizes real-time and continuous UAV delivery with low-complexity calculations, providing a framework foundation for efficient and rapid collaborative scheduling and path planning of multiple UAVs in urban environments.

[0099] In order to effectively avoid airspace congestion, the airspace scheduling module is responsible for realizing horizontal collaborative path planning and information sharing of drones within the altitude layer according to the current drone status and airspace traffic distribution. The multi-drone collaborative path-finding solver learns the path-finding state representation through the convolutional neural network, and uses the two-stage attention communication mechanism to share information and collaboratively resolve the path-finding conflict when the routes intersect. The learning of multi-drone collaborative strategies is realized through multi-agent reinforcement learning, and the current delivery drone selects a low congestion probability and a short delivery path altitude layer to reduce airspace congestion. The implicit association between drones is extracted through the multi-head attention mechanism, and deep reinforcement learning is used to achieve efficient decision-making for complex high-dimensional airspace, improve the collaborative ability of multiple drones in dynamic and complex airspace, and further improve the distribution efficiency of the system. The low-altitude distribution scheme proposed by the present invention enables the distribution system to continuously complete multi-drone logistics tasks, with a higher task throughput and a lower delivery delay rate.

[0100] The method of the present invention can be applied in industrial control systems in industries such as manufacturing, energy, and transportation, and can enhance the system's anti-attack capability by providing customized security services.

[0101] In another aspect of the present invention, an electronic device includes: a processor, a memory, a communication bus, and a communication interface.

[0102] in:

[0103] The processor, memory and communication interface communicate with each other through a communication bus.

[0104] Communication interface, used to communicate with other electronic devices or servers.

[0105] The processor is used to execute the program, and specifically can execute the steps of any one of the low-altitude logistics multi-UAV collaborative delivery methods based on deep reinforcement learning in the above-mentioned embodiments.

[0106] Specifically, the program may include program codes including computer operation instructions.

[0107] The processor may be a central processing unit (CPU), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application. The one or more processors included in the smart device may be processors of the same type, such as one or more CPUs; or processors of different types, such as one or more CPUs and one or more ASICs.

[0108] The memory is used to store programs. The memory may include a high-speed RAM memory, and may also include a non-volatile memory, such as at least one disk memory.

[0109] The program can be specifically used to enable the processor to execute the steps of any one of the low-altitude logistics multi-drone collaborative distribution methods based on deep reinforcement learning described in the embodiment. The specific implementation of each step in the program can refer to the corresponding descriptions in the steps and units executed by any one of the low-altitude logistics multi-drone collaborative distribution methods based on deep reinforcement learning in the above steps, which will not be repeated here. Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working process of the above-described equipment and modules can refer to the corresponding process description in the aforementioned method embodiment.

[0110] The exemplary embodiments of the present application also provide a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable a computer to execute the methods of the various embodiments of the present application.

[0111] The above-described method according to an embodiment of the present invention may be implemented in hardware, firmware, or as software or computer code that may be stored in a recording medium (such as a CD ROM, RAM, floppy disk, hard disk, or magneto-optical disk), or as computer code that is originally stored in a remote recording medium or a non-temporary machine-readable medium downloaded over a network and will be stored in a local recording medium, so that the method described herein may be stored in such software processing on a recording medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware (such as an ASIC or FPGA). It is understood that a computer, processor, microprocessor controller, or programmable hardware includes a storage component (e.g., RAM, ROM, flash memory, etc.) that can store or receive software or computer code, and when the software or computer code is accessed and executed by a computer, processor, or hardware, the method described herein is implemented. In addition, when a general-purpose computer accesses the code for implementing the method shown herein, the execution of the code converts the general-purpose computer into a dedicated computer for executing the method shown herein.

[0112] Thus far, specific embodiments of the present invention have been described. Other embodiments are within the scope of the appended claims. In some cases, the actions recited in the claims may be performed in a different order and still achieve the desired results. Additionally, the processes depicted in the accompanying drawings do not necessarily require the specific order or sequential order shown to achieve the desired results. In some embodiments, multitasking and parallel processing may be advantageous.

[0113] It should be noted that all directional indications in the embodiments of the present invention (such as up, down, left, right, back, etc.) are only used to explain the relative position relationship, movement status, etc. between the components under a certain specific posture (as shown in the accompanying drawings). If the specific posture changes, the directional indication will also change accordingly.

[0114] In the description of the present invention, the terms "first" and "second" are only used to facilitate the description of different components or names, and cannot be understood as indicating or implying a sequential relationship, relative importance, or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features.

[0115] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art of the present invention. The terms used in the specification of the present invention herein are only for the purpose of describing specific embodiments and are not intended to limit the present invention.

[0116] It should be noted that although the specific embodiments of the present invention are described in detail in conjunction with the accompanying drawings, it should not be understood as limiting the scope of protection of the present invention. Within the scope described in the claims, various modifications and variations that can be made by those skilled in the art without creative work still belong to the scope of protection of the present invention.

[0117] The examples of the embodiments of the present invention are intended to concisely illustrate the technical features of the embodiments of the present invention so that those skilled in the art can intuitively understand the technical features of the embodiments of the present invention, and are not intended to be improper limitations of the embodiments of the present invention.

[0118] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A low-altitude logistics multi-UAV collaborative delivery method based on deep reinforcement learning, characterized in that: include: Obtaining delivery tasks and multi-level dynamic airspace information, and assigning the delivery tasks to idle drones according to a task assignment mechanism; Based on the multi-level dynamic airspace information, an airspace scheduling mechanism is constructed, wherein the airspace scheduling mechanism includes an airspace scheduling altitude layer selector and a multi-UAV collaborative pathfinding solver; The airspace scheduling altitude layer selector determines the flight altitude layer of the UAV according to the delivery task and the multi-level dynamic airspace information to obtain an altitude layer decision; The multi-UAV collaborative path-finding solver performs collaborative path planning in a horizontal range at each altitude layer to obtain a path decision; The drone performs the delivery task according to the altitude decision and the path decision.

2. The method according to claim 1, characterized in that The dynamic airspace information includes the location information of the current drone, the layout of obstacles at each layer, the location information of other drones distributed in the airspace, the delivery path and the task execution status.

3. The method according to claim 2, characterized in that In the airspace scheduling altitude layer selector, the dynamic airspace information is aggregated through the attention layer, and the potential dependencies between drones are learned.

4. The method according to claim 3, characterized in that The attention layer calculates the weighted attention of the h-th head by the following formula: Among them, e h is the input vector, d k The dimension of the individual features of the dynamic spatial information vectorized by the encoder, is the query matrix, is the key matrix, is a value matrix.

5. The method according to claim 1, characterized in that Also includes: The communication vectors between the drones are merged through a bidirectional long short-term memory network, and the attention weight is obtained by the following formula: Here, i and j represent different drones, FC(·) is the fully connected layer, and Gumble(·) is the Gumbel-Softmax function used to implement back-propagation.

6. A low-altitude logistics multi-UAV collaborative delivery system based on deep reinforcement learning, characterized in that: include: Task allocation module: used to obtain delivery tasks and multi-level dynamic airspace information, and allocate the delivery tasks to idle drones according to the task allocation mechanism; An airspace scheduling module is used to construct an airspace scheduling mechanism based on the multi-level dynamic airspace information, wherein the airspace scheduling mechanism includes an airspace scheduling altitude layer selector and a multi-UAV collaborative path finding solver; the airspace scheduling altitude layer selector determines the flight altitude layer of the UAV according to the delivery task and the multi-level dynamic airspace information to obtain an altitude layer decision; the multi-UAV collaborative path finding solver performs collaborative path planning in a horizontal range at each altitude layer to obtain a path decision; Distribution module: used for the UAV to perform the distribution task according to the altitude decision and the path decision.

Images