Logistics unmanned aerial vehicle take-off and landing point selection method, device, equipment, medium and product

Abstract

The invention discloses a logistics unmanned aerial vehicle take-off and landing point selection method, device and equipment, a medium and a product, and relates to the technical field of unmanned aerial vehicle take-off and landing point planning. According to the method, after a solution of a logistics unmanned aerial vehicle take-off and landing point selection problem under the condition of a given instance is obtained, the given instance serves as a model input item, the solution serves as a model output item, a model adopting a deep reinforcement learning method is imported for model training, and an unmanned aerial vehicle take-off and landing point selection model is obtained; and then, the coordinates of all logistics demand points of the target area and the logistics demand quantity are imported into an unmanned aerial vehicle take-off and landing point selection model, and the coordinates of the selected unmanned aerial vehicle take-off and landing points are output to serve as a logistics unmanned aerial vehicle take-off and landing point selection result of the target area, so that the problem of complex logistics unmanned aerial vehicle take-off and landing point site selection decision can be solved. Therefore, distribution with logistics requirements can be realized with low cost, and practical application and popularization are facilitated.

Description

Technical Field

[0001] The present invention belongs to the technical field of drone take-off and landing point planning, and specifically relates to a method, device, equipment, medium and product for selecting a logistics drone take-off and landing point. Background Art

[0002] Drone logistics refers to the delivery of express deliveries from other cities or regions to drone take-off and landing points, and then to customers or their communities by drone delivery, completing the last step of the logistics supply chain. The distribution of logistics demand points and demand volume, as well as the set of candidate sites for drone take-off and landing points in a certain area are known. Logistics drones are used as transportation tools, and the following actual situations (A) to (E) are considered:

[0003] (A) The number of drone take-off and landing points is constrained, that is, at least one site needs to be selected from the set of candidate sites for drone take-off and landing points as the drone take-off and landing point;

[0004] (B) Distribution relationship constraints, that is, no logistics demand point is allowed to be omitted or unmet, so for any logistics demand point, there must be at least one corresponding drone take-off and landing point responsible for delivery;

[0005] (C) Service range constraints of the take-off and landing points, that is, the take-off and landing points of logistics drones can only cover a certain range of areas, that is, they can only provide logistics distribution services to logistics demand points within a certain range. This service range is usually subject to the maximum flight range of the logistics drone or the endurance of the drone: for any pair of logistics demand points and drone take-off and landing points, if there is a distribution relationship, the distribution distance shall not exceed the maximum service range of the drone take-off and landing point;

[0006] (D) Airspace environment restrictions, that is, there are many restrictive factors in low-altitude airspace (especially urban low-altitude airspace), such as restricted flight zones and no-fly zones, which restrict the specifications of logistics drones or prohibit any type of drone from operating. From the perspective of drone take-off and landing point layout planning, the take-off and landing points may take off and land multiple types of logistics drones in the future. Therefore, restricted airspace such as restricted flight zones and no-fly zones are uniformly designated as no-fly zones, and take-off and landing points are strictly prohibited from being set in no-fly zones, and all demand points in no-fly zones will be eliminated. In addition, drone delivery routes should also be strictly prohibited from crossing no-fly zones;

[0007] (E) The planning of drone take-off and landing points needs to consider the construction cost of the take-off and landing points as well as the logistics distribution cost. When calculating the construction cost, the differences in construction costs in different regions should also be considered. The distribution cost is also an issue that must be considered in the layout planning of logistics facilities. Assuming that the transportation cost is proportional to the distribution distance, the closer the distance to the logistics facility point, the lower the distribution cost.

[0008] How to obtain the distribution for realizing logistics demand at low cost and Figure 1The layout planning of the logistics drone take-off and landing points shown is a topic that technical personnel in this field urgently need to study. Summary of the invention

[0009] The purpose of the present invention is to provide a method, device, computer equipment, computer-readable storage medium and computer program product for selecting a take-off and landing point for a logistics drone, so as to solve the complex problem of site selection decision for the take-off and landing points of logistics drones, so as to achieve the distribution of logistics needs at a low cost.

[0010] In order to achieve the above object, the present invention adopts the following technical solutions:

[0011] In the first aspect, a method for selecting a take-off and landing point for a logistics drone is provided, comprising:

[0012] Obtain a solution to the problem of selecting a take-off and landing point for a logistics drone in a given instance, wherein the given instance includes coordinates of n′ given logistics demand points and logistics demand quantities, and the solution includes coordinates of p drone take-off and landing points selected from the n′ logistics demand points, wherein n′ represents a positive integer greater than or equal to 3, and p represents a positive integer less than n′;

[0013] The given instance is used as a model input item, and the solution is used as a model output item, and is imported into a model using a deep reinforcement learning method for model training, so as to obtain a UAV take-off and landing point selection model;

[0014] The coordinates of n″ logistics demand points and logistics demand quantities in the target area are imported into the drone take-off and landing point selection model, and the coordinates of p drone take-off and landing points selected from the n″ logistics demand points are output as the logistics drone take-off and landing point selection result for the target area, wherein n″ represents a positive integer greater than p.

[0015] Based on the above invention content, a new solution for selecting logistics drone take-off and landing points based on a deep reinforcement learning method is provided, that is, after obtaining a solution to the problem of selecting logistics drone take-off and landing points under a given instance, the given instance is used as a model input item, and the solution is used as a model output item, and a model using a deep reinforcement learning method is imported for model training to obtain a drone take-off and landing point selection model, and then the coordinates of all logistics demand points and logistics demand in the target area are imported into the drone take-off and landing point selection model, and the coordinates of the selected drone take-off and landing points are output as the logistics drone take-off and landing point selection results in the target area. In this way, the complex logistics drone take-off and landing point site selection decision problem can be solved, so as to realize the distribution of logistics needs at a low cost, which is convenient for practical application and promotion.

[0016] In one possible design, the policy network in the model using the deep reinforcement learning method is used to select a different logistics demand point from the input n logistics demand points as a drone take-off and landing point at each time step until p drone take-off and landing points are selected, and the remaining logistics demand points are allocated to the nearest drone take-off and landing points, and then the final objective function is calculated to provide a feasible solution to the logistics drone take-off and landing point selection problem, where n represents a positive integer greater than or equal to 3;

[0017] When training the model, the training goal is to try to maximize or minimize the objective function based on the drone take-off and landing points selected at each step, and use the deep reinforcement learning method to update the model parameters.

[0018] In one possible design, the objective function is expressed using the following formula to be minimized:

[0019]

[0020] In the formula, Cost represents the total cost, j represents a positive integer less than or equal to n, represents the jth logistics demand point, f j (C) represents the construction cost coefficient of building the j-th logistics demand point into a drone take-off and landing point, α represents the total number of regions, D1, D2, …, D α They represent different regions and there is no intersection between any two regions, c1, c2, …, c α Represent the construction cost coefficients of different regions, y j Indicates whether to select the jth logistics demand point as the drone take-off and landing point and use y j =1 means select and use y j =0 means no selection, c means unit construction fixed cost, i means a positive integer less than or equal to n, represents the unit mileage delivery cost of the logistics drone corresponding to the j-th logistics demand point when the j-th logistics demand point is used as the drone take-off and landing point, x ij Indicates whether the i-th logistics demand point is allocated to the drone take-off and landing point when the j-th logistics demand point is used as the drone take-off and landing point, and x ij =1 means allocation and use x ij =0 means no allocation, d ij N represents the distance from the i-th logistics demand point to the drone take-off and landing point when the j-th logistics demand point is used as the drone take-off and landing point. i represents the logistics demand of the j-th logistics demand point;

[0021] The constraints of the objective function are expressed as follows:

[0022]

[0023] In the formula, A ij Indicates whether there is a no-fly zone on the path from the i-th logistics demand point to the drone take-off and landing point when the j-th logistics demand point is used as the drone take-off and landing point, and uses A ij =0 means existence and A ij =1 means it does not exist. Indicates the maximum range of the logistics drone with full load, Indicates the maximum range of the logistics drone with no payload.

[0024] In one possible design, the neural network core in the model using the deep reinforcement learning method includes an interconnected encoder and a decoder, wherein the encoder is responsible for converting the original input features into a high-dimensional embedding representation, and the decoder is used to make a decision based on the high-dimensional embedding representation: selecting a drone take-off and landing point from n input logistics demand points.

[0025] In a possible design, the high-dimensional embedding representation is divided into a static embedding part and a dynamic embedding part, wherein the static embedding part is used to represent the two-dimensional coordinate information of each logistics demand point, and the dynamic embedding part represents that the node state of each logistics demand point changes continuously with the change of coverage;

[0026] The encoder includes a static embedding processing part and a dynamic embedding processing part;

[0027] The static embedding processing part generates the static embedding of each logistics demand point in the following manner:

[0028] After a layer of linear mapping, the original input coordinates of each logistics demand point are mapped to a high-dimensional static initial embedding:

[0029] f i (0) =W (0) × i +b (0)

[0030] In the formula, i is a positive integer less than or equal to n, and x is i represents the original input coordinates of the i-th logistics demand point, f i (0) represents the high-dimensional static initial embedding corresponding to the i-th logistics demand point, W (0) represents the first learnable parameter, b (0) represents the second learnable parameter;

[0031] For each logistics demand point, a final and corresponding high-dimensional static embedding is generated through N multi-head attention layers, wherein the high-dimensional static embedding calculation results of each multi-head attention layer in the N multi-head attention layers are as follows:

[0032]

[0033] In the formula, k is a positive integer less than or equal to N, and f i (k) represents the high-dimensional static embedding calculation result of the k-th multi-head attention layer corresponding to the i-th logistics demand point, BN() represents the function form of the normalization layer, FF() represents the function form of the feedforward layer, and f i (k-1) represents the high-dimensional static embedding calculation result of the k-th multi-head attention layer corresponding to the i-th logistics demand point, MHA() represents the function form of the multi-head attention layer, and the final high-dimensional static embedding corresponding to the i-th logistics demand point is f i (N) ;

[0034] The dynamic embedding processing part updates the dynamic embedding of each logistics demand point in the following manner:

[0035] Initializing the dynamic coverage vector of each logistics demand point and used to reflect the coverage status of the corresponding point to 1;

[0036] For each logistics demand point, at each step in the decoding process, the corresponding dynamic coverage vector is dynamically updated according to the following formula:

[0037]

[0038] In the formula, t is a positive integer less than or equal to p, and p is t represents the take-off and landing point of the UAV selected in the t-th step, NCr(p t ) means that in the set Cr(p t ) in the set Cr(p t ) means at the drone take-off and landing point p t is the collection of logistics demand points covered by the center and the preset coverage radius, c i When the i-th logistics demand point belongs to the set Cr(p t ) represents the sorting index of the i-th logistics demand point in the node sequence, and the node sequence refers to the order from the logistics demand point to the drone take-off and landing point p t The order of distance from near to far is the same as the set Cr(p t ) are sorted in sequence, c iWhen the i-th logistics demand point does not belong to the set Cr(p t ) takes the value of NCr(p t ), I i,t represents the dynamic coverage vector corresponding to the i-th logistics demand point updated in the t-th step of the decoding process, I i,t-1 represents the dynamic coverage vector corresponding to the i-th logistics demand point updated in the t-1th step of the decoding process, and I i,0 =1;

[0039] For each logistics demand point, the final dynamic embedding is calculated according to the following formula:

[0040] H i,t =I i,t ×W I

[0041] In the formula, H i,t represents the dynamic embedding corresponding to the i-th logistics demand point and finally obtained in the t-th step of the decoding process, W I represents a learnable third parameter.

[0042] In one possible design, the decoder includes an RNN part for outputting query results and an attention structure part for calculating the probability of each logistics demand point being selected;

[0043] The attention structure part calculates the selection probability of each logistics demand point as follows:

[0044] For each logistics demand point, in the tth step, the query result q is generated by the RNN part and the self-attention mechanism t Then, combined with the high-dimensional embedding representation obtained from the encoder, the corresponding vector is calculated according to the following formula:

[0045]

[0046] In the formula, represents the vector corresponding to the i-th logistics demand point and calculated in the t-th step, T represents the matrix transposition symbol, d h represents the total number of dimensions of the hidden layer, M represents the number of self-attentions in the multi-head attention layer, and W k represents the fourth learnable parameter;

[0047] For each logistics demand point, if the corresponding point belongs to the set of drone take-off and landing points selected after the first t-1 steps, the corresponding vector is corrected to -1, otherwise the corresponding vector is not corrected;

[0048] For each logistics demand point, the corresponding selection probability is calculated according to the following formula:

[0049]

[0050] Where η i,t represents the probability of being selected corresponding to the i-th logistics demand point at the t-th step, Represents the vector corresponding to the j-th logistics demand point and calculated in the t-th step.

[0051] In a second aspect, a logistics UAV take-off and landing point selection device is provided, including a solution acquisition unit, a selection model training unit, and a selection model application unit which are sequentially connected in communication;

[0052] The solution acquisition unit is used to acquire a solution to the problem of selecting a take-off and landing point for a logistics drone in a given instance, wherein the given instance includes coordinates of n′ given logistics demand points and logistics demand quantities, and the solution includes coordinates of p drone take-off and landing points selected from the n′ logistics demand points, wherein n′ represents a positive integer greater than or equal to 3, and p represents a positive integer less than n′;

[0053] The selected model training unit is used to use the given instance as a model input item and the solution as a model output item, import the model using the deep reinforcement learning method for model training, and obtain the drone take-off and landing point selection model;

[0054] The selection model application unit is used to import the coordinates and logistics demand quantities of n″ logistics demand points in the target area into the drone take-off and landing point selection model, and output the coordinates of p drone take-off and landing points selected from the n″ logistics demand points as the logistics drone take-off and landing point selection result for the target area, wherein n″ represents a positive integer greater than p.

[0055] In a third aspect, the present invention provides a computer device comprising a memory, a processor and a transceiver which are communicatively connected in sequence, wherein the memory is used to store computer programs, the transceiver is used to send and receive messages, and the processor is used to read the computer program to execute the method for selecting take-off and landing points for a logistics drone as described in the first aspect or any possible design of the first aspect.

[0056] In a fourth aspect, the present invention provides a computer-readable storage medium having instructions stored thereon. When the instructions are executed on a computer, the method for selecting take-off and landing points for a logistics drone as described in the first aspect or any possible design of the first aspect is executed.

[0057] In a fifth aspect, the present invention provides a computer program product, including a computer program or instructions, which, when executed by a computer, implements the method for selecting take-off and landing points for a logistics drone as described in the first aspect or any possible design of the first aspect.

[0058] Beneficial effects of the above scheme:

[0059] (1) The present invention provides a new solution for selecting the take-off and landing points of logistics drones based on a deep reinforcement learning method. That is, after obtaining a solution to the problem of selecting the take-off and landing points of logistics drones under a given instance, the given instance is used as a model input item, and the solution is used as a model output item. A model using a deep reinforcement learning method is imported for model training to obtain a drone take-off and landing point selection model. Then, the coordinates of all logistics demand points and logistics demand in the target area are imported into the drone take-off and landing point selection model, and the coordinates of the selected drone take-off and landing points are output as the selection results of the logistics drone take-off and landing points in the target area. In this way, the complex logistics drone take-off and landing point site selection decision problem can be solved, so as to realize the distribution of logistics demand at a low cost, which is convenient for practical application and promotion. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0061] Figure 1 Schematic diagram of the layout planning of logistics drone take-off and landing points.

[0062] Figure 2 A flow chart of a method for selecting take-off and landing points for a logistics drone provided in an embodiment of the present application.

[0063] Figure 3 A schematic diagram of the structure of the policy network provided in an embodiment of the present application.

[0064] Figure 4 A schematic diagram of the structure of a neural network provided in an embodiment of the present application.

[0065] Figure 5 A schematic diagram of the structure of a logistics drone take-off and landing point selection device provided in an embodiment of the present application.

[0066] Figure 6 A schematic diagram of the structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0067] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the present invention will be briefly introduced below in combination with the drawings and the description of the embodiments or the prior art. Obviously, the following description of the structures of the drawings is only some embodiments of the present invention. For ordinary technicians in this field, other embodiments can be obtained based on these embodiments without creative work. It should be noted that the description of these embodiments is used to help understand the present invention, but does not constitute a limitation of the present invention.

[0068] It should be understood that although the terms first and second, etc. may be used herein to describe various objects, these objects should not be limited by these terms. These terms are only used to distinguish one object from another object. For example, a first object can be referred to as a second object, and similarly, a second object can be referred to as a first object without departing from the scope of the exemplary embodiments of the present invention.

[0069] It should be understood that the term "and / or" that may appear in this document is merely a description of the association relationship between associated objects, indicating that there may be three relationships. For example, A and / or B can indicate three situations: A exists alone, B exists alone, or A and B exist at the same time. For another example, A, B and / or C can indicate the existence of any one of A, B and C or any combination of them. The term " / and" that may appear in this document describes another type of association object relationship, indicating that there may be two relationships. For example, A / and B can indicate two situations: A exists alone or A and B exist at the same time. In addition, the character " / " that may appear in this document generally indicates that the previous and next associated objects are in an "or" relationship.

[0070] Example

[0071] like Figure 2 As shown, the method for selecting the take-off and landing points of the logistics drone provided in the first aspect of this embodiment can be, but is not limited to, executed by a computer device with certain computing resources, such as a cloud server, a personal computer (PC, a multi-purpose computer with a size, price and performance suitable for personal use; desktops, laptops, small laptops, tablets and ultrabooks are all personal computers), a smart phone, a personal digital assistant (PDA) or a wearable device. Figure 2 As shown, the method for selecting the take-off and landing points of the logistics drone may include but is not limited to the following steps S1 to S3.

[0072] S1. Obtain a solution to the problem of selecting logistics drone take-off and landing points under a given instance, wherein the given instance includes but is not limited to the coordinates of given n′ logistics demand points and logistics demand quantities, and the solution includes but is not limited to the coordinates of p drone take-off and landing points selected from the n′ logistics demand points, wherein n′ represents a positive integer greater than or equal to 3, and p represents a positive integer less than n′.

[0073] In step S1, the given example and the solution can be conventionally extracted from sample data. In this embodiment, it is assumed that all demand points are candidate points, so that p points can be selected as drone take-off and landing points.

[0074] S2. Using the given instance as a model input item and the solution as a model output item, importing a model using a deep reinforcement learning method for model training, and obtaining a UAV take-off and landing point selection model.

[0075] In step S2, the deep reinforcement learning method (DRL) is an artificial intelligence method that combines the perception ability of deep learning and the decision-making ability of reinforcement learning. It can be controlled directly according to the input data and is an artificial intelligence method that is closer to human thinking. Deep reinforcement learning mainly consists of the following parts: (1) reinforcement learning (RL), which is a machine learning method that allows the intelligent agent to learn the optimal behavior strategy through interaction with the environment; (2) deep learning (DL), which is a machine learning model that uses multi-layer neural networks to learn data representation and abstraction; (3) policy network, which is used to predict the action taken in a given state; (4) value network, which is used to predict the expected return of taking a certain strategy in a given state; (5) experience replay, which is a technology used to stabilize the training process; (6) target network, which is used to reduce the instability in the training process. At present, deep reinforcement learning has been widely used in many fields such as games, autonomous driving, robotics, finance and medicine. Therefore, this embodiment can use deep reinforcement learning to optimize the logistics drone take-off and landing point selection strategy to solve complex site selection decision problems.

[0076] In step S2, this embodiment mainly applies the REINFORCE algorithm, which is a policy gradient method based on the Monte Carlo method, which aims to maximize the expected return by optimizing the policy function (which takes the state as input and outputs the probability distribution of the action). Its goal is to find the optimal strategy to maximize the expected return on this strategy. In order to facilitate the use of reinforcement learning for training and optimization, the problem of selecting the take-off and landing points for logistics drones can be restated as a Markov decision process (MDP). Specifically, the p-median is modeled as an MDP. MDP is usually composed of five elements: state, action, state transition matrix, reward, and discount factor. In this embodiment, the problem of selecting the take-off and landing points for logistics drones can be restated as follows:

[0077] (A) State: State S T =(M T ) represents the solution set of the current step T in the construction process, where M T Contains all the drone take-off and landing points selected up to time step T, and the initial state M0 is an empty solution set.

[0078] (B) Action: In the process of building a solution, each step requires selecting a node as the take-off and landing point of the drone. Action a T represents the selection of a node at time step T.

[0079] (C) State Transition: State transition means that in state S T Next and action a T Then enter the next state S T+1 =M T+1 =(M T ,z j ), that is, the solution to the next state is determined by the current state S T and action z j Made of stitching.

[0080] (D) Discount Factor: The discount factor is set to 1, that is, when calculating the reward function, the objective function is directly used as the final reward, and the cumulative reward is not calculated.

[0081] (E) Reward: The reward function determines the optimization direction of the policy network. Different spatial optimization problems have different optimization goals, so different reward functions need to be set. For example, for the p-median problem, the reward function can be the negative value of the total weighted distance. Specifically, the objective function of this embodiment is to minimize the total weighted distance.

[0082] In step S2, specifically, the policy network in the model using the deep reinforcement learning method is used to select a different logistics demand point from the input n logistics demand points as a drone take-off and landing point at each time step until p drone take-off and landing points are selected, and the remaining logistics demand points are allocated to the nearest drone take-off and landing points, and then the final objective function is calculated to provide a feasible solution to the logistics drone take-off and landing point selection problem, where n represents a positive integer greater than or equal to 3. The specific structure of the aforementioned policy network is as follows Figure 3 As shown, if the strategy is defined as π θ , then the aforementioned strategy logic can be expressed by the following formula:

[0083]

[0084] In the formula, s represents a given instance, p t represents the take-off and landing point of the UAV selected in step t, p 1:t-1 represents the take-off and landing point of the UAV selected from step 1 to step t-1. In detail, the objective function is expressed by the following formula to be minimized:

[0085]

[0086] In the formula, Cost represents the total cost, j represents a positive integer less than or equal to n, represents the jth logistics demand point, f j (C) represents the construction cost coefficient of building the j-th logistics demand point into a drone take-off and landing point, α represents the total number of regions, D1, D2, …, D α They represent different regions and there is no intersection between any two regions, c1, c2, …, c α Represent the construction cost coefficients of different regions, y j Indicates whether to select the jth logistics demand point as the drone take-off and landing point and use y j =1 means select and use y j =0 means no selection, c means unit construction fixed cost, i means a positive integer less than or equal to n, represents the unit mileage delivery cost of the logistics drone corresponding to the j-th logistics demand point when the j-th logistics demand point is used as the drone take-off and landing point, x ijIndicates whether the i-th logistics demand point is allocated to the drone take-off and landing point when the j-th logistics demand point is used as the drone take-off and landing point, and x ij =1 means allocation and use x ij =0 means no allocation, d ij N represents the distance from the i-th logistics demand point to the drone take-off and landing point when the j-th logistics demand point is used as the drone take-off and landing point. i represents the logistics demand of the j-th logistics demand point; the constraint condition of the objective function is expressed as follows:

[0087]

[0088] In the formula, A ij Indicates whether there is a no-fly zone on the path from the i-th logistics demand point to the drone take-off and landing point when the j-th logistics demand point is used as the drone take-off and landing point, and uses A ij =0 means existence and A ij =1 means it does not exist. Indicates the maximum range of the logistics drone with full load, Indicates the maximum range of the logistics drone with no load. That is the number of take-off and landing points for drones; the aforementioned That is the distribution relationship constraint condition; the aforementioned A ij -x ij ≥0 is the airspace environment constraint condition, where A ij Used as an airspace restriction variable to strictly prohibit delivery routes from crossing no-fly zones; the aforementioned That is, the service range constraint of the take-off and landing point, among which It represents the maximum service range R, which can be further determined by the performance parameters of the drone. That is, the energy consumption of the logistics drone is proportional to the delivery distance and load of the drone. Considering that the logistics drone needs to fly to the demand point with the package and return empty, the maximum service range R of the take-off and landing point can be deduced by the following formula: The aforementioned is the construction cost of the drone take-off and landing point, where f j The value of (C) is affected by the j-th logistics demand point Low price, house price or rental price in the city, etc., non-overlapping D1, D2, ..., D α That is, it constitutes the total area D, so that when calculating the fixed cost of building the logistics drone take-off and landing point, the construction cost differences in different areas are also taken into account. That is, the delivery cost of the drone take-off and landing points, which obeys the principle that the demand of the demand point can be used as the demand weight of the service point, and priority is given to serving the demand points with large weights.

[0089] In the step S2, specifically, the neural network core in the model using the deep reinforcement learning method includes but is not limited to an interconnected encoder and decoder, wherein the encoder is responsible for converting the original input features into a high-dimensional embedding representation, and the decoder is used to make a decision based on the high-dimensional embedding representation: select the drone take-off and landing points from the input n logistics demand points. In detail, the high-dimensional embedding representation is divided into a static embedding part and a dynamic embedding part, wherein the static embedding part is used to represent the two-dimensional coordinate information of each logistics demand point, and the dynamic embedding part represents that the node state of each logistics demand point changes continuously with the change of coverage, that is, the dynamic feature represents the coverage information.

[0090] In detail, the encoder includes but is not limited to a static embedding processing part and a dynamic embedding processing part, wherein, Figure 2 As shown, the static embedding processing part includes two parts: node embedding and multi-head attention (MHA) layer, and the static embedding of each logistics demand point can be generated according to the following steps S211 to S212.

[0091] S211. Map the original input coordinates of each logistics demand point to a high-dimensional static initial embedding through a layer of linear mapping:

[0092] f i (0) =W (0) × i +b (0)

[0093] In the formula, i is a positive integer less than or equal to n, and x is i represents the original input coordinates of the i-th logistics demand point, f i (0) represents the high-dimensional static initial embedding corresponding to the i-th logistics demand point, W (0) represents the first learnable parameter, b (0) represents the second learnable parameter.

[0094] In step S211, the original input coordinates are two-dimensional data, and the total number of dimensions of the high-dimensional static initial embedding can be 128 dimensions, for example.

[0095] S212. For each logistics demand point, generate a final and corresponding high-dimensional static embedding through N multi-head attention layers, wherein the high-dimensional static embedding calculation results of each multi-head attention layer in the N multi-head attention layers are as follows:

[0096]

[0097] In the formula, k is a positive integer less than or equal to N, and f i (k) represents the high-dimensional static embedding calculation result of the k-th multi-head attention layer corresponding to the i-th logistics demand point, BN() represents the function form of the normalization layer, FF() represents the function form of the feedforward layer, and f i (k-1) represents the high-dimensional static embedding calculation result of the k-th multi-head attention layer corresponding to the i-th logistics demand point, MHA() represents the function form of the multi-head attention layer, and the final high-dimensional static embedding corresponding to the i-th logistics demand point is f i (N) .

[0098] In step S212, Figure 4 As shown, each attention layer consists of an MHA layer and a feedforward neural network; each sublayer is connected to a skip connection structure and a normalization layer. In other words, for each sublayer neural network, given an input x, there is BN(x+sublayer(x)). Based on the aforementioned steps S211 to S212, after the original features are input into the network, a layer of linear mapping is first performed to obtain node embedding, and then N layers of attention layers are passed to finally generate a static embedding for each point.

[0099] This embodiment introduces dynamic coverage information as a dynamic feature. Specifically, there are two methods for processing coverage information: the first is that if a node is selected or covered, it will no longer be selected in subsequent operations. This idea is simple and easy to implement; the second method is to introduce a dynamic coverage vector to represent the coverage status of each point, and dynamically update the dynamic coverage vector at each step. In detail, the dynamic embedding processing part updates the dynamic embedding of each logistics demand point according to the following steps S221 to S223.

[0100] S221. Initialize the dynamic coverage vector of each logistics demand point, which is used to reflect the coverage status of the corresponding point, to 1.

[0101] S222. For each logistics demand point, at each step in the decoding process, dynamically update the corresponding dynamic coverage vector according to the following formula:

[0102]

[0103] In the formula, t is a positive integer less than or equal to p, and p is t represents the take-off and landing point of the UAV selected in the t-th step, NCr(p t ) means that in the set Cr(p t ) in the set Cr(p t ) means at the drone take-off and landing point p tis the collection of logistics demand points covered by the center and the preset coverage radius, c i When the i-th logistics demand point belongs to the set Cr(p t ) represents the sorting index of the i-th logistics demand point in the node sequence, and the node sequence refers to the order from the logistics demand point to the drone take-off and landing point p t The order of distance from near to far is the same as the set Cr(p t ) are sorted in sequence, c i When the i-th logistics demand point does not belong to the set Cr(p t ) takes the value of NCr(p t ), I i,t represents the dynamic coverage vector corresponding to the i-th logistics demand point updated in the t-th step of the decoding process, I i,t-1 represents the dynamic coverage vector corresponding to the i-th logistics demand point updated in the t-1th step of the decoding process, and I i,0 =1.

[0104] In step S222, according to the above formula, if the i-th logistics demand point is covered multiple times, I i,t will decrease, indicating that the probability of the node being selected decreases. In addition, the designed dynamic coverage vector contains the coverage information of each node. The coverage of the node and its distance from the central node can serve as a useful guide for selecting the next node.

[0105] S223. For each logistics demand point, the final dynamic embedding is calculated according to the following formula:

[0106] H i,t =I i,t ×W I

[0107] In the formula, H i,t represents the dynamic embedding corresponding to the i-th logistics demand point and finally obtained in the t-th step of the decoding process, W I represents a learnable third parameter.

[0108] This embodiment designs a learnable decoder that can select drone take-off and landing points based on the attention mechanism and build a feasible solution. Figure 4 As shown, the decoder includes an RNN (Recurrent Neural Network) part for outputting query results and an attention structure part for calculating the probability of each logistics demand point being selected; the attention structure part calculates the probability of each logistics demand point being selected according to the following steps S231 to S233.

[0109] S231. For each logistics demand point, in the tth step, the query result q is generated by the RNN part and the self-attention mechanism. t Then, combined with the high-dimensional embedding representation obtained from the encoder, the corresponding vector is calculated according to the following formula:

[0110]

[0111] In the formula, represents the vector corresponding to the i-th logistics demand point and calculated in the t-th step, T represents the matrix transposition symbol, d h represents the total number of dimensions of the hidden layer, M represents the number of self-attentions in the multi-head attention layer, and W k represents the fourth learnable parameter.

[0112] S232. For each logistics demand point, if the corresponding point belongs to the set of drone take-off and landing points selected after the first t-1 steps, the corresponding vector is corrected to -1, otherwise the corresponding vector is not corrected.

[0113] S233. For each logistics demand point, the corresponding selection probability is calculated according to the following formula:

[0114]

[0115] Where η i,t represents the probability of being selected corresponding to the i-th logistics demand point at the t-th step, Represents the vector corresponding to the j-th logistics demand point and calculated in the t-th step.

[0116] Based on the aforementioned steps S231 to S233, the decoding process can be performed sequentially, so that the decoder can make a decision on a drone landing point selection result based on the static and dynamic embedding of the node obtained from the encoder and the current state of the partial solution at each step. In addition, when selecting the drone landing point, a greedy strategy can be specifically applied to select the node, that is, select the node with the highest probability of being selected.

[0117] In step S2, specifically, when the model is trained, the training goal is to try to maximize or minimize the objective function based on the selected drone take-off and landing points at each step, and use the deep reinforcement learning method to update the model parameters. The above-mentioned model parameter update formula is as follows:

[0118]

[0119] Wherein, L(p) represents the objective function, b(s) represents the baseline function, which is used to reduce the variance, and the remaining formula symbols can be derived based on the conventional REINFORCE algorithm. Generally, after a triple (s, a, r) ​​is given, the parameters of the model can be updated by using the reinforcement learning algorithm (Williams 1992) using the baseline b(s) and by gradient descent, that is, the algorithm for solving the problem using reinforcement learning is as follows: Reinforcement learning training based on Rollout Baseline, including the following process in detail:

[0120] Input: training set S, number of training cycles N epoch , number of training steps T, batch size B

[0121] 1. Initialize the policy network parameters u and the baseline policy parameters u * ,u * ←u

[0122] 2. For each training cycle epoch = 1: N epoch

[0123] 3. For each step step=1: T

[0124] ■ Sample input i ←SampleInput(S), where i∈{1,...,B}

[0125] ■Strategy obtains p through sampling i ←p u (s i )

[0126] ■Baseline strategy is obtained by rolling out baseline

[0127] ■Calculate the gradient

[0128] ■Update parameters

[0129] 4. If p u Outperformed Then u * ←u

[0130] End of training cycle

[0131] S3. Import the coordinates of n″ logistics demand points and logistics demand quantities in the target area into the drone take-off and landing point selection model, and output the coordinates of p drone take-off and landing points selected from the n″ logistics demand points as the logistics drone take-off and landing point selection result for the target area, where n″ represents a positive integer greater than p.

[0132] Therefore, based on the logistics drone take-off and landing point selection method described in the aforementioned steps S1 to S3, a new scheme for logistics drone take-off and landing point selection based on deep reinforcement learning method is provided, that is, after obtaining the solution to the logistics drone take-off and landing point selection problem under a given instance, the given instance is used as the model input item, and the solution is used as the model output item, and the model using the deep reinforcement learning method is imported for model training to obtain the drone take-off and landing point selection model, and then the coordinates of all logistics demand points and logistics demand in the target area are imported into the drone take-off and landing point selection model, and the coordinates of the selected drone take-off and landing points are output as the logistics drone take-off and landing point selection result in the target area. In this way, the complex logistics drone take-off and landing point site selection decision problem can be solved, so as to realize the distribution of logistics needs at low cost, which is convenient for practical application and promotion.

[0133] like Figure 5 As shown, the second aspect of this embodiment provides a virtual device for implementing the method for selecting the take-off and landing points of the logistics drone described in the first aspect, comprising a solution acquisition unit, a selection model training unit and a selection model application unit which are sequentially connected in communication;

[0134] The solution acquisition unit is used to acquire a solution to the problem of selecting a take-off and landing point for a logistics drone in a given instance, wherein the given instance includes coordinates of n′ given logistics demand points and logistics demand quantities, and the solution includes coordinates of p drone take-off and landing points selected from the n′ logistics demand points, wherein n′ represents a positive integer greater than or equal to 3, and p represents a positive integer less than n′;

[0135] The selected model training unit is used to use the given instance as a model input item and the solution as a model output item, import the model using the deep reinforcement learning method for model training, and obtain the drone take-off and landing point selection model;

[0136] The selection model application unit is used to import the coordinates and logistics demand quantities of n″ logistics demand points in the target area into the drone take-off and landing point selection model, and output the coordinates of p drone take-off and landing points selected from the n″ logistics demand points as the logistics drone take-off and landing point selection result for the target area, wherein n″ represents a positive integer greater than p.

[0137] The working process, working details and technical effects of the aforementioned device provided in the second aspect of this embodiment can be found in the method for selecting the take-off and landing points of the logistics drone described in the first aspect, and will not be repeated here.

[0138] like Figure 6As shown, the third aspect of this embodiment provides a computer device for executing the method for selecting the take-off and landing points of the logistics drone as described in the first aspect, including a memory, a processor and a transceiver that are sequentially connected in communication, wherein the memory is used to store a computer program, the transceiver is used to send and receive messages, and the processor is used to read the computer program to execute the method for selecting the take-off and landing points of the logistics drone as described in the first aspect. For example, the memory may include, but is not limited to, a random access memory (RAM), a read-only memory (ROM), a flash memory, a first-in-first-out memory (FIFO) and / or a first-in-last-out memory (FILO), etc.; the processor may include, but is not limited to, a microprocessor of the STM32F105 series. In addition, the computer device may also include, but is not limited to, a power module, a display screen and other necessary components.

[0139] The working process, working details and technical effects of the aforementioned computer equipment provided in the third aspect of this embodiment can be found in the method for selecting the take-off and landing points of the logistics drone described in the first aspect, and will not be repeated here.

[0140] The fourth aspect of this embodiment provides a computer-readable storage medium storing instructions including the method for selecting the take-off and landing points of the logistics drone as described in the first aspect, that is, the computer-readable storage medium stores instructions, and when the instructions are executed on a computer, the method for selecting the take-off and landing points of the logistics drone as described in the first aspect is executed. The computer-readable storage medium refers to a carrier for storing data, which may include but is not limited to computer-readable storage media such as floppy disks, optical disks, hard disks, flash memory, USB flash drives, and / or memory sticks, and the computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices.

[0141] The working process, working details and technical effects of the aforementioned computer-readable storage medium provided in the fourth aspect of this embodiment can be referred to the method for selecting take-off and landing points for logistics drones as described in the first aspect, and will not be repeated here.

[0142] In a fifth aspect, the present embodiment provides a computer program product, including a computer program or an instruction, which, when executed by a computer, implements the method for selecting a take-off and landing point for a logistics drone as described in the first aspect. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device.

[0143] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for selecting take-off and landing points for a logistics drone, characterized in that: include: Obtain a solution to the problem of selecting a take-off and landing point for a logistics drone in a given instance, wherein the given instance includes coordinates of n′ given logistics demand points and logistics demand quantities, and the solution includes coordinates of p drone take-off and landing points selected from the n′ logistics demand points, wherein n′ represents a positive integer greater than or equal to 3, and p represents a positive integer less than n′; The given instance is used as a model input item, and the solution is used as a model output item, and is imported into a model using a deep reinforcement learning method for model training, so as to obtain a UAV take-off and landing point selection model; The coordinates of n″ logistics demand points and logistics demand quantities in the target area are imported into the drone take-off and landing point selection model, and the coordinates of p drone take-off and landing points selected from the n″ logistics demand points are output as the logistics drone take-off and landing point selection result for the target area, wherein n″ represents a positive integer greater than p.

2. The method for selecting the take-off and landing points of a logistics drone according to claim 1 is characterized in that: The policy network in the model using the deep reinforcement learning method is used to select a different logistics demand point from the input n logistics demand points as a drone take-off and landing point at each time step until p drone take-off and landing points are selected, and the remaining logistics demand points are allocated to the nearest drone take-off and landing points, and then the final objective function is calculated to provide a feasible solution to the logistics drone take-off and landing point selection problem, wherein n represents a positive integer greater than or equal to 3; When training the model, the training goal is to try to maximize or minimize the objective function based on the drone take-off and landing points selected at each step, and use the deep reinforcement learning method to update the model parameters.

3. The method for selecting the take-off and landing points of a logistics drone according to claim 2 is characterized in that: The objective function is expressed by the following formula to be minimized: In the formula, Cost represents the total cost, j represents a positive integer less than or equal to n, represents the jth logistics demand point, f j (C) represents the construction cost coefficient of building the j-th logistics demand point into a drone take-off and landing point, α represents the total number of regions, D1, D2, …, D α They represent different regions and there is no intersection between any two regions, c1, c2, …, c α Represent the construction cost coefficients of different regions, y j Indicates whether to select the jth logistics demand point as the drone take-off and landing point and use y j =1 means select and use y j =0 means no selection, c means unit construction fixed cost, i means a positive integer less than or equal to n, represents the unit mileage delivery cost of the logistics drone corresponding to the j-th logistics demand point when the j-th logistics demand point is used as the drone take-off and landing point, x ij Indicates whether the i-th logistics demand point is allocated to the drone take-off and landing point when the j-th logistics demand point is used as the drone take-off and landing point, and x ij =1 means allocation and use x ij =0 means no allocation, d ij N represents the distance from the i-th logistics demand point to the drone take-off and landing point when the j-th logistics demand point is used as the drone take-off and landing point. i represents the logistics demand of the j-th logistics demand point; The constraints of the objective function are expressed as follows: In the formula, A ij Indicates whether there is a no-fly zone on the path from the i-th logistics demand point to the drone take-off and landing point when the j-th logistics demand point is used as the drone take-off and landing point, and uses A ij =0 means existence and A ij =1 means it does not exist. Indicates the maximum range of the logistics drone with full load, Indicates the maximum range of the logistics drone with no payload.

4. The method for selecting the take-off and landing points of a logistics drone according to claim 2 is characterized in that: The neural network core in the model using the deep reinforcement learning method includes an interconnected encoder and a decoder, wherein the encoder is responsible for converting the original input features into a high-dimensional embedding representation, and the decoder is used to make a decision based on the high-dimensional embedding representation: selecting a drone take-off and landing point from the input n logistics demand points.

5. The method for selecting the take-off and landing points of a logistics drone according to claim 4 is characterized in that: The high-dimensional embedding representation is divided into a static embedding part and a dynamic embedding part, wherein the static embedding part is used to represent the two-dimensional coordinate information of each logistics demand point, and the dynamic embedding part represents that the node state of each logistics demand point changes continuously with the change of coverage; The encoder includes a static embedding processing part and a dynamic embedding processing part; The static embedding processing part generates the static embedding of each logistics demand point in the following manner: After a layer of linear mapping, the original input coordinates of each logistics demand point are mapped to a high-dimensional static initial embedding: In the formula, i is a positive integer less than or equal to n, and x is i represents the original input coordinates of the i-th logistics demand point, represents the high-dimensional static initial embedding corresponding to the i-th logistics demand point, W (0) represents the first learnable parameter, b (0) represents the second learnable parameter; For each logistics demand point, a final and corresponding high-dimensional static embedding is generated through N multi-head attention layers, wherein the high-dimensional static embedding calculation results of each multi-head attention layer in the N multi-head attention layers are as follows: In the formula, k represents a positive integer less than or equal to N, represents the high-dimensional static embedding calculation result of the k-th multi-head attention layer corresponding to the i-th logistics demand point, BN() represents the function form of the normalization layer, FF() represents the function form of the feedforward layer, represents the high-dimensional static embedding calculation result of the k-th multi-head attention layer corresponding to the i-th logistics demand point, MHA() represents the function form of the multi-head attention layer, and the final high-dimensional static embedding corresponding to the i-th logistics demand point is The dynamic embedding processing part updates the dynamic embedding of each logistics demand point in the following manner: Initializing the dynamic coverage vector of each logistics demand point and used to reflect the coverage status of the corresponding point to 1; For each logistics demand point, at each step in the decoding process, the corresponding dynamic coverage vector is dynamically updated according to the following formula: In the formula, t is a positive integer less than or equal to p, and p is t represents the take-off and landing point of the UAV selected in the t-th step, NCr(p t ) means that in the set Cr(p t ) in the set Cr(p t ) means at the drone take-off and landing point p t is the collection of logistics demand points covered by the center and the preset coverage radius, c i When the i-th logistics demand point belongs to the set Cr(p t ) represents the sorting index of the i-th logistics demand point in the node sequence, and the node sequence refers to the order from the logistics demand point to the drone take-off and landing point p t The order of distance from near to far is the same as the set Cr(p t ) are sorted in sequence, c i When the i-th logistics demand point does not belong to the set Cr(p t ) takes the value of NCr(p t ), I i,t represents the dynamic coverage vector corresponding to the i-th logistics demand point updated in the t-th step of the decoding process, I i,t-1 represents the dynamic coverage vector corresponding to the i-th logistics demand point updated in the t-1th step of the decoding process, and I i,0 =1; For each logistics demand point, the final dynamic embedding is calculated according to the following formula: H i,t =I i,t ×W I In the formula, H i,t represents the dynamic embedding corresponding to the i-th logistics demand point and finally obtained in the t-th step of the decoding process, W I represents the third learnable parameter.

6. The method for selecting the take-off and landing points of a logistics drone according to claim 5 is characterized in that: The decoder includes an RNN part for outputting query results and an attention structure part for calculating the probability of each logistics demand point being selected; The attention structure part calculates the selection probability of each logistics demand point as follows: For each logistics demand point, in the tth step, the query result q is generated by the RNN part and the self-attention mechanism t Then, combined with the high-dimensional embedding representation obtained from the encoder, the corresponding vector is calculated according to the following formula: In the formula, represents the vector corresponding to the i-th logistics demand point and calculated in the t-th step, T represents the matrix transposition symbol, d h represents the total number of dimensions of the hidden layer, M represents the number of self-attentions in the multi-head attention layer, and W k represents the fourth learnable parameter; For each logistics demand point, if the corresponding point belongs to the set of drone take-off and landing points selected after the first t-1 steps, the corresponding vector is corrected to -1, otherwise the corresponding vector is not corrected; For each logistics demand point, the corresponding selection probability is calculated according to the following formula: Where η i,t represents the probability of being selected corresponding to the i-th logistics demand point at the t-th step, Represents the vector corresponding to the j-th logistics demand point and calculated in the t-th step.

7. A logistics drone take-off and landing point selection device, characterized in that: It includes a solution acquisition unit, a selected model training unit and a selected model application unit which are sequentially connected in communication; The solution acquisition unit is used to acquire a solution to the problem of selecting a take-off and landing point for a logistics drone in a given instance, wherein the given instance includes coordinates of n′ given logistics demand points and logistics demand quantities, and the solution includes coordinates of p drone take-off and landing points selected from the n′ logistics demand points, wherein n′ represents a positive integer greater than or equal to 3, and p represents a positive integer less than n′; The selected model training unit is used to use the given instance as a model input item and the solution as a model output item, import the model using the deep reinforcement learning method for model training, and obtain the drone take-off and landing point selection model; The selection model application unit is used to import the coordinates and logistics demand quantities of n″ logistics demand points in the target area into the drone take-off and landing point selection model, and output the coordinates of p drone take-off and landing points selected from the n″ logistics demand points as the logistics drone take-off and landing point selection result for the target area, wherein n″ represents a positive integer greater than p.

8. A computer device, characterized in that: It includes a memory, a processor and a transceiver which are communicatively connected in sequence, wherein the memory is used to store computer programs, the transceiver is used to send and receive messages, and the processor is used to read the computer program to execute the method for selecting the take-off and landing points of a logistics drone as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores instructions, and when the instructions are executed on a computer, the method for selecting a take-off and landing point for a logistics drone as described in any one of claims 1 to 6 is executed.

10. A computer program product comprising a computer program or instructions, characterized in that When the computer program or the instruction is executed by a computer, the method for selecting a take-off and landing point for a logistics drone as described in any one of claims 1 to 6 is implemented.

Images