A UAV-assisted delivery system and method based on reinforcement learning

Abstract

The present invention discloses a UAV-assisted delivery system and method based on reinforcement learning, which relates to the field of UAV delivery technology. The prediction method comprises the following steps: by collecting the flight environment parameters and equipment status parameters of the UAV, the environmental parameters and the equipment status parameters are subjected to data preprocessing, and effective features are extracted through feature engineering. Then, the endurance of the UAV is predicted using a trained deep neural network model, and the prediction result is compared with the set performance threshold to determine whether the UAV needs to adjust its flight strategy. The method can monitor the endurance status of the UAV in real time, predict the endurance in advance, reduce energy consumption, and optimize the flight path, thereby improving the overall flight efficiency of the UAV and the success rate of mission execution. In addition, the present invention can effectively cope with various changes in meteorological conditions and improve the adaptability and stability of the UAV in complex environments.

Description

Technical Field

[0001] The present invention relates to the technical field of drone delivery, and in particular to a drone-assisted delivery system and method based on reinforcement learning. Background Art

[0002] In recent years, drone-assisted delivery systems have attracted much attention due to their high efficiency and flexibility. However, existing technologies still have significant defects when facing complex dynamic environments. First, many drone delivery systems rely on fixed path planning, which makes it difficult for drones to respond quickly in emergencies such as sudden obstacles and bad weather, often causing delivery delays or failures. Therefore, there is an urgent need for a new type of drone delivery system that can perceive the environment in real time and flexibly adjust the operation strategy according to actual conditions to achieve more efficient delivery services.

[0003] In the prior art, the publication number CN118170013A discloses a UAV-assisted delivery system and method based on reinforcement learning, which realizes continuous tracking and reasonable obstacle avoidance of ground random moving targets through the PPO-PSO algorithm, the task allocation algorithm of the UAV task allocation model with LSTM-CNN neural network architecture, and the route planning algorithm for the UAV path planning model, as well as the construction of an autonomous guidance and tracking obstacle avoidance model. However, the UAV delivery environment is not always fixed. The flight conditions of UAVs in different environments are different. The impact of weather and the impact of obstacle avoidance on endurance may lead to delivery failure. Therefore, using only a set of reinforcement learning algorithms for UAV delivery training not only affects the delivery task, but also makes UAV delivery lose efficiency.

[0004] The above information disclosed in this Background section is only for enhancement of understanding of the background of the present disclosure and therefore it may contain information that does not constitute the prior art that is already known to one of ordinary skill in the art. Summary of the invention

[0005] The purpose of the present invention is to provide a UAV-assisted delivery system based on reinforcement learning to solve the problems raised in the above background technology.

[0006] To achieve the above object, the present invention provides the following technical solutions:

[0007] A drone-assisted delivery system based on reinforcement learning, comprising:

[0008] The data acquisition module is used to obtain the location information of the delivery drone and the three-dimensional point cloud data of the drone's surrounding environment in real time, collect the meteorological information during the drone delivery, and obtain the environmental map data around the drone through the three-dimensional point cloud data. The meteorological information includes wind speed, temperature, humidity and rainfall;

[0009] The meteorological analysis module is used to compare the collected wind speed, temperature, humidity and rainfall data with the maximum tolerable meteorological parameters during the performance test of the drone, build an analysis model based on the comparison results, and generate the real-time meteorological anomaly index of the drone; the endurance analysis module is used to obtain the location information of the delivery destination, and determine the height and width information of the obstacles based on the surrounding environment map data, obtain the minimum crossing distance data, and combine the minimum crossing distance data with the acquired drone location information, the location information of the delivery destination and the battery power information to generate a calculation formula for the endurance index of the drone's current remaining delivery distance;

[0010] The flight range analysis module is used to build a deep learning model and obtain weather information and flight range information from historical data. The weather information and flight range information in the historical data are used as the training set and label of the deep learning model for training. The trained deep learning model is used to predict the remaining flight range of the current drone.

[0011] The delivery management module is used to analyze the remaining estimated distance and the remaining range of the drone, and generate the current drone's delivery necessity index based on the drone's delivery necessity index. If the drone does not need to continue delivery, it will go to the nearest parking station and wait.

[0012] Furthermore, the logic for obtaining the environmental map data around the drone through the three-dimensional point cloud data is as follows:

[0013] The voxel grid method is used to divide the entire three-dimensional space into cubes of fixed size. These cubes are voxels, that is, volume pixels, which represent three-dimensional data. For each point in the point cloud, the voxel position is calculated and the point is assigned to the corresponding voxel. In each voxel, a representative point is retained, and the coordinates of the representative point are the average coordinates of all points in the voxel to generate new point cloud data; the RANSAC ground segmentation algorithm is used to randomly select three points from the point cloud to fit a plane model to generate a three-dimensional fitting plane model. All points in the entire point cloud are iterated, and the distance from each point to the fitting plane is calculated to find those distance attribute values ​​less than the threshold c. points that are consistent with the plane model, record the number of inliers and model parameters, repeat the steps, retain the model parameters with the largest number of inliers, use this model to identify and extract ground points, distinguish ground from non-ground points, build a geometric model of the ground, cluster non-ground points, each cluster corresponds to a potential obstacle, for each obstacle cluster, take the position of the drone as the coordinate origin, calculate the difference between its maximum and minimum coordinates on the vertical z-axis to obtain the obstacle height H, take the flight direction of the drone as the front, calculate the difference between the maximum and minimum coordinates on the front and rear x-axis and the left and right y-axis, and select the largest difference as the obstacle width W.

[0014] Furthermore, the method for determining the height and width information of obstacles based on the surrounding environment map data is as follows:

[0015]

[0016] in, is the obstacle avoidance coefficient, H is the obstacle height, W is the obstacle width, if , the drone chooses to increase its altitude to cross the obstacle; if , the drone chooses to bypass the obstacle horizontally; if , then continue to maintain the current heading and use this formula to obtain the minimum distance data.

[0017] Furthermore, the method for generating the real-time meteorological anomaly index of the drone is:

[0018]

[0019]

[0020] in, Indicates wind speed, Indicates temperature, Indicates humidity Indicates rainfall; is the wind speed weight, ,The direct impact of wind speed on UAV flight; is the temperature weight, ,Temperature changes affect battery performance and flight stability; is the humidity weight, , humidity can cause malfunctions in electrical systems; is the rainfall weight, ,Precipitation has an impact on flight safety, but will not harm the drone; ;

[0021] Wind speed ratio: When the wind speed exceeds the maximum tolerable wind speed, the value is positive, indicating that the weather conditions are unfavorable; when the value is negative, it indicates that the wind speed is within the tolerable range; Temperature difference: When the temperature is close to the maximum tolerable temperature, the value is close to 0, indicating unfavorable meteorological conditions; when the temperature is close to the minimum tolerable temperature, the value is close to 1, indicating favorable meteorological conditions; Humidity difference: It means that when the humidity is close to the maximum tolerable humidity, the value is close to 0, indicating unfavorable meteorological conditions; when the humidity is close to the minimum tolerable humidity, the value is close to 1, indicating favorable meteorological conditions;

[0022] if , it means that the weather conditions have a negative impact on the flight of the drone; if , it means that the weather conditions are favorable.

[0023] Furthermore, the calculation formula for generating the endurance index of the current remaining delivery distance of the drone is:

[0024]

[0025]

[0026] in, To check the remaining battery capacity after avoiding obstacles, is the energy consumption per unit time of the drone, To estimate the required flight time, Indicates the battery life index. For the remaining distance, is the set flight speed of the drone.

[0027] Furthermore, the meteorological information in the historical data is used as the input feature, and the battery life information is used as the target label to build a deep learning model. The historical data is divided into a training set at a ratio of 80%, and a test set at a ratio of 20%. 10% of the data is further extracted from the training set as a validation set, and the long short-term memory network LSTM method is used as a deep learning model for processing time series data. The input layer is constructed according to the number of features of the meteorological information, the corresponding time series data is input into the model, the LSTM layer is added, and the ReLU activation function is selected to construct the hidden layer. In the output layer, the model will predict the battery life information. The mean square error MSE is selected as the loss function to evaluate the difference between the model prediction and the true value. The Adam optimizer is used to update the parameters, and the model performance is monitored in real time through the validation set.

[0028] Furthermore, the distribution necessity index is generated The calculation formula is:

[0029]

[0030] in, To deliver the necessary index, is the battery life index, is the meteorological anomaly index, when hour, The value of decreases, reflecting the negative impact of meteorological conditions on delivery capacity.

[0031] Furthermore, the method for judging whether the drone needs to continue delivery according to the delivery necessity index of the drone is: When the UAV has sufficient endurance and good weather conditions, it can continue to deliver. When the drone is not able to fly, it means that the drone’s endurance is insufficient or the weather conditions are not suitable, and it needs to find the nearest landing station to wait.

[0032] Furthermore, based on the drone's delivery necessity index, it is determined whether it needs to continue delivering. If the drone determines that it cannot complete the delivery task temporarily, it will go to the nearest parking station and wait. The specific process is to search for nearby parking stations based on location information; use map navigation to select routes to various nearby parking stations, and compare the length of each route to select the shortest route; when navigating on the map, it will follow the ground road planning and no longer avoid obstacles to improve endurance and ensure that it can reach the parking station smoothly; the parking station can charge the drone; if the navigation-planned parking station is farther away from the destination than the destination of the delivery task, the drone will continue to perform the delivery task.

[0033] The present invention also provides a UAV-assisted delivery method based on reinforcement learning, wherein the delivery method is obtained by executing the above-mentioned UAV-assisted delivery system, and the specific steps include:

[0034] Step 1: Acquire the location information of the delivery drone and the three-dimensional point cloud data of the drone's surrounding environment in real time, collect the weather information during the drone delivery, and obtain the environmental map data around the drone through the three-dimensional point cloud data. The weather information includes wind speed, temperature, humidity and rainfall.

[0035] Step 2: Based on the collected wind speed, temperature, humidity and rainfall data, compare them with the maximum weather parameters that the drone can withstand during performance testing, build an analysis model based on the comparison results, and generate a real-time meteorological anomaly index for the drone;

[0036] Step 3: Obtain the location information of the delivery destination, and determine the height and width information of obstacles based on the surrounding environment map data to obtain the minimum crossing distance data. Combine the minimum crossing distance data with the obtained drone location information and the delivery destination location information to generate a calculation formula for the drone's current remaining delivery distance endurance index.

[0037] Step 4: Build a deep learning model and obtain weather information and endurance information from historical data. Use the weather information and endurance information in the historical data as the training set and label of the deep learning model, respectively, and use the trained deep learning model to predict the remaining range of the current drone.

[0038] Step 5: Analyze the remaining estimated distance and the remaining range of the drone, and generate the current drone's delivery necessity index in combination with the weather anomaly index. Determine whether the drone needs to continue delivery based on the drone's delivery necessity index. If the drone does not need to continue delivery, it will go to the nearest parking station and wait.

[0039] Compared with the prior art, the present invention has the following beneficial effects:

[0040] The present invention significantly improves the ability of drones to cope with complex environments by introducing dynamic path planning based on deep reinforcement learning. Specifically, the system can collect and analyze environmental meteorological data in real time through three-dimensional point cloud data, and compare it with the maximum tolerable meteorological parameters during the performance test of the drone. According to the comparison results, an analysis model is constructed to dynamically adjust the flight path, thereby effectively reducing the delivery time and improving the delivery success rate. A deep learning model is constructed to obtain meteorological information and endurance information in historical data, and the meteorological information and endurance information in the historical data are used as training sets and labels for the deep learning model, respectively, for training. The trained deep learning model is used to predict the remaining range of the current drone. Compared with the prior art, this solution not only improves the autonomy of the drone through an intelligent decision-making mechanism, but also enhances the flexibility of the system in responding to emergencies, and ultimately achieves efficient and reliable delivery services. The implementation of this innovative technology can bring significant technological progress to the drone delivery industry and meet the growing market demand. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 It is a schematic diagram of the overall system design of the present invention.

[0042] Figure 2 It is a schematic diagram of the overall method flow of the present invention. DETAILED DESCRIPTION

[0043] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with specific embodiments.

[0044] It should be noted that, unless otherwise defined, the technical terms or scientific terms used in the present invention should be understood by people with ordinary skills in the field to which the present invention belongs. The words "first", "second" and similar words used in the present invention do not indicate any order, quantity or importance, but are only used to distinguish different components. "Include" or "comprise" and similar words mean that the elements or objects appearing before the word include the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0045] Example:

[0046] See also Figure 1, the present invention provides a technical solution:

[0047] A UAV-assisted delivery system based on reinforcement learning, comprising a data acquisition module, a meteorological analysis module, a flight endurance analysis module, a voyage analysis module and a delivery management module, wherein the UAV stores the location coordinates of all parking platforms;

[0048] The data acquisition module is used to obtain the location information of the delivery drone and the three-dimensional point cloud data of the drone's surrounding environment in real time, and to collect the meteorological information during the drone delivery, and to obtain the environmental map data around the drone through the three-dimensional point cloud data, wherein the meteorological information includes wind speed, temperature, humidity and rainfall;

[0049] The 3D point cloud data is a collection of data about the geometric shape of the environment in the form of points within a preset range around the drone. Each point is represented by a spatial coordinate system. , set the drone position as the origin of the space coordinates , as well as color and intensity attribute information. Intensity information distinguishes objects of different materials, identifies and classifies different types of objects, and color information is used for object classification and segmentation. The Gaussian filter algorithm is used to smooth the data by weighting the surrounding points, and removes points in the point cloud whose spatial distance to other points around it is greater than the distance f. According to expert experience, the value of f can be set to 0.1-0.5 meters. This setting is based on the consideration of noise in the point cloud data. A smaller f value can remove outliers and noise points, thereby improving the quality of the point cloud; the Gaussian filter weight threshold is lower than the point g, and setting the g value to 0.5 is based on experience. The purpose is to ensure that only points with higher weights are retained, helping the algorithm to better focus on important features; points above the threshold usually represent more reliable measurement data, and retaining these points can reduce errors caused by noise in subsequent processing;

[0050] The entire three-dimensional space is divided into cubes of fixed size through the voxel grid method. First, the voxel size is determined to generate new point cloud data. These cubes are voxels, that is, volume pixels, representing three-dimensional data. Then the voxel network is calculated. According to the minimum and maximum coordinates in the point cloud data, the boundary of the three-dimensional space is calculated. For each point in the point cloud, its voxel position is calculated, the voxel grid is initialized, and the point is assigned to the corresponding voxel. The representative point is calculated. In each voxel, a representative point is retained. The coordinates of the representative point are the average coordinates of all points in the voxel to generate new point cloud data; the RANSAC ground segmentation algorithm is used to randomly select three points from the point cloud to fit a plane model to generate a three-dimensional fitted plane model. All points in the entire point cloud are iterated, and the distance from each point to the fitting plane is calculated to find those points whose distance attribute values ​​are less than the threshold c. According to According to expert experience, the value of c can be set to 0.05-0.1. This setting is based on the understanding of ground features. This range can effectively identify points that are consistent with the ground model and exclude non-ground points that deviate significantly from the plane. These points are consistent with the plane model. The number of points and model parameters are recorded. The steps are repeated to retain the model parameters with the largest number of internal points. This model is used to identify and extract ground points, distinguish between ground and non-ground points, build a geometric model of the ground, cluster non-ground points, and each cluster corresponds to a potential obstacle. For each obstacle cluster, the position of the drone is used as the coordinate origin, and the difference between its maximum and minimum coordinates on the vertical z-axis is calculated to obtain the obstacle height H. With the drone's flight direction as the front, the difference between the maximum and minimum coordinates on the front-to-back x-axis and the left-to-right y-axis is calculated, and the largest difference is selected as the obstacle width W.

[0051] The method for determining the height and width of obstacles based on the surrounding environment map data is as follows:

[0052]

[0053] in, is the obstacle avoidance coefficient, H is the obstacle height, W is the obstacle width, if , the drone chooses to increase its altitude to cross the obstacle; if , the drone chooses to bypass the obstacle horizontally; if , then continue to maintain the current heading and use this formula to obtain the minimum span distance data;

[0054] It should be noted that the three-dimensional point cloud data of the drone during operation is obtained through the laser radar installed on the body;

[0055] Therefore, obtaining the drone's surrounding environment map and obstacle information can ensure the drone's delivery situation.

[0056] The meteorological analysis module is used to compare the collected wind speed, temperature, humidity and rainfall data with the maximum meteorological parameters that can be tolerated during the performance test of the drone, build an analysis model based on the comparison results, and generate a real-time meteorological anomaly index for the drone;

[0057] The method for generating the real-time meteorological anomaly index of the drone is:

[0058]

[0059]

[0060] in, Indicates wind speed, Indicates temperature, Indicates humidity Indicates rainfall; is the wind speed weight, ,Wind speed is the most direct factor affecting the flight stability of drones. Strong winds cause drones to lose control, affecting their heading and altitude. Under strong wind conditions, it becomes more difficult to control drones, which may lead to flight accidents. The weight of wind speed should be the highest; is the temperature weight, Temperature changes have a significant impact on battery performance. Extreme temperatures may cause battery efficiency to decrease, which in turn affects flight time and endurance. Temperature also affects air density, which in turn affects lift and flight performance. Although the impact is smaller than wind speed, it cannot be ignored. is the humidity weight, ,Humidity can cause failures in electrical systems. It has less direct impact on flight stability. High humidity may cause failures, but these failures are usually not immediately fatal. Humidity may indirectly affect the long-term operation of the drone by causing condensation or corrosion, so its weight is lower than wind speed and temperature; is the rainfall weight, ,Precipitation has an impact on flight safety, but will not damage the drone.,Precipitation does not directly affect flight control like wind speed, temperature, and humidity, so it has the lowest weight; .

[0061] Wind speed ratio: When the wind speed exceeds the maximum tolerable wind speed, the value is positive, indicating that the weather conditions are unfavorable; when the value is negative, it indicates that the wind speed is within the tolerable range; Temperature difference: When the temperature is close to the maximum tolerable temperature, the value is close to 0, indicating unfavorable meteorological conditions; when the temperature is close to the minimum tolerable temperature, the value is close to 1, indicating favorable meteorological conditions; Humidity difference: It means that when the humidity is close to the maximum tolerable humidity, the value is close to 0, indicating unfavorable meteorological conditions; when the humidity is close to the minimum tolerable humidity, the value is close to 1, indicating favorable meteorological conditions;

[0062] if , it means that the weather conditions have a negative impact on the flight of the drone; if , it means that the weather conditions are favorable and suitable for continued delivery.

[0063] It should be noted that meteorological detection is obtained through equipment installed on the drone body; these equipment are the following:

[0064] Temperature sensor: used to monitor the temperature in real time to ensure that the drone flies under suitable temperature conditions; Humidity sensor: used to measure air humidity and evaluate the humidity level of the flight environment; Rainfall sensor: such as a rain gauge, which can be used to monitor rainfall and determine the safety of flight conditions; Wind speed sensor: such as anemometers and wind direction sensors, used to detect changes in wind speed and wind direction; Multi-sensor integrated system: In order to obtain more comprehensive meteorological data, an integrated weather station can be used to combine all the above sensors to provide real-time meteorological information;

[0065] When the weather changes extremely, drone delivery may have the following impacts:

[0066] Temperature influence: If the temperature is too low, the battery performance may decline and the endurance may be reduced; if the temperature is too high, the battery may overheat and affect the safety. Extreme temperature may affect the flight performance and stability of the drone.

[0067] Humidity impact: High humidity may cause electronic equipment to malfunction, increasing the risk of failure; humidity changes can affect air density, which in turn affects lift and flight stability; in high humidity environments, higher lift may be required, which can affect endurance and flight altitude;

[0068] Impact of rainfall: Rainfall will increase the weight of the drone, affect flight stability, and may cause landing and flight accidents; rain may affect sensor readings and data accuracy; in the case of rainfall, the drone may need to suspend delivery, affecting delivery time and user experience;

[0069] Impact of wind speed: In strong wind conditions, the drone may face the risk of losing control, affecting flight stability; strong winds may change the flight path, causing deviation or increased energy consumption; excessive wind speeds may cause delivery delays or require adjustments to the flight route, increasing flight time and energy consumption;

[0070] Therefore, installing relevant meteorological sensors can help drones perceive the surrounding weather conditions in real time, so as to make corresponding flight decisions to ensure the safety and efficiency of delivery. Different meteorological factors have a direct impact on the flight performance and mission completion rate of drones.

[0071] The endurance analysis module is used to obtain the location information of the delivery destination, and determine the height and width information of obstacles based on the surrounding environment map data to obtain the minimum crossing distance data, and combine the minimum crossing distance data with the obtained drone location information, the delivery destination location information and the battery power information to generate the calculation formula for the drone's current remaining delivery distance endurance index:

[0072]

[0073]

[0074] in, To check the remaining battery capacity after avoiding obstacles, is the energy consumption per unit time of the drone, To estimate the required flight time, Indicates the battery life index. For the remaining distance, is the set flight speed of the drone;

[0075] It can be seen from the formula that the energy consumption of the drone per unit time is constant. The greater the remaining battery capacity after avoiding obstacles and the shorter the expected flight time, the greater the endurance index, and the better the drone's endurance status; the smaller the remaining battery capacity after avoiding obstacles and the longer the expected flight time, the smaller the endurance index, and the worse the drone's endurance status.

[0076] The flight range analysis module is used to build a deep learning model and obtain weather information and endurance information in historical data. The weather information and endurance information in the historical data are used as training sets and labels of the deep learning model for training, and the trained deep learning model is used to predict the remaining flight range of the current UAV.

[0077] The meteorological information in the historical data is used as the input feature, and the battery life information is used as the target label to build a deep learning model. The historical data is divided into a training set at a ratio of 80%, and a test set at a ratio of 20%. 10% of the data is further extracted from the training set as the validation set. The long short-term memory network LSTM method is used as a deep learning model for processing time series data. The input layer is constructed according to the number of features of the meteorological information, the corresponding time series data is input into the model, the LSTM layer is added, and the ReLU activation function is used to construct the hidden layer. In the output layer, the model will predict the battery life information. The mean square error MSE is selected as the loss function to evaluate the difference between the model prediction and the true value. The Adam optimizer is used to update the parameters, and the model performance is monitored in real time through the validation set.

[0078] The delivery management module is used to analyze the remaining estimated distance and the remaining flight range of the drone, and generate the delivery necessity index of the current drone in combination with the weather anomaly index, and judge whether the drone needs to continue delivery according to the delivery necessity index of the drone. If it does not need to continue delivery, it will go to the nearest parking station to wait;

[0079] Generate delivery necessity index The calculation formula is:

[0080]

[0081] in, To deliver the necessary index, is the battery life index, is the meteorological anomaly index, hour, The value of decreases, reflecting the negative impact of meteorological conditions on delivery capacity.

[0082] From the formula, we can see that In order to ensure that the denominator is not zero, when the endurance index is large, that is, the remaining battery capacity after avoiding obstacles is large, the required delivery time is short, and the weather index is small, that is, the weather conditions have little impact on the drone, the weather conditions are good and the battery is sufficient, the larger the delivery necessity index is, the greater the possibility of the drone completing the delivery; when the endurance index is small, that is, the remaining battery capacity after avoiding obstacles is small, the required delivery time is long, and the weather index is large, that is, the weather conditions have a great impact on the drone, the weather conditions are bad and the battery is insufficient, the delivery necessity index is small, and the possibility of the drone completing the delivery is small;

[0083] when When the UAV has sufficient endurance and good weather conditions, it can continue to deliver. , it means that the drone’s endurance is insufficient or the weather conditions are not suitable, and it needs to find the nearest landing station to wait. If the endurance cannot reach the nearest landing station, it will automatically land, send its own location, and wait for manual rescue.

[0084] A landing pad is a facility or area designed specifically for drones, primarily used for the docking, charging, maintenance and mission management of drones. It is usually equipped with a charging station that can provide charging services for the drone's batteries. It has a flat docking area for drones to land and park safely. The platform is designed to withstand all kinds of weather conditions to ensure the safety of drones when not in use. The landing pad is equipped with technicians and related facilities to provide the necessary technical support and services to ensure that any problems encountered during the drone's delivery process can be resolved in a timely manner.

[0085] Determine whether the drone needs to continue delivering based on its delivery necessity index. If the drone determines that it cannot complete the delivery task temporarily, it will go to the nearest parking station and wait. The specific process is to search for nearby parking stations based on location information; use map navigation to select routes to various nearby parking stations, and compare the length of each route to select the shortest route; when navigating on the map, it will follow the ground road planning and no longer avoid obstacles to improve endurance and ensure that it can reach the parking station smoothly; the parking station can charge the drone; if the navigation-planned parking station is farther away from the destination than the destination of the delivery task, the drone will continue to perform the delivery task.

[0086] See also Figure 2 The present invention also provides a UAV-assisted delivery method based on reinforcement learning. The delivery method is obtained by executing the above-mentioned UAV-assisted delivery system, and the specific steps include:

[0087] Step 1: Acquire the location information of the delivery drone and the three-dimensional point cloud data of the drone's surrounding environment in real time, collect the weather information during the drone delivery, and obtain the environmental map data around the drone through the three-dimensional point cloud data. The weather information includes wind speed, temperature, humidity and rainfall;

[0088] Step 2: Based on the collected wind speed, temperature, humidity and rainfall data, compare them with the maximum weather parameters that the drone can withstand during performance testing, build an analysis model based on the comparison results, and generate a real-time meteorological anomaly index for the drone;

[0089] Step 3: Obtain the location information of the delivery destination, and determine the height and width information of obstacles based on the surrounding environment map data to obtain the minimum crossing distance data. Combine the minimum crossing distance data with the obtained drone location information and the delivery destination location information to generate a calculation formula for the drone's current remaining delivery distance endurance index;

[0090] Step 4: Build a deep learning model and obtain weather information and endurance information from historical data. Use the weather information and endurance information in the historical data as the training set and label of the deep learning model, respectively, and use the trained deep learning model to predict the remaining range of the current drone.

[0091] Step 5: Analyze the remaining estimated distance and the remaining range of the drone, and generate the current drone's delivery necessity index in combination with the weather anomaly index. Determine whether the drone needs to continue delivery based on the drone's delivery necessity index. If the drone does not need to continue delivery, it will go to the nearest parking station and wait.

[0092] The above formulas are all dimensionless and numerical calculations. The formula is a formula for the most recent real situation obtained by collecting a large amount of data and performing software simulation. The preset parameters in the formula are set by technicians in this field according to actual conditions.

[0093] The above embodiments may be implemented in whole or in part by software, hardware, firmware or any other combination thereof. When implemented by software, the above embodiments may be implemented in whole or in part in the form of a computer program product. Those skilled in the art may appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein may be implemented by electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed by hardware or software methods depends on the specific application and design constraints of the technical solution.

[0094] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, and may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0095] The above description is only a specific implementation manner of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application.

Claims

1. A UAV-assisted delivery system based on reinforcement learning, characterized in that: The drone stores the location coordinates of all parking platforms, including: The data acquisition module is used to obtain the location information of the delivery drone and the three-dimensional point cloud data of the drone's surrounding environment in real time, collect the meteorological information during the drone delivery, and obtain the environmental map data around the drone through the three-dimensional point cloud data. The meteorological information includes wind speed, temperature, humidity and rainfall; The meteorological analysis module is used to compare the collected wind speed, temperature, humidity and rainfall data with the maximum meteorological parameters that the drone can withstand during performance testing, build an analysis model based on the comparison results, and generate a real-time meteorological anomaly index for the drone; The method for generating the real-time meteorological anomaly index of the drone is: ； in, Indicates wind speed, Indicates temperature, Indicates humidity Indicates rainfall; is the wind speed weight, is the temperature weight, is the humidity weight, is the rainfall weight; ; The endurance analysis module is used to obtain the location information of the delivery destination, and determine the height and width information of obstacles based on the surrounding environment map data to obtain the minimum crossing distance data. The minimum crossing distance data is combined with the obtained drone location information, the location information of the delivery destination, and the battery power information to generate the endurance index of the drone's current remaining delivery distance; The flight range analysis module is used to build a deep learning model and obtain weather information and flight range information from historical data. The weather information and flight range information in the historical data are used as the training set and label of the deep learning model for training. The trained deep learning model is used to predict the remaining flight range of the current drone. The delivery management module is used to analyze the remaining estimated distance and the remaining range of the drone, and generate the current drone's delivery necessity index based on the drone's delivery necessity index. If the drone does not need to continue delivery, it will go to the nearest parking station and wait.

2. The UAV-assisted delivery system based on reinforcement learning according to claim 1, characterized in that: The logic for obtaining the environmental map data around the drone through the three-dimensional point cloud data is as follows: The entire three-dimensional space is divided into cubes of fixed size by the voxel grid method, and each cube is regarded as a voxel. For each point in the point cloud, the point is assigned to the corresponding voxel according to its voxel position. In each voxel, a representative point is retained, and the coordinates of the representative point are the average coordinates of all points in the voxel to generate new point cloud data; the RANSAC ground segmentation algorithm is used to randomly select three points from the point cloud to fit a plane model to generate a three-dimensional fitting plane model, and iterate all points in the entire point cloud. By calculating the distance from each point to the fitting plane, the points whose distance attribute values ​​are less than the threshold c are found. These The points are consistent with the plane model, the number of inliers and model parameters are recorded, the steps are repeated, the model parameters with the largest number of inliers are retained, and this model is used to identify and extract ground points, distinguish ground from non-ground points, build a geometric model of the ground, cluster the non-ground points, and each cluster corresponds to a potential obstacle. For each obstacle cluster, the position of the drone is used as the coordinate origin, and the difference between its maximum and minimum coordinates on the vertical z-axis is calculated to obtain the obstacle height H. With the flight direction of the drone as the front, the difference between the maximum and minimum coordinates on the front and rear x-axis and the left and right y-axis is calculated, and the largest difference is selected as the obstacle width W.

3. The UAV-assisted delivery system based on reinforcement learning according to claim 2 is characterized in that: The method for generating obstacle avoidance coefficients based on obstacle data from surrounding environment maps is: in, is the obstacle avoidance coefficient, H is the obstacle height, and W is the obstacle width.

4. The UAV-assisted delivery system based on reinforcement learning according to claim 1, characterized in that: The calculation formula for generating the endurance index of the current remaining delivery distance of the drone is: in, To check the remaining battery capacity after avoiding obstacles, is the energy consumption per unit time of the drone, To estimate the required flight time, Indicates the battery life index. For the remaining distance, is the set flight speed of the drone.

5. The UAV-assisted delivery system based on reinforcement learning according to claim 1, characterized in that: Taking the meteorological information in the historical data as the input feature and the battery life information as the target label, a deep learning model is constructed. The historical data is divided into 80% of the training set and 20% of the test set, and 10% of the data is further extracted from the training set as the validation set. The long short-term memory network LSTM method is used as the deep learning model for processing time series data. The input layer is constructed according to the number of features of the meteorological information, the corresponding time series data is input into the model, the LSTM layer is added, and the ReLU activation function is selected to construct the hidden layer. In the output layer, the model will predict the battery life information. The mean square error MSE is selected as the loss function to evaluate the difference between the model prediction and the true value. The Adam optimizer is used to update the parameters, and the model performance is monitored in real time through the validation set.

6. The UAV-assisted delivery system based on reinforcement learning according to claim 1, characterized in that: The calculation formula for generating the distribution necessity index is: in, To deliver the necessary index, is the battery life index, It is the weather anomaly index.

7. The UAV-assisted delivery system based on reinforcement learning according to claim 1, characterized in that: The method of judging whether the drone needs to continue delivery based on the drone's delivery necessity index is: in, When the UAV has sufficient endurance and good weather conditions, it can continue to deliver. When the drone is on standby, it means that the drone’s endurance is insufficient or the weather conditions are not suitable. It is necessary to use the drone’s real-time location coordinates and the location coordinates of all landing pads stored inside the drone to find the nearest landing pad and wait.

8. The UAV-assisted delivery system based on reinforcement learning according to claim 7 is characterized in that: Based on the delivery necessity index of the drone, it is determined whether it needs to continue delivery. If the drone determines that it cannot complete the delivery task temporarily, it will go to the nearest parking station and wait for orders. The specific process is to search for nearby parking stations based on location information; use map navigation to select routes to various nearby parking stations, and compare the length of each route to select the shortest route; when navigating on the map, it will follow the ground road planning and no longer avoid obstacles to ensure that it can reach the parking station smoothly; the parking station charges the drone; if the distance between the navigation-planned parking station and the destination is greater than the destination of the delivery task, the drone will continue to perform the delivery task.

9. A drone-assisted delivery method based on reinforcement learning, characterized in that: The delivery method is obtained by executing the drone-assisted delivery system according to any one of claims 1 to 8, and the specific steps include: Step 1: Acquire the location information of the delivery drone and the three-dimensional point cloud data of the drone's surrounding environment in real time, collect the weather information during the drone delivery, and obtain the environmental map data around the drone through the three-dimensional point cloud data. The weather information includes wind speed, temperature, humidity and rainfall. Step 2: Based on the collected wind speed, temperature, humidity and rainfall data, compare them with the maximum weather parameters that the drone can withstand during performance testing, build an analysis model based on the comparison results, and generate a real-time meteorological anomaly index for the drone; Step 3: Obtain the location information of the delivery destination, and determine the height and width information of obstacles based on the surrounding environment map data to obtain the minimum crossing distance data. Combine the minimum crossing distance data with the obtained drone location information and the delivery destination location information to generate a calculation formula for the drone's current remaining delivery distance endurance index. Step 4: Build a deep learning model and obtain weather information and endurance information from historical data. Use the weather information and endurance information in the historical data as the training set and label of the deep learning model, respectively, and use the trained deep learning model to predict the remaining range of the current drone. Step 5: Analyze the remaining estimated distance and the remaining range of the drone, and generate the current drone's delivery necessity index in combination with the weather anomaly index. Determine whether the drone needs to continue delivery based on the drone's delivery necessity index. If the drone does not need to continue delivery, it will go to the nearest parking station and wait.

Images