Reinforcement pointer network-based rechargeable unmanned aerial vehicle data acquisition method
By combining reinforced pointer networks with short-term prediction, the problem of balancing solution speed and optimal solution in UAV data acquisition is solved, improving acquisition efficiency and optimizing behavior strategies, and shortening the solution time.
Patent Information
- Application Number
- CN202411373903.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-29
- Publication Date
- 2025-10-21
- Estimated Expiration
- 2044-09-29
AI Technical Summary
Existing rechargeable UAV data acquisition methods struggle to balance solution speed and optimal solution in large-scale problems, while intelligent optimization algorithms suffer from state transition complexity and slow optimization speed.
We adopt a method based on reinforcement pointer networks combined with short-term prediction. By combining pointer networks with reinforcement learning, we can improve the performance of solutions and shorten the computation time of solutions through short-term prediction.
It improves the efficiency of drone data collection, increases the probability of making better behavioral strategies under current battery conditions, reduces the time required for data collection, and improves the effectiveness of data collection.
Smart Images
Figure CN119417018B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the fields of drone data collection, deep reinforcement learning, and more specifically, to a rechargeable drone data collection method based on a reinforcement pointer network. Background Art
[0002] In recent years, with the rapid development of the Internet of Things (IoT), wireless sensor networks (WSNs) utilize multiple sensors to acquire and analyze surrounding information, improving data acquisition efficiency and real-time performance. When natural disasters disrupt connectivity with ground-based equipment or complex terrain presents challenges with ground communications, drones (UAVs) offer a flexible and efficient method for collecting sensor data, reducing the energy consumption of transmitting data and the deployment and maintenance costs of wireless sensors. Within limited energy constraints, UAVs plan an optimal flight path based on mission requirements and environmental information, prioritizing the order in which they visit sensor nodes and charging stations.
[0003] Unmanned aerial vehicle (UAV) path planning, a complex combinatorial optimization problem, has attracted the attention of researchers both domestically and internationally in recent years. This problem can be modeled as a variation of the traveling salesman problem. Early research favored precise algorithms to solve such problems, with common approaches including dynamic programming, branch-and-bound, and linear programming. However, as the problem scales, dimensionality explosion can occur, and the optimal balance between solution speed and optimal solution is difficult to achieve, making this problem of significant research interest. Numerous intelligent optimization algorithms have been proposed, such as ant colony algorithms, particle swarm algorithms, genetic algorithms, tabu search algorithms, and simulated annealing algorithms. While these algorithms demonstrate certain advantages in solving these problems, they also face challenges such as the complexity of state transitions and relatively slow optimization speeds. Summary of the Invention
[0004] The purpose of this invention is to address the shortcomings of existing rechargeable UAV data collection methods and propose a rechargeable UAV data collection method based on enhanced pointer network combined with short-term prediction, which effectively improves the collection efficiency.
[0005] The present invention mainly solves the rechargeable drone data collection problem through the following method: the rechargeable drone data problem is modeled as a variant of the traveling salesman problem, and through the combination of pointer networks and reinforcement learning, short-term prediction is used to improve the solution effect and shorten the solution calculation time.
[0006] The specific implementation steps of the present invention are as follows:
[0007] S1: Build a rechargeable drone scenario and acquire drone-related data. This includes obtaining the distribution of sensors and charging stations, as well as drone metrics such as power consumption per meter during flight, flight distance per second, power consumption per unit of data collected, data volume collected per second, and the drone's charging rate. The system also acquires dynamic information about the drone and its sensors at every moment, including the drone's location, current battery level, and remaining data volume at sensor points.
[0008] S2: Build a pointer network short-term prediction model, perform state encoding on the data obtained in S1, input the state encoding result into the encoder to obtain the intermediate variable; then use the decoder to obtain the hidden layer information through the intermediate variable, and use the attention mechanism combined with short-term prediction to obtain the drone's action at the next moment, and repeat this cycle until all sensor data are collected. Specifically including:
[0009] S2.1: Take the current battery level and remaining data of the drone as a dynamic input set in Represents the battery level e of the drone at the current time t t and the remaining data As a dynamic input collection.
[0010] The charging pile and sensor position coordinate set is used as the static input set N = (n0, n1, ..., n n ), where there is a charging station n0 and n sensors collecting data; static input and dynamic input components are obtained The output of the state encoding As input to the encoder module.
[0011] S2.2: The encoder module uses the first embedding layer composed of a convolutional network to Collection Mapping to a one-dimensional vector As an intermediate variable input to the decoder, After the first embedding layer, we get n i , After the embedding layer, we get
[0012] S2.3: The decoder is composed of a GRU network, which converts the output obtained in S2.2 As the initial input of the decoder module, the output is the hidden layer information h t And the current UAV position information x at each time step t After the second embedding layer composed of convolutional networks, we get At each time step of decoding, the hidden layer information h tand the drone's position at that moment Use the attention mechanism combined with short-term prediction model information to derive the probability distribution of each behavior strategy of the drone at the next moment in This behavior indicates the sensor number that the drone will go to for data collection or return to the charging station for charging at the next moment; finally, a greedy strategy is used to select the action with the maximum probability value as the drone's next behavior.
[0013] S3: Use the policy gradient method to train the pointer network short-term prediction model, adopt the Actor-Critic framework, generate the actor network and the critic network, use the reward function to calculate the update gradient to continuously update the network, and finally make the reward converge.
[0014] The present invention incorporates the results of short-term prediction fusion into its algorithm. This addresses the data collection problem for rechargeable drones, allowing them to take their own battery life into account when making decisions. This effectively optimizes subsequent decision-making and increases the probability that the drone will adopt the optimal behavior strategy given its current battery life. Furthermore, the present invention reduces learning time and improves learning effectiveness. BRIEF DESCRIPTION OF THE DRAWINGS
[0015] Figure 1 This is a graph of the pointer network short-term prediction model of the present invention;
[0016] Figure 2 It is the flow chart of the algorithm of the present invention;
[0017] Figure 3 This is a visualization diagram of the short-term prediction effect of the deep reinforcement pointer network of the present invention;
[0018] Figure 4 This is a comparison chart of the short-term prediction convergence effect of the deep enhanced pointer network of the present invention;
[0019] Figure 5 This is a comparison chart of the solution time effect of the deep reinforcement pointer network short-term prediction learning algorithm of the present invention. DETAILED DESCRIPTION
[0020] The specific embodiments of the present invention are described in more detail below with reference to the accompanying drawings.
[0021] While the following detailed description provides a thorough understanding of the present invention, the present invention can be implemented in a variety of different ways, which may differ from those described herein. Those skilled in the art will appreciate that appropriate adjustments and variations based on the core principles of the present invention can be made without departing from its essence. Therefore, the scope of protection of the present invention should not be limited to the specific embodiments described in detail herein.
[0022] The details include:
[0023] S1: Build scenario and obtain data: In a search area, obtain the location distribution N=(n0,n1,...,n n ), where i∈(1,2,...,n), the amount of remaining data for each sensor is d i , the drone uses v x Flying at a constant speed and v d The transmission rate is v p The ideal charging efficiency is used to charge the drone; at the same time, dynamic information of the drone and sensors is obtained at every moment, including the drone's location, current battery level, and the remaining data volume of the sensor points.
[0024] S2: Construct a pointer network short-term prediction model, such as Figure 1 As shown in Figure 1, the data acquired in S1 is encoded and fed into the encoder to generate an intermediate variable. The decoder then uses this intermediate variable to obtain the hidden layer state. The hidden layer is then combined with the current drone's dynamic information using an attention mechanism to generate short-term predictions to determine the drone's next action. This cycle continues until all sensor data has been collected.
[0025] Specifically include:
[0026] S2.1: Input status code:
[0027] State encoding: The current time point of the drone's battery and remaining data are used as a dynamic input set in Represents the battery level e of the drone at the current time t t and the remaining data As a dynamic input set. The power and location information of the drones in this set will change with time t. The charging pile and sensor location coordinate set is used as a static input set N = (n0, n1, ..., n n ), the data in this set will not change with the change of time t. Static input and dynamic input components are obtained The output of the state encoding As input to the encoder module.
[0028] S2.2: Encoder module: The encoder module uses the first embedding layer composed of a convolutional network to Collection Mapped to a one-dimensional vector As an intermediate variable input to the decoder, After the first embedding layer, we get n i , After the embedding layer, we get This module connects the states of each point and weakens the association between the points, making the output result independent of the sequence representation of the input state.
[0029] S2.3: Decoder module: The decoder is composed of a GRU network, which converts the output obtained in S2.2 As the initial input of the decoder module, the output is the hidden layer information h t And the current UAV position information x at each time step t After the second embedding layer composed of convolutional networks, we get At each time step of decoding, the hidden layer information h t and the drone's position at that moment Use the attention mechanism combined with short-term prediction model information to derive the probability distribution of each behavior strategy of the drone at the next moment in This behavior indicates the next sensor number the drone will head to for data collection or return to the charging station for recharging. Finally, a greedy strategy is used to select the action with the highest probability as the drone's next behavior. The short-term prediction model is constructed as follows.
[0030] S2.3.1: Short-term prediction model construction: By obtaining the current time t drone's location and power information, the power consumption of all possible actions of the drone at the next time t+1 is calculated. Calculate the remaining power information matrix and position information matrix when all possible actions of the drone are completed at time t+1, and then calculate the remaining power after returning to the charging station at time t+2 to obtain the remaining power matrix of the drone in two time steps. in
[0031] Where, e t The total power of the drone at the current moment, The power consumed by the charging pile returned at time t+2, The remaining power for the short-term prediction of the future at that moment.
[0032] S2.3.2 Attention Mechanism: Static Input and Dynamic Input Set at Time t After Embedding and the decoder output state h at decoding step t t , calculate the alignment vector
[0033] Where v a , W a are the training parameters.
[0034] The conditional probability is calculated by weighted accumulation of the context vector of the input sequence:
[0035] The value calculated using the embedded input and context vector is then normalized by the Glimpse mechanism softmax function to output the probability Combine short-term predictions to output the final action probability distribution in is the result obtained by accumulating and weighting the attention mechanism results through the Glimpse mechanism, λ is a hyperparameter, and v c 、W c is the training parameter.
[0036] S3: The network model is trained using the policy gradient method. Using the actor-critic framework, an actor network and a critic network are generated. The critic network evaluates the performance of the actor network and collaboratively updates the network parameters. The weight vector associated with the actor network is θ, and the weight vector associated with the critic network is φ. During training, m sample problems are drawn from M sets of data, and Monte Carlo simulation is used to generate a sequence of feasible solutions relative to the current policy. The network continuously updates the network parameters using the reward function to calculate the update gradient, ultimately achieving reward convergence.
[0037] The critic network update gradient formula is Where R i is the reward value obtained by solving the sequence, Represents the critic network's response to the i-th sample sequence Estimated reward value obtained.
[0038] The actor network update gradient formula is In the formula For the sequence The output solution Y i The probability product of the drone's behavior selection.
[0039] Reward function design includes:
[0040] S3.1: Design reward function. To avoid the impact of sparse rewards on network updates, the reward value assigned by the strategy is as follows: R = aT r +bT h +cT c +r
[0041] The parameters in the formula can be calculated through the dynamic information and indicators of the drone at each moment: Indicates the total time the drone takes to fly. π Indicates the total length of the distance flown by the UAV under this solution sequence; It represents the time required for the drone to hover over the sensor and collect data in a single flight. π Indicates the amount of data that the drone needs to collect to complete its mission; Indicates the time it takes for the drone to be fully charged at the charging station, E π represents the total amount of electricity consumed by the drone to complete the task; a, b, c are hyperparameters, which are the proportional weights of the reward function, and r is the penalty given to the drone when completing the task with negative power.
[0042] The algorithm flow chart of the present invention is as follows Figure 2 visible.
[0043] This paper constructs a model for the data collection problem of rechargeable drones and uses deep reinforcement learning to train and solve it. The trained model can directly perform end-to-end reasoning on input sequences, eliminating the need for traditional iterative problem-solving steps. By incorporating the concept of short-term prediction, it significantly improves problem-solving efficiency and understanding.
[0044] The results of the present invention are visualized as follows Figure 3 As shown, Figure 3 The result graph of 50 sensor collection points is shown in the figure, where the five-pointed star represents the location of the charging pile, the dot represents the location of the sensor, the dotted arrow represents the direction of the drone's flight, and the number above the sensor represents the remaining power of the drone after it completes the collection task at that location. Figure 4 As can be seen (hereinafter referred to as PN for the pointer network and the method of the present invention as SF-PN), Table 1 shows the reward values of the two algorithms after convergence and the improvement effect of the present invention for different sensor numbers (10-50). The smaller the reward value, the less time the drone takes to complete the data collection task:
[0045] Table 1
[0046]
[0047] When the number of sensors is small, the SF-PN algorithm will be in a relatively stable state from the beginning of training; when the number of sensors increases, the SF-PN algorithm and the PN algorithm will have a rapid convergence process in the first few rounds, and the performance of the SF-PN algorithm is better than the PN algorithm. In scenarios of the same scale, the SF-PN algorithm converges faster and better than the PN algorithm. The convergence rate of the algorithm has increased by more than 10%, and the optimal result of the algorithm has increased by 7% to 17%. This is due to the fusion of short-term predictions in the algorithm, which increases the probability that the drone will make a more optimal behavior strategy considering the current battery level. Figure 5It can be seen that the computational time of the proposed SF-PN algorithm is lower than that of the PN algorithm, and the computational efficiency is significantly improved with the increase in the number of sensors.
Claims
1. A data collection method for a rechargeable drone based on an enhanced pointer network, characterized in that: The following steps are involved: S1: Build a rechargeable drone scene and obtain drone-related data; S2: Build a pointer network short-term prediction model, perform state encoding on the data obtained in S1, and input the state encoding result into the encoder to obtain the intermediate variable; then use the decoder to obtain the hidden layer information through the intermediate variable, and use the attention mechanism combined with short-term prediction to obtain the drone's action at the next moment until all sensor data is collected. The specific implementation process is as follows: S2.1: Take the current battery level and remaining data of the drone as a dynamic input set in Represents the battery level e of the drone at the current time t t and the remaining data As a dynamic input set; the power and location information of the drones in this set change with time t; The charging pile and sensor position coordinate set is used as the static input set N = (n0, n1, ..., n n ), where there is a charging station n0 and n sensors collecting data. The data in this set does not change with the change of time t; static input and dynamic input components are obtained The output of the state encoding As input to the encoder module; The short-term prediction model is implemented as follows: by obtaining the current time t drone's position and power information, the power consumed by the drone in all possible actions at the next time t+1 is calculated. Calculate the remaining power information matrix and position information matrix when all possible actions of the drone are completed at time t+1, and then calculate the remaining power after returning to the charging station at time t+2 to obtain the remaining power matrix of the drone in two time steps. in Where, e t The total power of the drone at the current moment, The power consumed by the charging pile returned at time t+2, The remaining power for the short-term prediction of the future at that moment; S2.2: The encoder module uses the first embedding layer composed of a convolutional network to Collection Mapping to a one-dimensional vector As an intermediate variable input to the decoder, After the first embedding layer, we get n i , After the embedding layer, we get S2.3: The decoder is composed of a GRU network, which converts the output obtained in S2.2 As the initial input of the decoder module, the output is the hidden layer information h t And the current UAV position information x at each time step t After the second embedding layer composed of convolutional networks, we get At each time step of decoding, the hidden layer information h t and the drone's position at that moment Use the attention mechanism combined with short-term prediction model information to derive the probability distribution of each behavior strategy of the drone at the next moment in This behavior indicates the sensor number that the drone will go to next moment to collect data or return to the charging station to charge. Finally, a greedy strategy is used to select the action with the maximum probability value as the drone's next behavior. The attention mechanism is implemented as follows: through the static input and dynamic input set at time t after embedding and the decoder output state h at decoding step t t , calculate the alignment vector Where v a , W a are the training parameters; The conditional probability is calculated by weighted accumulation of the context vector of the input sequence: The value calculated using the embedded input and context vector is then normalized by the Glimpse mechanism softmax function to output the probability Combine short-term predictions to output the final action probability distribution in is the result obtained by accumulating and weighting the attention mechanism results through the Glimpse mechanism, λ is a hyperparameter, and v c 、W c is the training parameter; S3: Use the policy gradient method to train the pointer network short-term prediction model, adopt the Actor-Critic framework to generate the actor network and the critic network, and use the reward function to calculate the update gradient to continuously update and make the reward converge.
2. The rechargeable drone data acquisition method based on enhanced pointer network according to claim 1 is characterized in that: The drone-related data includes the distribution of sensors, charging piles, drone indicators, and dynamic information of drones and sensors at every moment; The drone indicators include: power consumption per meter during flight, flight distance per second, power consumption when collecting unit data, amount of data collected per second, and drone charging rate during charging; The dynamic information of the drone and the sensor at each moment includes: the location of the drone, the current battery level of the drone, and the remaining data volume of the sensor point.
3. The data acquisition method for a rechargeable drone based on an enhanced pointer network according to claim 2, characterized in that: The reward function is R = aT r +bT h +cT c +r; The parameters in the formula are calculated based on the dynamic information and indicators of the UAV at each moment: T r Indicates the total time the drone takes on the flight route; T h T represents the time required for the drone to hover over the sensor and collect data in a single flight; c represents the time it takes for the drone to be fully charged at the charging station; a, b, c are hyperparameters, which are the proportional weights of the reward function; r is the penalty given to the drone when completing the task with negative power.