Reinforcement learning method for unmanned aerial vehicle data collection oriented to information collision avoidance

Abstract

This invention relates to the field of wireless communication technology, and more particularly to a reinforcement learning method for UAV data collection aimed at avoiding information collisions. It solves the problem of information collisions between users transmitting information close to the UAV within a time slot. The technical solution is as follows: S1: Establish a UAV-ground user channel model; S2: Optimize the establishment of the problem model; S3: Train the agent and update the state-action value function using reinforcement learning: Q-learning method; S4: Train the agent in a custom environment; S5: Reset the environment parameters and conduct testing. The beneficial effects of this invention are: It optimizes the trajectory of the UAV and achieves efficient data collection.

Description

Technical Field

[0001] This invention relates to the field of wireless communication technology, and in particular to a reinforcement learning method for UAV data collection aimed at avoiding information collisions. Background Technology

[0002] In recent years, the explosive growth in the number of network users and their demands has placed higher demands on the effective transmission and successful collection of user data in future communication systems. Unmanned aerial vehicles (UAVs), with their characteristics of rapid deployment, high maneuverability, and flexibility, are particularly suitable for data transmission and collection in air-to-ground communication environments. They can quickly collect effective data from users by flexibly adjusting their positions and optimizing their trajectories, thereby improving the transmission performance of communication networks in complex environments.

[0003] For the trajectory optimization problem of UAVs, there are many traditional solutions, among which dynamic programming is a common optimization algorithm. It requires a relatively simple model and is heavily constrained by the state space; once the space is too large, it is prone to combinatorial explosion. Optimal control is another common method, but it needs to avoid communication environments with relatively complex models and parameters. These traditional methods do not consider heuristics and cannot achieve intelligent search. Once the state space is large or the environment changes, they are prone to getting stuck in local optima and are time-consuming. In intelligent programming algorithms, heuristic optimization search algorithms are widely used. The paper "Research and Application of Two Heuristic Optimization Algorithms" (2015) mentions that heuristic search is a search in the state space, evaluating each search position to obtain the best position, and then searching from that position until the target. The A* algorithm is the most commonly used heuristic algorithm. This method is generally applied in static programming. As the search space increases, the search time becomes too long, and its real-time performance cannot be guaranteed. To solve this problem, a series of derivative algorithms of the A* algorithm have been proposed, such as the three-dimensional sparse A* algorithm. This method can adjust the weight coefficients according to environmental changes and has high adaptability to dynamic environments. In addition, there is the sparse A* algorithm, mentioned in the existing paper "Research on UAV 3D Track Planning Based on Genetic Algorithm and Sparse A* Algorithm" (2018). This algorithm aims to improve search efficiency by reducing search nodes through partitioned search and combining flight constraints. However, the A* algorithm is still limited by space complexity and cannot balance search speed and target performance. Besides these, there are various other intelligent optimization algorithms such as genetic algorithms and ant colony algorithms. Each algorithm has its own advantages and disadvantages, but they generally suffer from drawbacks such as slow search speed, large memory consumption, and inability to guarantee real-time performance.

[0004] In IoT-based drone-assisted data collection scenarios, drones, as mobile data collectors, are not limited by ground transportation mobility and can achieve automatic search and access to node users, real-time path planning, obstacle avoidance, etc. In addition, during the data collection process, they have a high coverage of node users, which can reduce the delay of data collection. For the problem of multi-target data collection by UAVs, there are many optimization methods. For example, the existing paper "Intelligent Path Planning for UAVs in IoT Data Collection" (2021) proposes a global path planning method for UAV data collection to maximize data collection benefits while minimizing the time spent by UAVs in collecting node data. The paper "Research on UAV-Assisted IoT Information Age Optimization Method" (2022) uses dynamic programming to find the optimal flight trajectory of UAVs so that the node data information age collected by UAVs along this trajectory is minimized, which involves the search and grouping problem of the optimal data collection position of UAVs. The paper "Minimizing Energy Consumption of UAV-Assisted Wireless Powered Networks under Information Age Constraint: A DQN-Based Method" (2021) proposes to focus on the UAV-assisted data collection problem based on energy harvesting technology within the framework of deep reinforcement learning. By jointly optimizing the transmission opportunities of ground nodes and the flight trajectory of UAVs, the information age of nodes is minimized, so that UAVs can effectively use harvested energy for data collection while avoiding crashes due to lack of energy. Most current research aims to optimize data collection methods for unmanned aerial vehicle (UAV) systems, simplifying or characterizing the data collection process between the UAV and the user to achieve maximum user data throughput or minimum data transmission latency within a given timeframe or limited UAV flight energy. This also mitigates the potential for data collisions that can occur when multiple users send data within a single time slot.

[0005] In summary, existing technologies suffer from slow search speed, high memory consumption, and inability to guarantee real-time performance. Furthermore, they fail to consider the issue of collisions between user information transmitted from nearby UAVs within a single time slot. Summary of the Invention

[0006] The purpose of this invention is to provide a reinforcement learning method for drone data collection aimed at avoiding information collisions.

[0007] To achieve the above-mentioned objectives, the present invention adopts the following technical solution:

[0008] A reinforcement learning method for drone data collection aimed at information collision avoidance includes the following steps:

[0009] S1: Establish a channel model between the UAV and ground users;

[0010] S2: Establishing the optimization problem model;

[0011] S3: Train the agent and update the state-action value function, i.e., the Q-value function, using reinforcement learning: Q-learning. π (s,a);

[0012] S4: Train the agent in a custom environment;

[0013] S5: Reset environment parameters for testing; in the original training environment, reset the user information sending status, that is, the user sends information with probability p according to the Poisson process for testing.

[0014] Step S1 specifically includes:

[0015] The drone starts from its initial position at a constant speed V and altitude H within a finite time T. Arrive at the destination

[0016] The effective flight domain of the UAV is [x0, x T ]×[y0,y T ];

[0017] Where: (x0, y0) are the initial coordinates of the UAV, ... and (x0, y0) are the initial coordinates of the UAV. T ,y T ) represents the coordinates of the UAV endpoint, further (x t ,y t () represents the coordinates of the UAV's location in time slot t;

[0018] Based on this, a channel model for UAV-ground users with large-scale path loss in the sub-6GHz band is established.

[0019] Among these constraints are imposed on the randomness of user-sent information, the effective communication range between users and drones, and user information collisions.

[0020] Randomness of user-sent messages: Consider that users send messages every 10 time slots according to a Poisson process (a basic independent incremental process that accumulates the number of occurrences of random events). Each time slot generates a 0-1 array of 5 user-sent messages; 0 means that the user sent a message in the current time slot; otherwise, it means that the user did not send a message in the current time slot.

[0021] Effective communication range limits for users and drones: Provide effective communication thresholds (0.95) for users and drones respectively. Even if multiple users send information in the current time slot, if the drone is too far away from the user (when the user's transmission and rate are below the communication threshold), the user information still cannot be successfully transmitted to the drone.

[0022] User Information Collision: Consider the issue of user information collision when multiple users send information in the current time slot and the drone's information receiving distance limit is met. Set a collision threshold range (1, 1.01), and select the two users with the highest information intensity for information collision judgment. Other users are considered as noise. The judgment is based on whether the ratio of the two information intensities is within the set collision threshold range. If it is within the range, it means that the two user information has collided, and the drone did not receive information in that time slot. Otherwise, the drone received user information with a higher information intensity (if the ratio of the two information intensities exceeds the threshold range, i.e., exceeds 1.01, because before this judgment, the information intensities were sorted by size, and the dividend is larger than the divisor, i.e., the information of the user corresponding to the dividend of the information intensity ratio is selected for reception).

[0023] Step S2 specifically includes:

[0024] By finding the optimal trajectory for the UAV (a trajectory that satisfies the target, i.e., the UAV starts flying from the initial position, reaches the destination in time T, and maximizes the user information transmission delay during the flight), the delay at which the UAV successfully receives user signals within a finite flight time is minimized (the final result is the minimum value obtained during training). The average transmission signal delay of all users accumulated in each time slot is expressed as t. average / k (t), that is, the final accumulated average total delay of the user's transmitted signal is t. average / k (T), then the optimization problem is expressed as:

[0025] min t average / k (T)

[0026]

[0027]

[0028] r k ≥r th_k (c)

[0029] r m / r n >r col (d)

[0030] d UAV-k ≤d th_UAV (e)

[0031] in: The two-dimensional position of the UAV in time slot t, i.e. st: constraint condition: subject to; The UAV arrives at its destination at time T, and the effective communication domain radius of the UAV is d. th_UAV; distance d between the drone and the user UAV-k ;

[0032] Equation (a) represents the flight of the UAV within a finite time T;

[0033] Equation (b) indicates that the UAV must stop at the maximum position of the effective flight domain when time T is reached;

[0034] € indicates that the user's effective communication range is limited, and the distance limit is represented by the sum of the digits and the rate.

[0035] Where the sum rate r k r is inversely proportional to the distance from the drone to user k. th-k The set sum and rate threshold;

[0036] (d) represents the judgment of user information collision, where r m r n Let r represent the sum rate values ​​of the two users with the highest information transmission strength in the current time slot. col This represents the sum and rate threshold related to collision detection;

[0037] € indicates that the effective communication range of the drone is limited; if the distance between the drone and the user exceeds the threshold d... th_UAV In this case, the drone will be unable to collect information.

[0038] Step S3 specifically includes:

[0039] Set initial values ​​for each element in the state, and ensure that each training session starts from this initial state after the episode ends; using the Q-learning algorithm, the steps include:

[0040] S3.1. UAV State Space Design: The state space design of the UAV is obs = [x t ,y t A GU ,t,t average / k (t)], where (x UAV ,y UAV This represents the location information of the drone on the two-dimensional grid in the current time slot;

[0041] A GUThis is a drone trajectory guidance array, an 8-element array where each element index represents the drone's directional range (8 in total). It displays the directional range of users who can successfully transmit information in the current time slot, with each element showing a trade-off value for the distance between these users and the drone. In each time slot, only one index in this array has a non-zero value, indicating that a user is transmitting information in that direction during the current time slot, and the element at that index represents a trade-off value for the distance between the drone and the user, considering whether to avoid a user's distance from the drone after a collision with their information. If the element at that index is 0, it means that no user is successfully transmitting information in that direction. The presence of a user transmitting information in the direction of a non-zero index during the current time slot guides the UAV to fly in that direction to some extent.

[0042] t represents the current time slot; t average / k (t) represents the average transmission delay accumulated for all users in the current time slot;

[0043] S3.2. Design of UAV Action Space: The UAV action space includes four discrete actions: "up", "down", "left" and "right", which are represented by the numbers 1, 2, 3 and 4, i.e. [1,2,3,4].

[0044] S3.3. Action Selection: If the current training has not reached the maximum number of repetitions, then proceed with the action selection step;

[0045] In the current state s, the UAV employs an ε-greedy action selection strategy (every time slot, a random number is generated, and the relationship between this random number and ε is compared; if the random number is greater than ε, exploration is performed: a random action is selected in the current state; otherwise, the action is selected using the function that produces the maximum Q-value in the current state). π The action chosen is 'a', which decreases as the training iterations (ε) increase, indicating a shift from exploration to exploitation. Initially, ε is 1, meaning there cannot be a random number (within the range of 0-1) larger than 1, indicating complete exploration at the beginning. Later, during high exploitation, ε becomes 0.05, meaning the random number will generally be larger than 0.05.

[0046] S3.4. Reward R Design: To avoid more non-target trajectories occurring within the specified time and to circumvent the limitations of sparse rewards, the reward, excluding the reward for reaching the target and the additional penalty reward at the target, is set as a continuous negative reward, specifically as follows:

[0047] R = c1r ave_diff +c2r d +c3r t

[0048] Where, r ave_diff =t average / k (t)-taverage / k (t+1) is the difference between the average total transmission delay of users in the current time slot and the average total transmission delay of users in the next time slot;

[0049] r d It is a function of the distance from the current time slot UAV to the endpoint; the closer the UAV is to the endpoint, the smaller its value becomes; r t It is a function of the time from the current time to the specified time; the longer the flight time, the larger its value. In addition, c1, c2, and c3 are adjustment weight values.

[0050] When a UAV reaches the destination exactly within the specified time, it is given a large positive reward; conversely, if the UAV does not reach the destination within the specified time, it receives a large negative reward.

[0051] S3.5. Obtain and update the Q-value function; use the tile-coded state linear mapping method (at this point, the approximate Q-value function is no longer represented as a table, but as a weighted vector). The parameterized function, where the number of weights is much smaller than the number of states.

[0052] The state space has too large a dimension, resulting in an excessively large number of states. If the classic table-based retrieval method is used to obtain the Q-value, it becomes impossible to traverse all states during training. Therefore,

[0053] By employing a linear approximation technique, the number of states is reduced by mapping them to a 0-1 feature vector with fewer elements than the existing number of states, thus improving the feasibility of traversing all states. The states are approximated by mapping them to eight overlapping tiles within a covered network of eight tiles, resulting in a 0-1 state vector, called the basis vector F. basis Finally, through Q(s,a,w)=w T *F basis We obtain an approximate Q value Q(s,a,w), where w represents the weight value of each Tiles;

[0054] Update the Q-value in state s; the Q-value update rule of its Q-learning algorithm is as follows:

[0055] Q(s,a)←(s,a)+α[R+γmaxQ(s′,a′)-Q(s,a)]

[0056] Where α is the learning rate, γ is the discount factor, maxQ(s′,a′) represents the maximum action value of the agent in the next time slot state s′, and a is the action value.

[0057] Step S4 specifically includes:

[0058] Repeat step S3, continuously updating the Q-value using the Q-value update rule to optimize the UAV's action selection strategy. This yields the average time taken for the UAV to successfully receive a user's transmitted signal, minimizing the time taken during flight (assuming the UAV can reach the destination position from the initial position via T during all training processes, and the final calculated user transmission delay converges to the minimum). Training ends when the maximum number of steps for each training session is reached, and training is complete when the total number of training iterations is reached. After successful training, the optimal UAV action selection strategy is obtained.

[0059] In practical use, this invention:

[0060] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0061] 1. This invention utilizes Q-learning to replace traditional optimization algorithms for UAV trajectory optimization and achieves efficient data collection. For the random and variable communication environment, Q-learning enables the UAV to continuously adjust its trajectory to adapt to its random and variable characteristics, resulting in the optimal strategy that minimizes the average time required to complete effective data collection for one user.

[0062] 2. In this invention, a function approximation method is used to solve the problem of excessively large state space. The mapped action value function Q(s,a,w) obtained by linear approximation is used to replace the actual action value function Q in the tabular form. π (s,a) (The action-value function is used to estimate how good it is for an agent to choose a certain action in a certain state. This "good" is defined as the expected reward in the future. It represents the expected reward that can be obtained by choosing action a in state s and then choosing action according to policy π). After the transformation, the amount of data samples and memory space required for traversal are greatly reduced, improving the convergence speed and learning efficiency.

[0063] 3. To address the user information collision avoidance problem, this invention adds an array A to the UAV state space design to guide the UAV's trajectory direction. GU This array represents the direction for selecting the optimal information-transmitting user for the drone in each time slot, after weighting the distance between the user and the drone and the probability of collisions between the sending users. The addition of this array significantly improves performance regarding the objective of minimizing the average user transmission latency. Attached Figure Description

[0064] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used together with the embodiments of the invention to explain the invention and do not constitute a limitation thereof.

[0065] Figure 1 This is a schematic diagram of the overall process of the present invention.

[0066] Figure 2 This is a flowchart of the reinforcement learning Q-learning process of this invention.

[0067] Figure 3 This is a schematic diagram of the system model of the present invention. Detailed Implementation

[0068] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. Of course, the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0069] Example 1

[0070] See Figure 1 and Figure 3 ,like Figure 3 This demonstrates a UAV uplink transmission scenario where K ground users (Gus) are fixedly distributed within the UAV's defined feasible domain Φ, with each ground user having an effective communication domain. The UAV flies at a constant altitude and constant speed, acting as a mobile base station to collect information from the ground users. It's important to note that when the UAV passes through the shared communication range of some ground users, information collisions between these users must be considered; the UAV cannot communicate with multiple ground users simultaneously. In the current time slot t, the UAV starts from its initial position... The mission begins, and within a limited flight time T, while minimizing collisions with ground user information, it rapidly collects sufficient ground user information and minimizes user information transmission latency until it reaches its destination. The technical solution provided by this invention is as follows:

[0071] A reinforcement learning method for drone data collection aimed at information collision avoidance includes the following steps:

[0072] S1: Establish a channel model between the UAV and ground users;

[0073] S2: Establishing the optimization problem model;

[0074] S3: Train the agent and update the state-action value function, i.e., the Q-value function, using reinforcement learning: Q-learning. π (s,a);

[0075] S4: Train the agent in a custom environment;

[0076] S5: Reset environment parameters for testing; in the original training environment, reset the user information sending status, that is, the user sends information with probability p according to the Poisson process for testing.

[0077] Step S1 specifically includes:

[0078] The drone travels from its initial position within a finite time T (set to 51s in this embodiment) at a constant speed V (m / s) and altitude H (m). Arrive at the destination

[0079] The effective flight domain of the UAV is [x0, x T ]×[y0,y T ];

[0080] Where: (x0, y0) are the initial coordinates of the UAV, ... and (x0, y0) are the initial coordinates of the UAV. T ,y T ) represents the coordinates of the UAV endpoint, further (x t ,y t () represents the coordinates of the UAV's location in time slot t;

[0081] Based on this, a channel model for UAV-ground users with large-scale path loss in the sub-6GHz band is established.

[0082] Among these constraints are imposed on the randomness of user-sent information, the effective communication range between users and drones, and user information collisions.

[0083] Randomness of user-sent messages: Consider that users send messages every 10 time slots according to a Poisson process (a basic independent incremental process that accumulates the number of occurrences of random events). Each time slot generates a 0-1 array of 5 user-sent messages; 0 means that the user sent a message in the current time slot; otherwise, it means that the user did not send a message in the current time slot.

[0084] Effective communication range limits for users and drones: Provide effective communication thresholds (0.95) for users and drones respectively. Even if multiple users send information in the current time slot, if the drone is too far away from the user (when the user's transmission and rate are below the communication threshold), the user information still cannot be successfully transmitted to the drone.

[0085] User Information Collision: Consider the issue of user information collision when multiple users send information in the current time slot and the drone's information receiving distance limit is met. Set a collision threshold range (1, 1.01), and select the two users with the highest information intensity for information collision judgment. Other users are considered as noise. The judgment is based on whether the ratio of the two information intensities is within the set collision threshold range. If it is within the range, it means that the two user information has collided, and the drone did not receive information in that time slot. Otherwise, the drone received user information with a higher information intensity (if the ratio of the two information intensities exceeds the threshold range, i.e., exceeds 1.01, because before this judgment, the information intensities were sorted by size, and the dividend is larger than the divisor, i.e., the information of the user corresponding to the dividend of the information intensity ratio is selected for reception).

[0086] Step S2 specifically includes:

[0087] By finding the optimal trajectory for the UAV (a trajectory that satisfies the target, i.e., the UAV starts flying from the initial position, reaches the destination in T (i.e., 51s), and maximizes the user information transmission delay during the flight), the delay in which the UAV successfully receives user signals within a finite flight time (51s) is minimized (the final result is the minimum value obtained during training). The average transmission signal delay of all users accumulated in each time slot is expressed as t. average / k (t), that is, the final accumulated average total delay of the user's transmitted signal is t. average / k (T), then the optimization problem is expressed as:

[0088] min t average / k (T)

[0089]

[0090]

[0091] r k ≥r th_k (c)

[0092] r m / r n >r col (d)

[0093] d UAV-k ≤d th_UAV (e)

[0094] in: The two-dimensional position of the UAV in time slot t, i.e. st: constraint condition: subject to; The UAV arrives at its destination at time T, and the effective communication domain radius of the UAV is d.th_UAV ; distance d between the drone and the user UAV-k ;

[0095] Equation (a) represents the flight of the UAV within a finite time T;

[0096] Equation (b) indicates that the UAV must stop at the maximum position of the effective flight domain when time T is reached;

[0097] (c) indicates that the user's effective communication range is limited, and the distance limit is represented by the sum and rate;

[0098] Where the sum rate r k r is inversely proportional to the distance from the drone to user k. th-k The set sum and rate threshold;

[0099] (d) represents the judgment of user information collision, where r m r n Let r represent the sum rate values ​​of the two users with the highest information transmission strength in the current time slot. col This represents the sum and rate threshold related to collision detection;

[0100] (e) indicates that the effective communication range of the drone is limited, and if the distance between the drone and the user exceeds the threshold d. th_UAV In this case, the drone will be unable to collect information.

[0101] Step S3 specifically includes:

[0102] Set initial values ​​for each element in the state, and ensure that each training session starts from this initial state after the episode ends; using the Q-learning algorithm, the steps include:

[0103] S3.1. UAV State Space Design: The state space design of the UAV is obs = [x t ,y t A GU ,t,t average / k (t)], where (x UAV ,y UAV This represents the location information of the drone on the two-dimensional grid in the current time slot;

[0104] A GUThis is a drone trajectory guidance array, an 8-element array where each element index represents the drone's directional range (8 in total). It displays the directional range of users who can successfully transmit information in the current time slot, with each element showing a trade-off value for the distance between these users and the drone. In each time slot, only one index in this array has a non-zero value, indicating that a user is transmitting information in that direction during the current time slot, and the element at that index represents a trade-off value for the distance between the drone and the user, considering whether to avoid a user's distance from the drone after a collision with their information. If the element at that index is 0, it means that no user is successfully transmitting information in that direction. The presence of a user transmitting information in the direction of a non-zero index during the current time slot guides the UAV to fly in that direction to some extent.

[0105] t represents the current time slot; t average / k (t) represents the average transmission delay accumulated for all users in the current time slot;

[0106] S3.2. Design of UAV Action Space: The UAV action space includes four discrete actions: "up", "down", "left" and "right", which are represented by the numbers 1, 2, 3 and 4, i.e. [1,2,3,4].

[0107] S3.3. Action Selection: If the current training has not reached the maximum number of training sessions (custom setting, set to 50,000 in this embodiment), then the action selection step is executed;

[0108] In the current state s, the UAV employs an ε-greedy action selection strategy (every time slot, a random number is generated, and the relationship between this random number and ε is compared; if the random number is greater than ε, exploration is performed: a random action is selected in the current state; otherwise, the action is selected using the function that produces the maximum Q-value in the current state). π The action chosen is 'a', which decreases as the training iterations (ε) increase, indicating a shift from high exploration to high utilization. Initially, ε is 1, meaning there cannot be a random number (within the range of 0-1) larger than 1, indicating complete exploration at the beginning. Later, during high utilization, ε becomes 0.05, meaning the random number will generally be larger than 0.05.

[0109] S3.4. Reward R Design: To avoid more non-target trajectories occurring within the specified time and to circumvent the limitations of sparse rewards, the reward, excluding the reward for reaching the target and the additional penalty reward at the target, is set as a continuous negative reward, specifically as follows:

[0110] R = c1r ave_diff +c2r d +c3r t

[0111] Where, r ave_diff=t average / k (t)-t average / k (t+1) is the difference between the average total transmission delay of users in the current time slot and the average total transmission delay of users in the next time slot;

[0112] r d It is a function of the distance from the current time slot UAV to the endpoint; the closer the UAV is to the endpoint, the smaller its value becomes; r t It is a function of the time from the current time to the specified time; the longer the flight time, the larger its value. In addition, c1, c2, and c3 are adjustment weight values.

[0113] When a UAV reaches the destination exactly within the specified time, it is given a large positive reward; conversely, if the UAV does not reach the destination within the specified time, it receives a large negative reward.

[0114] S3.5. Obtain and update the Q-value function; use the tile-coded state linear mapping method (at this point, the approximate Q-value function is no longer represented as a table, but as a weighted vector). The parameterized function, at this time the number of weights will be much smaller than the number of states (the number of weights defined in this embodiment is 2048)

[0115] The state space has too large a dimension, resulting in an excessively large number of states. If the classic table-based retrieval method is used to obtain the Q-value, it becomes impossible to traverse all states during training. Therefore,

[0116] By employing a linear approximation technique, the number of states is reduced by mapping them to a 0-1 feature vector with fewer elements than the existing number of states, thus improving the feasibility of traversing all states. The states are approximated by mapping them to eight overlapping tiles within a covered network of eight tiles, resulting in a 0-1 state vector, called the basis vector F. basis Finally, through Q(s,a,w)=w T *F basis We obtain an approximate Q value Q(s,a,w), where w represents the weight value of each Tiles;

[0117] Update the Q-value in state s; the Q-value update rule of its Q-learning algorithm is as follows:

[0118] Q(s,a)←(s,a)+α[R+γmaxQ(s′,a′)-Q(s,a)]

[0119] Where α is the learning rate, γ is the discount factor, maxQ(s′,a′) represents the maximum action value of the agent in the next time slot state s′, and a is the action value.

[0120] Step S4 specifically includes:

[0121] Repeat step S3, continuously updating the Q-value using the Q-value update rule to optimize the UAV's action selection strategy. The goal is to minimize the average time taken for the UAV to successfully receive a user's transmitted signal during the flight time (assuming the UAV can reach the destination position from the initial position in 51 seconds throughout all training processes, the final calculated user transmission delay converges to the minimum). Training ends when the maximum number of steps (51 steps, i.e., one step per second) is reached in each training session, and training is complete when the total number of training iterations (50,000) is reached. Successful training yields the optimal UAV action selection strategy.

[0122] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A reinforcement learning method for UAV data collection aimed at information collision avoidance, characterized in that, Includes the following steps: S1: Establish a channel model between the UAV and ground users; the UAV operates within a limited time... At a constant speed and height From the initial position Arrive at the destination ; The effective flight range of the drone is ; in: The initial coordinates of the UAV, For the coordinates of the UAV endpoint, further... Here are the coordinates of the UAV's location in time slot t; Based on this, a channel model for UAV-ground users with large-scale path loss in the sub-6 GHz band is established; Among them, constraints are imposed on the randomness of user-sent information, the effective communication range between users and drones, and user information collisions; S2: Establishing the optimization problem model; by finding the optimal trajectory of the UAV, minimizing the delay in the UAV successfully receiving user signals within a limited flight time; expressing the average transmission signal delay of all users accumulated in each time slot as... That is, the final accumulated average total delay of user transmission signal is The optimization problem can then be expressed as: ； (a)； (b)； (c)； (d)； (e)； in: ; The effective communication domain radius of the UAV is T, which is exactly when the UAV reaches its destination. Distance between drone and user ; Equation (a) represents the drone's performance within a finite time. Domestic flight; Equation (b) represents the UAV It must stop at the maximum position of the effective flight zone; (c) indicates that the user's effective communication range is limited, and the distance limit is represented by the sum and rate; Among them, the rate From drones to users The distance is inversely proportional to the distance. The set sum and rate threshold; (d) represents the judgment of user information collision, where , These represent the sum rate values ​​of the two users with the highest information transmission strength in the current time slot. This represents the sum and rate threshold related to collision detection; (e) indicates that the effective communication range of the drone is limited, and if the distance between the drone and the user exceeds the threshold. In this case, the drone will be unable to collect information; S3: Use reinforcement learning: Q-learning method to train the agent and update the state-action value function, i.e., the Q-value function: ; Set initial values ​​for each element in the state, and ensure that each training session starts from the initial state after each episode ends; utilize the Q-learning algorithm, including the following steps: S3.

1. The drone's state is designed as follows: ,in, This indicates the location information of the drone on the two-dimensional grid in the current time slot; It is an array of drone trajectory guidance. Indicates the current time slot; This represents the average transmission latency accumulated across all users in the current time slot; S3.

2. The UAV's action space includes four discrete actions: "up," "down," "left," and "right," represented by the numbers 1, 2, 3, and 4, respectively. ; S3.

3. If the current training session has not reached the maximum number of repetitions, proceed with the action selection step; The drone in its current state The following uses -greedy action selection strategy selects actions , The number of training sessions increases, decreasing from large to small, thus shifting from exploration to utilization; S3.

4. To prevent more non-target trajectories from appearing within the specified time and to avoid the limitations of sparse rewards, the rewards, except for the reward for reaching the target and the additional penalty reward at the target, are set as continuous negative rewards, as follows: ； in, It is the difference between the average total transmission delay of users in the current time slot and the average total transmission delay of users in the next time slot; It is a function of the distance from the current time slot UAV to the endpoint; the closer the UAV is to the endpoint, the smaller its value becomes. It is a function of the time from the current time to the specified time; the longer the flight time, the larger its value. Furthermore, , , It is the adjustment weight value; When a UAV reaches the destination exactly within the specified time, it is given a large positive reward; conversely, if the UAV does not reach the destination within the specified time, it receives a large negative reward. S3.

5. Obtain and update the Q-value function; utilize the tile-coded state linear mapping method; By employing a linear approximation technique, the number of states is reduced by mapping them to a 0-1 feature vector with fewer elements than the existing number of states, thus improving the feasibility of traversing all states. The states are approximated by mapping them to eight smaller tiles with overlapping portions among the eight covered tiles, resulting in a 0-1 state vector, called the basis vector. Finally passed To obtain an approximate Q value ,in This represents the weight value of each Tiles; Update status The Q-value is given below; the Q-value update rule of its Q-learning algorithm is as follows: Q(s,a)←Q(s,a)+α[R+γmaxQ(s′,a′)-Q(s,a)]; in For learning rate, As a discount factor, Indicates the agent's state in the next time slot. The maximum action value is 'a', where 'a' is the action value. S4: Train the agent in a custom environment; S5: Reset environment parameters for testing; in the original training environment, reset the user information sending status, that is, the user sends information with probability p according to the Poisson process for testing.

2. The reinforcement learning method for UAV data collection oriented towards information collision avoidance according to claim 1, characterized in that, Step S4 specifically includes: Repeat step S3, continuously update the Q value using the Q value update rule, optimize the drone's action selection strategy, and obtain the average time that minimizes the time required for the drone to successfully receive a user's transmitted signal within the flight time. The training ends when the maximum number of steps for each training session is reached, and the training is completed when the total number of training iterations is reached. After successful training, the optimal strategy for drone action selection is obtained.

Images