A multi-target connected unmanned relay control method and device based on reinforcement learning

Abstract

The application provides a multi-target connected unmanned relay control method and device based on reinforcement learning, which is aimed at an emergency communication scene containing a command center, a relay unmanned aerial vehicle and multiple mobile task nodes, utilizes an attention mechanism to aggregate features of variable-length multi-node observation data, processes a historical state sequence by introducing a memory-enhanced deep reinforcement learning model, extracts a belief state and outputs a flight acceleration instruction of the relay unmanned aerial vehicle, simultaneously constructs a multi-target reward function containing communication quality and flight energy consumption, dynamically adjusts weights by using an adaptive balance factor based on real-time signal strength, and performs model training in combination with signal offset exploration and a priority sequence experience playback mechanism. The application solves the technical problems of poor generalization ability, easy falling into local optimization and difficulty in dynamically balancing each optimization target of existing methods when facing dynamic changes in the number of task nodes, partial observability of the environment and multi-target conflicts of communication and energy consumption.

Description

Technical Field

[0001] This invention relates to the technical field of unmanned aerial vehicle (UAV) communication, and in particular to a multi-target connected unmanned relay control method and apparatus based on reinforcement learning. Background Technology

[0002] In emergency communication scenarios following natural disasters such as earthquakes and floods, ground communication infrastructure is often severely damaged. Unmanned aerial vehicles (UAVs), with their advantages of flexible deployment and high probability of line-of-sight communication, are ideally suited as aerial relay nodes to quickly establish temporary communication links between command centers and ground mission nodes. Within relay communication networks, a key issue in emergency rescue is how to dynamically adjust the UAV's position in real time based on the movement of ground nodes to ensure the quality and stability of the communication link.

[0003] Current research on UAV relay deployment largely assumes complete environmental information, requiring precise location, trajectory, and global channel state information for all nodes, and often employs heuristic algorithms such as genetic algorithms and particle swarm optimization for solutions. However, in complex real-world rescue scenarios, the environment is highly dynamic and uncertain, and UAVs often only acquire partial observation information, resulting in a severe partial observability problem. Traditional optimization algorithms often struggle to converge in real time and are prone to getting trapped in local optima when faced with incomplete information and highly dynamic environments.

[0004] While reinforcement learning methods have advantages in dynamic decision-making, conventional reinforcement learning algorithms struggle to effectively handle information gaps in partially observable environments and adapt to the input dimension mismatch challenges arising from dynamic changes in the number of task nodes. Furthermore, existing optimization schemes often neglect the dynamic balance between communication quality and UAV flight energy consumption, making it difficult to effectively extend network service time while ensuring communication reliability. Therefore, achieving multi-objective coordinated optimization of communication and energy consumption in complex dynamic environments has become a critical issue urgently needing to be addressed in the field of UAV relay technology. Summary of the Invention

[0005] This invention aims to solve the technical problems of existing UAV relay deployment methods, such as poor generalization ability, easy getting trapped in local optima, and difficulty in dynamically balancing various optimization objectives when facing dynamic changes in the number of task nodes, partially observable environment, and multiple conflicts between communication and energy consumption.

[0006] To address the aforementioned problems, this invention provides a multi-objective connected unmanned relay control method and apparatus based on reinforcement learning. This method is applicable to emergency communication network scenarios comprising a command center node, relay UAV nodes, and multiple mobile task nodes.

[0007] For mobile task nodes with dynamically changing numbers, this invention designs an attention mechanism to extract and aggregate features from multi-source observation data. The relay UAV first acquires a set of observations from task nodes, each observation including relative position, received signal strength indication, and task priority. To address the issue of variable input dimensions, the system encodes the observation of each task node as a value vector. and key vector And use the policy network to generate a global query vector. Then, the attention score for each task node is calculated. This score measures the importance of a task node in the current network state, and is calculated using the following formula:

[0008]

[0009] in Let be the dimension of the key vector. This represents the transpose of the key vector. Based on this score, the normalized attention weights are calculated using the Softmax function. To explicitly model the competitive relationships between multiple tasks:

[0010]

[0011] Finally, a fixed-dimensional aggregated observation vector is generated by weighted summation of the value vectors of all task nodes. :

[0012]

[0013] This mechanism enables the algorithm to adapt to any number of task nodes and automatically focus on key nodes based on communication conditions.

[0014] Specifically, the encoding process is implemented using a learnable linear transformation matrix. For the first... The original observation feature vector of each task node The system utilizes three independent weight matrices , and Linear projections are performed on the vector to extract feature representations in different semantic spaces. Key vector Sum value vector The calculation formula is as follows:

[0015]

[0016]

[0017]

[0018] in, The feature vector of the relay drone itself is used to generate a global query request; For the first Feature vectors of each ground mission node; , , This is the transformation matrix that is updated synchronously with the network parameters during training. Through this linear projection, the observation data of the original physical space is mapped to a high-dimensional latent feature space, thereby enhancing the model's ability to represent complex environmental features.

[0019] To address the issues of partial observability and temporal correlation in emergency communication environments, this invention constructs a deep reinforcement learning framework based on memory enhancement. This framework introduces a memory network to compress historical observation action sequences, thereby inferring the underlying belief states of the environment. The system constructs a fixed-length historical sequence. Including the current moment and the past State-action pair at each moment:

[0020]

[0021] The memory network consists of a multilayer perceptron (MLP) and a long short-term memory network. Its input is a historical sequence, and its output is a memory feature vector that encodes long-term dependencies. Both the memory-enhanced policy network and the value network utilize this memory feature vector for evaluation and decision-making, thereby making more robust control decisions in dynamic environments.

[0022] To reduce flight energy consumption while ensuring communication reliability, this invention designs a composite reward function, including a communication quality reward, a flight energy consumption reward, and an adaptive balancing mechanism. Communication quality reward The aim is to maximize the weighted minimum received signal strength indication. Based on the minimum received signal strength indication... The value is divided into an interruption zone (minimum received signal strength indicator is less than the connection interruption threshold), a hold zone (minimum received signal strength indicator is between the connection interruption threshold and the high reliability communication threshold), and a high reliability zone (minimum received signal strength indicator is greater than the high reliability communication threshold), and calculated using a piecewise function:

[0023]

[0024] in The connection interruption threshold, High-reliability communication threshold. Flight energy consumption bonus. Designed to guide the drone at the optimal cruising speed Smooth flight. Actions deviating from optimal speed and excessive acceleration. Apply secondary punishment:

[0025]

[0026] in At the current speed, and These are weighting coefficients. Adaptive balance factor. This is used to dynamically adjust the weights of communication and energy consumption. A sigmoid function-based adjustment mechanism is constructed, prioritizing communication optimization when signal strength is low and shifting to energy conservation when signal strength is high.

[0027]

[0028] in This is the steepness coefficient. To establish a balancing threshold. Total reward function. The final expression is:

[0029]

[0030] To improve sample utilization and convergence stability, this invention employs a priority-based sequence experience replay mechanism. This mechanism uses experience sequences spanning multiple consecutive time steps as the basic unit for data storage, preserving the temporal correlation of state transitions. The experience replay pool stores sequence samples containing states, actions, rewards, next states, and priorities. Sampling priority is also considered. Due to time difference error Decision. The time difference error measures the deviation from the current state value prediction and is calculated as follows:

[0031]

[0032] The priority calculation formula is:

[0033]

[0034] in To prevent small constants with zero probability, importance sampling weights are introduced to correct distribution bias caused by preferential sampling. :

[0035]

[0036] These weights are used to adjust the magnitude of the loss function during gradient updates. The importance sampling weights are inversely proportional to the sampling probability of the samples to ensure the unbiasedness of the gradient estimation.

[0037] To avoid the strategy getting trapped in local optima, this invention introduces a signal offset-guided exploration strategy during action selection. Unlike traditional random noise, this invention uses biased pink noise, utilizing signal gradient information to guide the exploration direction. (Exploration noise) The update incorporates target orientation offset, temporal correlation, and random perturbations:

[0038]

[0039] in Pointing towards the task node with the worst communication quality. The noise correlation coefficient. This is a Gaussian random vector. Noise amplitude. With the number of training iterations It exhibits a cosine-like decay pattern to balance exploration and development:

[0040]

[0041] This mechanism ensures extensive exploration in the early stages of training, and convergence to the optimal policy in the later stages.

[0042] This invention also provides a UAV relay positioning device based on memory-enhanced deep reinforcement learning, comprising a multi-source state perception module, a memory decision control module, a multi-objective reward calculation module, and a model training and optimization module. These modules work collaboratively to perform the aforementioned tasks of state acquisition, belief state extraction, adaptive reward calculation, and parameter update.

[0043] Furthermore, this invention constructs a mathematical model based on a partially observable Markov decision process, aiming to jointly optimize the flight trajectory and communication coverage quality of a relay UAV. The objective function defined by the system is to maximize the cumulative expected reward, which encompasses the weighted minimum received signal strength indication and the flight energy consumption penalty. To solve for this optimal strategy, a memory-enhanced deep reinforcement learning model employs an Actor-Critic architecture. The Critic value network updates its parameters by minimizing the Bellman error. The loss function (Loss) of the value network is defined as the mean squared error between the predicted value and the target value at the current time step.

[0044]

[0045] In the formula, For value network parameters, For experience replay pool, The target network computes the time-difference objective value. The policy network (Actor) updates its parameters by maximizing the output of the value network, and its gradient update direction follows the deterministic policy gradient theorem:

[0046]

[0047] In the formula, For policy network parameters, This is a deterministic strategy function.

[0048] Compared with the prior art, the present invention has at least the following beneficial effects:

[0049] This invention successfully solves the technical challenges of traditional reinforcement learning algorithms in handling dynamic changes in the number of task nodes and input dimension mismatch by introducing an attention mechanism, and achieves adaptive feature extraction for any number of moving nodes.

[0050] This invention utilizes a memory-enhanced network structure to process historical observation sequences, effectively overcoming some observability problems commonly found in emergency communication scenarios, and significantly improving the decision-making stability of UAVs in complex dynamic environments.

[0051] The adaptive reward aggregation mechanism proposed in this invention can prioritize connection protection when communication deteriorates and automatically switch to energy-saving mode when communication is good, thus achieving the best balance between communication quality and task endurance.

[0052] The signal offset exploration mechanism designed in this invention utilizes signal gradient information in the physical environment, which significantly improves the efficiency and convergence speed of finding the optimal relay location compared to traditional random exploration strategies. Attached Figure Description

[0053] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0054] Figure 1a This is a diagram of the memory-enhanced deep reinforcement learning model architecture provided in an embodiment of the present invention;

[0055] Figure 1b This is a schematic diagram illustrating the data interaction relationship between a policy network and a value network, which includes memory units, provided in an embodiment of the present invention.

[0056] Figure 2 This is a schematic diagram of an emergency communication network scenario provided by an embodiment of the present invention, showing the spatial topology relationship between the command center, relay drones and multiple mobile task nodes;

[0057] Figure 3This is a flowchart of the single relay location optimization method provided in the embodiments of the present invention, which shows the complete implementation steps from environmental observation, feature aggregation, decision generation to model update;

[0058] Figure 4 This is a feature aggregation structure diagram of the attention mechanism provided in the embodiments of the present invention, which shows the calculation logic of query vector, key vector and value vector and the aggregation process of variable-length observation data. Detailed Implementation

[0059] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0060] The present invention will now be described in detail through specific embodiments.

[0061] See Figure 2 , Figure 2 This paper illustrates an emergency communication network scenario applied in an embodiment of the present invention. The scenario includes a fixed-location ground command center node, a relay drone node with adjustable flight position, and multiple randomly moving ground mission nodes. The relay drone, acting as an aerial base station, is responsible for establishing communication links between the command center and each mission node. Considering the complexity of the disaster site environment, the number of ground mission nodes may dynamically increase or decrease as the rescue process progresses, and their movement trajectories are highly random. The relay drone needs to continuously adjust its position in three-dimensional space to maximize the communication quality of all links while minimizing its own flight energy consumption.

[0062] See Figure 1a This invention demonstrates the algorithm / model architecture for deep reinforcement learning based on memory enhancement proposed in this paper. Figure 1b This paper demonstrates the data interaction between the policy network and the value network, which incorporate memory units. The architecture follows an Actor-Critic structure and primarily consists of two core networks: the policy network and the value network. To address the issue of varying numbers of mobile nodes, an attention mechanism feature extraction module is designed at the network input. To address the observability and temporal relevance of the environment, a Long Short-Term Memory (LSTM) network is introduced as the memory unit after feature extraction. The policy network is responsible for outputting continuous flight acceleration control commands based on the current environmental belief state, while the value network is responsible for evaluating the merits of these commands in terms of long-term cumulative rewards.

[0063] See Figure 3 , Figure 3The specific workflow of this embodiment of the invention is illustrated. First, at each decision-making moment, the relay UAV uses onboard sensors and communication modules to perceive the environmental state. The observed information mainly includes two categories: one is the UAV's own motion state, including three-dimensional coordinates, three-dimensional velocity vector, and three-dimensional acceleration vector; the other is the communication link state, including the received signal strength indication between the UAV and the command center, as well as the relative position coordinates, received signal strength indication, and task priority between the UAV and each mobile task node.

[0064] When acquiring communication link state parameters, this embodiment uses an air-to-ground probabilistic line-of-sight channel model to simulate signal transmission characteristics in complex rescue environments. The wireless channel state between the relay UAV and the ground node depends on the line-of-sight transmission probability, which is related to the UAV's flight altitude and elevation angle. The formula for calculating the line-of-sight transmission probability is as follows:

[0065]

[0066] In the formula, The angle of elevation is in degrees. and Let be an environmental constant. The corresponding non-line-of-sight transmission probability is... Therefore, the path loss of a communication link is obtained by weighting the line-of-sight path loss and the non-line-of-sight path loss:

[0067]

[0068] Specifically, line-of-sight path loss and non-line-of-sight path loss are expressed as follows:

[0069]

[0070]

[0071] In the formula, For transmission distance, For carrier frequency, At the speed of light, and These represent the average additional losses under line-of-sight and non-line-of-sight propagation, respectively. The Received Signal Strength Indication (RSSI) is then calculated by subtracting the aforementioned path loss from the transmit power and serves as the observation input for the deep reinforcement learning model.

[0072] The state update of the relay UAV follows a discrete-time kinematic model. The decision interval is set to... In the At time, the acceleration vector based on the policy network output The velocity vector of the drone and position vector Update according to the following formula:

[0073]

[0074]

[0075] To ensure flight safety, the system performs constraint checks on the updated state. If the calculated speed exceeds the maximum flight speed... Then it will be truncated: Similarly, location updates are also restricted to a preset rescue airspace. If the location exceeds the boundary, it will be forcibly pulled back to the boundary and an additional boundary penalty will be imposed.

[0076] Because the number of task nodes is not fixed, the dimension of the directly input observation vector changes over time. This embodiment adopts the following approach: Figure 4 The attention mechanism shown aggregates this variable-length data. The system first inputs the observed features of each task node into a fully connected layer for encoding. For the i-th task node, its relative position, signal strength, and priority are mapped to a key vector K and a value vector V. Simultaneously, a query vector Q is generated using the UAV's own state or global context information. Subsequently, the relevance between the query vector and each key vector is calculated, i.e., the attention score. This score reflects the urgency of a specific task node under the current network conditions. For numerical stability, the dot product result is scaled. The specific formula for calculating the attention score is as follows:

[0077]

[0078] In the formula, This represents the dimension of the key vector. Next, the Softmax function is used to transform the attention scores of all nodes into attention weights in the form of a probability distribution. This step allows the model to automatically select the node with the worst current communication quality or the highest priority, assigning it greater weight.

[0079]

[0080] Finally, the value vectors of all task nodes are weighted and summed using this weight to obtain a fixed-dimensional aggregated feature vector Z. This vector highly condenses the key state information of all moving nodes at the current moment:

[0081]

[0082] After obtaining the aggregated feature vector at the current moment, the system concatenates it with the UAV's own state and the action at the previous moment to form the current complete observation vector. To address the issue that the UAV cannot perceive the future movement trends of ground nodes in some observable aspects, this invention constructs a historical state sequence. This sequence contains observation data from the current moment and several past moments. This historical sequence is input into a Long Short-Term Memory (LSTM) network. The LSTM unit utilizes its internal forget gate, input gate, and output gate mechanisms to extract the implicit environmental evolution patterns, i.e., belief states, from the time-series data. These belief states not only contain current spatial location information but also implicitly contain information about the movement speed and direction of ground nodes, thus providing a basis for predicting future channel changes.

[0083] Specifically, the Long Short-Term Memory (LSTM) network incorporates cell states. To preserve long-term memories. At each time step The LSTM unit is based on the current input vector That is, the current item and the hidden state of the previous time step in the historical sequence. Perform the following calculations and updates:

[0084] First, the Gate of Oblivion The decision is based on the cell state at the previous moment. How much information is discarded:

[0085]

[0086] Secondly, the input gate This determines how much new information to store in the cell state and simultaneously generates candidate memory units. :

[0087]

[0088]

[0089] Next, update the current cell state. Combining the retained parts of the old state with the input parts of the new state:

[0090]

[0091] Finally, output gate Determine how much information to output as the current hidden state (i.e., belief state). :

[0092]

[0093]

[0094] In the formula, It is the Sigmoid activation function. It represents the Hadamah accumulation. and These are the corresponding weight matrices and bias terms, respectively. The final output is... They are fed into the policy network and value network for decision-making.

[0095] After the relay UAV updates its position based on the acceleration command output by the policy network, the system needs to calculate the immediate reward to guide model training. This embodiment designs an adaptive multi-objective reward function to dynamically balance communication quality and flight energy consumption. The total reward function consists of a weighted average of communication reward and energy consumption reward, with the weights determined by an adaptive balancing factor.

[0096]

[0097] in, This indicates the minimum received signal strength across all communication links at the current moment. To automatically switch optimization targets based on communication urgency, this invention constructs an adaptive balance factor based on the Sigmoid function. When the detected minimum signal strength is below a preset balance threshold, this factor approaches 1, causing the multi-target reward value to primarily depend on the communication quality reward value, thus forcing the system to focus on communication recovery. When the minimum signal strength is above the preset balance threshold, the minimum signal strength is within a safe range, and this factor decreases, causing the multi-target reward value to primarily depend on the flight energy consumption reward value, thus gradually shifting the system towards energy-efficient flight.

[0098]

[0099] Communication quality awards The design employs a piecewise function approach to finely guide drone behavior. When extremely low signal strength leads to an interruption, a large, fixed penalty is applied; when the signal strength is within the connectivity maintenance range, a positive reward that increases linearly with signal strength is given; and when the signal strength reaches a high reliability standard, a fixed reward is applied to avoid over-optimization.

[0100]

[0101] Flight energy reward The penalty is based on the drone's kinematic parameters. It consists of two parts: one is a penalty for deviating from the optimal cruise speed, and the other is a penalty for excessive maneuvering acceleration. Both are in quadratic form to smoothly constrain the drone's flight attitude.

[0102]

[0103] The aforementioned flight energy consumption reward function is derived from a simplified physical power model of a rotary-wing UAV. Theoretically, the flight propulsion power of a rotary-wing UAV... With flight speed The following nonlinear relationship exists:

[0104]

[0105] In the formula, and These are the leaf profile induced power and the hovering induced power, respectively. The rotor blade tip velocity, The average induced velocity during hovering. The drag coefficient of the fuselage. air density, For rotor realism, Let be the rotor disk area. This embodiment approximates the convexity of the physical model near the optimal cruise speed using a quadratic penalty term, thereby effectively guiding the UAV into a low-energy flight state while reducing computational complexity.

[0106] To improve the convergence speed of the algorithm in complex environments, this embodiment incorporates special designs in the action generation and parameter update stages. In the action generation stage, this invention introduces a signal offset exploration mechanism. Unlike traditional zero-mean Gaussian noise, this invention constructs biased pink noise. The mean vector of this noise is not fixed at zero, but dynamically points towards the task node with the worst current communication quality. This is equivalent to utilizing signal gradient information in the physical environment to guide random exploration, making the UAV more likely to move towards areas with weak signals. Simultaneously, the noise amplitude decays cosinely with the number of training rounds, ensuring extensive exploration in the early stages of training and convergence to the optimal strategy in later stages.

[0107]

[0108] During the parameter update phase, this invention employs a priority sequence experience replay mechanism. The system stores the collected experience sequences in a replay pool. During sampling, uniform sampling is not used; instead, the temporal difference error (TD-error) for each sequence is calculated. The TD-error reflects the accuracy of the current model's prediction of the sample's value; a larger error indicates greater learning value for the sample. The positive correlation formula between sampling probability and TD-error is as follows:

[0109]

[0110] After correcting the loss function using importance sampling weights, the parameters of the policy network and the value network are updated simultaneously using the gradient descent algorithm until the model converges, ultimately obtaining the optimal relay policy that can adapt to the dynamic environment.

[0111] Because priority sampling alters the distribution of the original empirical data, directly using the sampled data for training can lead to model bias. Therefore, this embodiment introduces importance sampling weights to correct the direction of gradient updates. For the... The importance sampling weights of the sampled empirical sequences are calculated as follows:

[0112]

[0113] In the formula, The current total number of samples in the experience replay pool. This is the sampling probability of the sample. The annealing parameter gradually increases from its initial value to 1 as training progresses. To normalize the weights and maintain a stable update magnitude, the weights actually used are the normalized values:

[0114]

[0115] When calculating the loss function of the value network, this weight is multiplied into the mean squared error term, so that the model can estimate the overall value function unbiasedly while focusing on learning high-error samples.

[0116] To ensure training stability during model iterative optimization, this embodiment introduces a target policy network and a target value network. The parameters of the target network are updated with a lag using a soft update mechanism, rather than directly copying the parameters of the main network. Specifically, for each time step, the temporal difference target value is calculated. Then, use the target network to evaluate the value at the next time step:

[0117]

[0118] In the formula, As a discount factor, and These are the parameters for the target value network and the target policy network, respectively. After the main network parameters are updated, the parameters for the target network are... Perform a soft update using the following formula:

[0119]

[0120] In the formula, This is a soft update coefficient, and its value range is usually very small to ensure that the changes in the target network parameters are smooth, thereby suppressing the oscillation and divergence phenomenon during the training process.

[0121] In this embodiment, the network structure of the memory-enhanced deep reinforcement learning model is configured such that both the policy network (Actor) and the value network (Critic) contain an input layer, two fully connected hidden layers, and an output layer. The activation function used is the Modified Linear Unit (ReLU). The output layer of the policy network uses the hyperbolic tangent function (Tanh) for activation to constrain action instructions. Within the interval, then multiply by the maximum acceleration. Scaling is performed.

[0122] The output layer of the policy network does not directly output physical acceleration values, but instead uses a hyperbolic tangent activation function. Constrain the output value to a dimensionless interval. Internally, to prevent network output divergence. Then, by adjusting the drone's maximum maneuvering acceleration... Multiplication maps dimensionless motion back to the real physical motion space. The mathematical expression for this mapping process is:

[0123]

[0124]

[0125] This processing method not only ensures that the output actions always meet the dynamic constraints of the drone, avoiding illegal actions, but also utilizes... The linearity of the function near zero allows the drone to generate precise control commands when fine-tuning its position is required.

[0126] In the specific parameter update process, this embodiment uses the Adam adaptive moment estimation optimization algorithm to minimize the loss function. An independent adaptive learning rate is designed for each network parameter by calculating the first and second moment estimates of the gradient. For the... gradient of the next iteration The update process is as follows:

[0127] First-moment estimate of gradient calculation and second-order moment estimation :

[0128]

[0129]

[0130] and The exponential decay rates are estimated by the first and second moments, respectively.

[0131] To eliminate initialization bias, bias corrections are applied to the first and second moments to obtain corrected estimates. and :

[0132]

[0133]

[0134] Based on the set base learning rate sum of numerical stability constants Update network parameters :

[0135]

[0136] By applying the Adam optimizer, the algorithm proposed in this invention can rapidly decrease in the early stages of training and automatically reduce the learning step size when approaching the optimal solution, thereby effectively avoiding oscillations on the loss function surface and significantly improving the convergence accuracy and stability of the deep reinforcement learning model.

[0137] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A multi-objective connected unmanned relay control method based on reinforcement learning, characterized in that, Includes the following steps: S1. Obtain environmental observation information: Use the relay UAV to collect its own motion status information in real time, as well as the received signal strength and relative position coordinates between it and the fixed command center node and each mobile task node; S2. State Feature Aggregation and Sequence Construction: For mobile task nodes with dynamically changing numbers, an attention mechanism is used to extract and aggregate features from multi-source observation data, generating a fixed-dimensional current state feature vector. This current state feature vector is then combined with historical state-action pairs to construct a historical state sequence containing temporal information. Specifically, the feature extraction and aggregation using the attention mechanism includes: encoding and mapping the observation and task priority information of each mobile task node into key and value vectors, respectively, and generating a query vector using the relay UAV's own state or global context information; determining the attention weight of each mobile task node by calculating the correlation between the query vector and the key vector; and weighted summing the value vectors of all mobile task nodes based on the attention weights to generate a fixed-dimensional aggregated observation vector. This aggregated observation vector is then concatenated with the relay UAV's own state and the action of the previous moment to form the current state feature vector. S3. Action command generation: The historical state sequence is input into a pre-constructed memory-enhanced deep reinforcement learning model. The long short-term memory network inside the model is used as a memory unit to process the data of the historical state sequence. The hidden state output by the long short-term memory network is used as the belief state. The belief state represents the dynamic changes of the environment. Based on the belief state, the flight acceleration command of the relay UAV is output. S4. Position Update and Reward Feedback: Control the relay UAV to execute the flight acceleration command to update its spatial position, and simultaneously calculate a multi-target reward value based on the updated network status. The multi-target reward value is obtained by weighting the communication quality reward value and the flight energy consumption reward value. The weight ratio of the communication quality reward value and the flight energy consumption reward value is adaptively and dynamically adjusted according to the real-time monitored minimum received signal strength indication. The calculation of the multi-target reward value adopts an adaptive balancing mechanism, constructing an adaptive balancing factor that changes non-linearly with the minimum received signal strength indication. This factor dynamically adjusts the composition weight of the total reward value: when the monitored minimum received signal strength indication is lower than a preset balancing threshold, the multi-target reward value mainly depends on the communication quality reward value; when the monitored minimum received signal strength indication is higher than the preset balancing threshold, the multi-target reward value mainly depends on the flight energy consumption reward value. S5. Model Iteration and Optimization: The empirical data containing the current state feature vector, flight acceleration command, multi-target reward value and the next moment state feature vector is stored in the empirical replay pool, and empirical samples are extracted from the empirical replay pool using the priority sequence sampling mechanism to train and update the memory-enhanced deep reinforcement learning model.

2. The multi-objective connected unmanned relay control method based on reinforcement learning according to claim 1, characterized in that, In step S3, when the memory-enhanced deep reinforcement learning model generates acceleration commands, a signal offset exploration mechanism is introduced. An exploration noise vector pointing to the direction of the task node with the worst current communication quality is superimposed on the deterministic policy output. The amplitude of the noise vector decreases in a cosine manner with the number of training iterations.

3. The multi-objective connected unmanned relay control method based on reinforcement learning according to claim 1, characterized in that, In step S4, the communication quality reward value is calculated using a piecewise function: a fixed negative penalty is given in the communication interruption region, a positive reward that increases linearly with signal strength is given in the connection maintenance region, and a fixed positive reward is given in the high-reliability communication region.

4. The multi-objective connected unmanned relay control method based on reinforcement learning according to claim 1, characterized in that, In step S4, the calculation logic of the flight energy consumption reward value is related to the flight state of the relay UAV: ​​the optimal cruising speed of the relay UAV is set, the deviation between the real-time flight speed and the optimal cruising speed is calculated, and a first type of energy consumption penalty is applied according to the deviation to guide the UAV to maintain the optimal speed; the real-time flight acceleration of the relay UAV is obtained, and a second type of energy consumption penalty is applied according to the magnitude of the acceleration to guide the UAV to reduce abrupt maneuvers and maintain stable flight.

5. The multi-objective connected unmanned relay control method based on reinforcement learning according to claim 1, characterized in that, In step S5, the experience replay pool adopts a priority sequence experience replay mechanism, specifically including: using experience sequences containing multiple consecutive time steps as the basic unit for storing data to preserve the temporal correlation of state transitions; calculating the time difference error of each experience sequence and using the absolute value of the time difference error as the basis for determining the sampling priority, so that experience sequences with larger prediction errors have a higher probability of being sampled; at the same time, introducing importance sampling weights when calculating the loss function to correct the bias caused by non-uniform sampling, the importance sampling weights being inversely proportional to the sampling probability of the samples to ensure the unbiasedness of gradient estimation.

6. A multi-objective connected unmanned relay control device based on reinforcement learning, used to implement the method described in any one of claims 1 to 5, characterized in that, include: The multi-source state perception module is used to acquire the relay UAV's own motion state, relative position and link quality information with the command center and all mobile task nodes in real time, and to aggregate the variable-length multi-node observation data into a fixed-dimensional state feature vector using an attention mechanism. The memory decision control module has a built-in deep reinforcement learning model containing memory units, which is used to receive historical state sequences, extract belief states, and generate flight acceleration commands for relay drones. The multi-objective reward calculation module is used to calculate a comprehensive reward value that includes communication quality and flight energy consumption based on real-time minimum received signal strength indication and flight energy consumption data, using adaptive dynamic adjustment. The model training optimization module manages the priority sequence experience replay pool, generates training samples through a signal offset exploration mechanism, and updates the network parameters of the memory decision control module using a gradient descent algorithm.

Images