Unmanned aerial vehicle maritime emergency formation control method based on deep reinforcement learning

Abstract

This invention discloses a method for controlling UAV maritime emergency formations based on deep reinforcement learning, comprising the following steps: UAVs collect their own state information and the state information of neighboring UAVs to obtain raw state feature data; based on a preset physical distance threshold, an adjacency matrix of the formation structure graph is generated; the raw state feature data and the adjacency matrix of the formation structure graph are input into the UAV control strategy network, and an improved TD3 algorithm is used to output continuous action decisions; each UAV performs autonomous flight control based on the continuous action decisions; in the event of communication limitations or link interruption, each UAV makes autonomous decisions based on the adjusted continuous action decisions; during the autonomous flight of the UAVs based on the continuous action decisions, task execution data is collected, stored in an experience replay pool, and used for batch training of the UAV control strategy network. This invention employs structure perception and deep reinforcement learning to achieve autonomous, efficient, and collaborative control of UAV formations.

Description

Technical Field

[0001] This invention relates to the field of intelligent control technology for unmanned aerial vehicles (UAVs), and in particular to a UAV maritime emergency formation control method based on deep reinforcement learning. Background Technology

[0002] With the rapid development of unmanned aerial vehicle (UAV) technology, multi-UAV collaboration is becoming increasingly common in maritime emergency search and rescue and disaster monitoring scenarios. Existing technologies mostly employ centralized control or pre-set formation strategies, achieving formation maintenance, trajectory tracking, and basic obstacle avoidance by real-time sensing of UAV positions, speeds, and some environmental information. Some research attempts to introduce deep reinforcement learning into UAV swarm control, utilizing algorithms such as DDPG and MADDPG to improve the autonomy and adaptability of individual UAVs in complex environments. In recent years, structured perception and graph neural network methods have begun to be applied to multi-agent systems to express formation structure and adjacency relationships, thereby improving the local collaboration and information fusion capabilities among multiple UAVs.

[0003] While the aforementioned methods have improved the control capabilities of multi-UAV systems to some extent, they still have some limitations. First, traditional solutions lack the ability to perceive the real-time dynamic structure and adaptively reconstruct formations among UAVs, making it difficult to guarantee the robustness of the overall system under conditions of limited communication or unstable links. Second, most existing deep reinforcement learning methods only process individual states and fail to effectively integrate dynamic formation structure information, resulting in limited global response capabilities when facing dynamic task allocation and complex environmental changes.

[0004] Therefore, how to provide a method for controlling unmanned aerial vehicle (UAV) maritime emergency formations based on deep reinforcement learning is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0005] One objective of this invention is to propose a deep reinforcement learning-based method for unmanned aerial vehicle (UAV) maritime emergency formation control. This invention combines multi-UAV structure perception, graph neural networks, and an improved TD3 algorithm. It details how, through dynamic modeling of the formation structure, fusion of structured features, and optimization of continuous action strategies, autonomous coordination and formation control of UAVs in maritime emergency missions are achieved. This method possesses advantages such as strong structural adaptability, high formation coordination capability, and good robustness to complex environments.

[0006] The UAV maritime emergency formation control method based on deep reinforcement learning according to embodiments of the present invention includes the following steps: Each drone collects its own state information and the state information of neighboring drones to obtain raw state characteristic data; Based on the original state feature data and according to the preset communication reachability or physical distance threshold, the drone formation structure diagram is dynamically constructed, and the adjacency matrix of the formation structure diagram is generated. The original state feature data and the adjacency matrix of the formation structure graph are input into the UAV control strategy network. The front end of the UAV control strategy network integrates a graph neural network module and adopts an improved TD3 algorithm to output continuous action decisions. Each UAV performs autonomous flight control based on continuous action decisions, including formation maintenance, formation reconfiguration, path replanning, and task allocation. In the event of communication restrictions or link interruption, the continuous action decision is adjusted, and each UAV makes autonomous decisions based on the adjusted continuous action decision. During the autonomous flight of the UAV based on continuous action decisions, task execution data is periodically collected, stored in the experience replay pool, and used for batch training of the UAV control strategy network.

[0007] Optionally, the original state characteristic data includes the current position coordinates, speed, remaining battery power, and heading angle of each UAV, the current position coordinates, speed, remaining battery power, and heading angle of neighboring UAVs, the ambient wind speed, obstacle positions, and communication signal quality.

[0008] Optionally, the step of dynamically constructing a UAV formation structure diagram based on the original state feature data and according to preset dynamic formation adjustment rules, and generating the adjacency matrix of the formation structure diagram, specifically includes: Based on the original state feature data, extract the current position coordinates, communication signal quality, set of neighboring drones, and current position coordinates of each neighboring drone for each drone; For each pair of drones, the spatial distance between them is calculated in real time using the Euclidean spatial distance formula, and the communication signal quality between them at the current moment is obtained. Based on the preset dynamic formation adjustment rules, combined with the spatial distance and communication signal quality between each pair of drones, each pair of drones is judged. If the spatial distance between a pair of drones is less than or equal to the set physical distance threshold, and the communication signal quality between them is higher than or equal to the set signal quality threshold, then it is determined that there is a connection between the two drones; otherwise, no connection is established. Within each refresh cycle, each drone updates the edge determination result synchronously based on its own data and that of neighboring drones, and generates a local substructure edge set in a distributed and dynamic manner. After all local substructure edge sets are aggregated, a global drone formation structure diagram is formed. The results of the edge determination between all pairs of drone nodes are collected to form an adjacency matrix of the formation structure graph.

[0009] Optionally, the step of inputting the original state feature data and the adjacency matrix of the formation structure graph into the UAV control strategy network, and using the improved TD3 algorithm to output continuous action decisions specifically includes: The original state feature data and the adjacency matrix of the formation structure graph are used as inputs and synchronously input into the UAV control strategy network. At the front end of the UAV control strategy network, there is a graph neural network module. Each UAV is a node in the graph structure, and the adjacency matrix of the formation structure graph is the connection relationship. The original state feature data of each UAV and the original state feature data of neighboring UAVs are fused to generate the structure perception state feature vector of each UAV. The structural perception state feature vector of each UAV is input into the Actor network and the Critic network respectively to output the UAV's continuous action decision vector, first Q value and second Q value; The UAV control strategy network is trained and updated based on the TD3 reinforcement learning mechanism, and outputs continuous action decisions for each UAV.

[0010] Optionally, at the front end of the UAV control strategy network, a graph neural network module is provided, which generates structure-aware state feature vectors using the adjacency matrix of the formation structure graph as the edge relationships. Specifically, this includes: At the front end of the UAV control strategy network, a feature fusion module containing a multi-layer graph neural network is set up, with each UAV as a node in the graph structure, and the edge relationship between nodes is determined by the adjacency matrix of the formation structure graph. For each layer of the graph neural network, the original state feature data of each drone is used as the initial node feature, and the original state feature data of all neighboring drones that have an edge relationship with the drone are used as the neighbor node feature. At the same time, the information transmission weight between nodes is determined according to the adjacency matrix. In each layer of the graph neural network, the features of each node are iteratively updated by weighted aggregation of the features of neighboring nodes; Through information fusion and feature propagation via a multi-layer graph neural network, the structural perception state feature vector of each UAV is finally obtained.

[0011] Optionally, the step of inputting the structural perception state feature vector of each UAV into the Actor network and the Critic network respectively, and outputting the UAV's continuous action decision, first value function, and second value function specifically includes: The structural perception state feature vector of each UAV is input into the Actor network and Critic network in the UAV control policy network; The Actor network takes the structure-aware state feature vector as input and passes it through several layers of fully connected neural networks and activation functions to generate continuous action decision vectors. The Critic network takes the concatenation result of the structure-aware state feature vector and the continuous action decision vector as input. The Critic network includes two sets of value function sub-networks with the same structure but independent parameters. They are passed through several layers of fully connected neural networks and activation functions in sequence to output the first Q value and the second Q value, respectively. Both the Actor network and the Critic network employ a multi-layer fully connected structure adapted to the dimensions of the structure-aware state feature vector and the continuous action decision dimension.

[0012] Optionally, the training and updating of the UAV control policy network based on the TD3 reinforcement learning mechanism specifically includes: The UAV control strategy network is trained based on the improved TD3 algorithm. During the training process, the experience replay pool is first initialized, and the sampling batch size, learning rate, discount factor, and noise perturbation coefficient are set as training parameters. In each training iteration, a batch of samples are randomly sampled from the experience replay pool. Each sample includes the current structure-aware state feature vector, the current continuous action decision, the immediate reward, the new structure-aware state feature vector, and the task termination marker. The target Actor network is used to calculate the next continuous action decision based on the state feature vector of each new structure. The first Q-value and the second Q-value are calculated using the target Critic network, and the smaller of the two values ​​is taken as the target Q-value. The mean square loss is calculated based on the target Q value and the first and second value functions output by the current Critic network. After every few updates to the Critic network parameters, the Actor network takes the structure-aware state feature vector as input and aims to maximize the first Q value of the Critic network output. The parameters of the target Actor network and the target Critic network are iterated using a soft update method, and the adjacency matrix and structure-aware state features are dynamically updated in each iteration.

[0013] Optionally, each UAV performs autonomous flight control based on continuous action decisions, including formation maintenance, formation reconfiguration, path replanning, and task allocation, specifically including: Each UAV periodically acquires its own structure perception state feature vector and inputs it into the current Actor network to obtain the UAV's continuous action decisions, which include flight heading adjustment, speed control, and formation relative position transformation. Based on the flight heading adjustment made by continuous action decisions, the UAV flight control system adjusts the flight heading angle in real time. Speed ​​control based on continuous action decisions is achieved by dynamically adjusting the flight speed of the UAV flight control system. Based on the changes in the relative positions of the formation according to the continuous action decision, the UAV flight control system adjusts the relative positions of the formation. During flight control, the connectivity and coordination of the overall formation are monitored in real time, and path replanning and task reallocation are dynamically triggered based on changes in UAV formation, environmental dynamics and mission requirements.

[0014] Optionally, the step of adjusting continuous action decisions in the event of communication limitations or link interruption, with each UAV making autonomous decisions based on the adjusted continuous action decisions, specifically includes: In the event of communication restrictions or link interruption, each UAV periodically detects the communication status and signal quality between itself and neighboring UAVs, and dynamically updates the adjacency matrix of the formation structure graph based on the detection results, retaining only the edge relationships that currently meet the physical distance threshold and signal quality threshold conditions. Based on the adjacency matrix of the adjusted formation structure graph, each UAV redetermines its node connection relationship in the adjusted formation structure graph and updates its own structure perception state feature vector. The updated structure-aware state feature vector is input into the current Actor network to obtain the adjusted continuous action decision. Each UAV independently executes flight control based on the adjusted continuous action decision.

[0015] Optionally, during the autonomous flight process where the UAV makes decisions based on continuous actions, task execution data is periodically collected, stored in an experience replay pool, and used for batch training of the UAV control strategy network, specifically including: During the process of autonomous flight and cooperative control executed by UAVs based on continuous action decisions, each UAV periodically collects task execution data; The collected task execution data is stored in the experience replay pool in chronological order. Periodically or on-demand random sampling from the experience replay pool, with a batch size of [missing information]. Task execution data, The set sampling batch size; Using the batch task execution data obtained from sampling, the target Q-value is calculated, the Critic network loss function is calculated, the Actor network loss function is calculated, and the network parameters are updated in sequence.

[0016] The beneficial effects of this invention are: This invention effectively enhances the collaborative decision-making capabilities of UAVs in formation flight by introducing a structure-aware mechanism and feature fusion based on graph neural networks. Compared to control methods that rely solely on individual states or static topologies, this invention can reflect the dynamic structural relationships between UAVs in real time, achieving efficient modeling of local environmental changes and the overall formation structure, thus providing a richer and more accurate input basis for collaborative control strategies.

[0017] Employing an improved TD3 deep reinforcement learning algorithm, this invention not only achieves efficient policy optimization in a continuous action space but also integrates structural cooperative regularization terms and target action perturbation mechanisms, thereby enhancing the control network's adaptability to complex dynamic environments and changing task requirements. Through experience replay pooling and batch training methods, this invention can continuously optimize the control policy network parameters, improving the autonomous decision-making performance and global coordination capabilities of UAV formations in maritime emergency missions.

[0018] Furthermore, this invention designs a decentralized autonomous control mechanism for situations where communication is limited or links are interrupted. Each UAV can independently make flight decisions based on locally available information, achieving adaptive adjustment and robust control of the formation structure. Combining the above technical solutions, this invention demonstrates significant advantages in improving formation maintenance, formation reconfiguration, path replanning, and task allocation capabilities, meeting the practical application needs of multi-UAV systems in complex maritime emergency scenarios. Attached Figure Description

[0019] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings: Figure 1 The flowchart shows the UAV maritime emergency formation control method based on deep reinforcement learning proposed in this invention. Figure 2 This is a schematic diagram of the UAV control strategy network in the UAV maritime emergency formation control method based on deep reinforcement learning proposed in this invention. Figure 3 This is a schematic diagram illustrating the process of training the control strategy network using the improved TD3 algorithm in the UAV maritime emergency formation control method based on deep reinforcement learning proposed in this invention. Detailed Implementation

[0020] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, illustrating only the basic structure of the invention, and therefore only show the components relevant to the invention.

[0021] refer to Figure 1-3 A method for controlling unmanned aerial vehicle (UAV) maritime emergency formations based on deep reinforcement learning includes the following steps: Each drone collects its own state information and the state information of neighboring drones to obtain raw state characteristic data; Based on the original state feature data and according to the preset communication reachability or physical distance threshold, the drone formation structure diagram is dynamically constructed, and the adjacency matrix of the formation structure diagram is generated. The original state feature data and the adjacency matrix of the formation structure graph are input into the UAV control strategy network. The front end of the UAV control strategy network integrates a graph neural network module and adopts an improved TD3 algorithm to output continuous action decisions. Each UAV performs autonomous flight control based on continuous action decisions, including formation maintenance, formation reconfiguration, path replanning, and task allocation. In the event of communication restrictions or link interruption, the formation structure diagram is adjusted based on the current available information of the adjacency matrix of the formation structure diagram, and each UAV makes autonomous decisions based on the adjusted continuous action decision. During the autonomous flight of the UAV based on continuous action decisions, task execution data is periodically collected, stored in the experience replay pool, and used for batch training of the UAV control strategy network.

[0022] In this embodiment, the step of obtaining raw state feature data by each UAV collecting its own state information and the state information of neighboring UAVs specifically includes: Each drone collects its own status information through onboard sensors, including the drone's current position coordinates, speed, remaining battery power, and heading angle. Each drone collects status information of neighboring drones through a communication module. The status information of neighboring drones includes their current position coordinates, speed, remaining battery power, and heading angle. Each drone collects environmental parameter information, including ambient wind speed, obstacle location, and communication signal quality. The system splices together its own status information, the status information of nearby UAVs, and environmental parameter information according to a predetermined format to form raw status feature data.

[0023] In this embodiment, the step of dynamically constructing a UAV formation structure diagram based on the original state feature data and according to preset dynamic formation adjustment rules, and generating the adjacency matrix of the formation structure diagram, specifically includes: Based on the original state feature data, extract the current position coordinates, communication signal quality, set of neighboring drones, and current position coordinates of each neighboring drone for each drone; For each pair of drones, the spatial distance between them is calculated in real time using the Euclidean spatial distance formula, and the communication signal quality between them at the current moment is obtained. Based on the preset dynamic formation adjustment rules, and combined with the spatial distance and communication signal quality between each pair of drones, each pair of drones is judged. If the spatial distance between a pair of drones is less than or equal to the set physical distance threshold, and the communication signal quality between them is higher than or equal to the set signal quality threshold, then it is determined that there is a connection between the pair of drones; otherwise, no connection is established. Within each refresh cycle, each drone updates the edge determination result synchronously based on its own data and that of neighboring drones, and generates a local substructure edge set in a distributed and dynamic manner. After all local substructure edge sets are aggregated, a global drone formation structure diagram at the current moment is formed. The structure diagram includes a drone node set and an edge set at the current moment. The formation structure diagram is checked for structural sparsity, connectivity and redundancy. If the loss of connection edges caused by changes in physical distance, signal attenuation or the influence of obstacles is detected, the relevant thresholds are dynamically adjusted or local structure reconstruction is triggered to ensure smooth transmission of formation information and structural robustness. The results of edge determination between all pairs of drone nodes are collected to form an adjacency matrix of the formation structure graph with all drones as rows and columns. Each element in the adjacency matrix is ​​used to indicate whether there is an edge relationship between the corresponding two drones at the current time. The adjacency matrix fully reflects the structural connection state of the entire drone formation at the current time, and the adjacency matrix is ​​input into the subsequent drone control strategy network.

[0024] In this embodiment, the step of inputting the original state feature data and the adjacency matrix of the formation structure diagram into the UAV control strategy network, and using the improved TD3 algorithm to output continuous action decisions specifically includes: The original state feature data and the adjacency matrix of the formation structure graph are used as inputs and synchronously input into the UAV control strategy network. At the front end of the UAV control strategy network, there is a graph neural network module. Each UAV is a node in the graph structure, and the adjacency matrix of the formation structure graph is the connection relationship. The original state feature data of each UAV and the original state feature data of neighboring UAVs are fused to generate the structure perception state feature vector of each UAV. The structural perception state feature vector of each UAV is input into the Actor network and the Critic network respectively to output the UAV's continuous action decision vector, first Q value and second Q value; The UAV control policy network is trained and updated based on the TD3 reinforcement learning mechanism. This includes batch training using historical state, action, reward and next state data sampled from the experience replay pool, updating the Actor network and Critic network using a delayed policy update mechanism, and introducing noise perturbation during the target network action output process to enhance policy robustness. The system outputs continuous action decisions for each UAV to drive its flight heading adjustment, speed control, and formation relative position transformation, thereby achieving structure-aware collaborative control of multiple UAVs. The continuous action decisions include flight heading adjustment, speed control, and formation relative position transformation.

[0025] In this embodiment, the step of providing a graph neural network module at the front end of the UAV control strategy network, and generating a structure-aware state feature vector using the adjacency matrix of the formation structure graph as the connection relationship, specifically includes: At the front end of the UAV control strategy network, a feature fusion module containing a multi-layer graph neural network is set up, with each UAV as a node in the graph structure, and the edge relationship between nodes is determined by the adjacency matrix of the formation structure graph. For each layer of the graph neural network, the original state feature data of each drone is used as the initial node feature, and the original state feature data of all neighboring drones that have an edge relationship with the drone are used as the neighbor node feature. At the same time, the information transmission weight between nodes is determined according to the adjacency matrix. In each layer of the graph neural network, the features of each node are iteratively updated through weighted aggregation of neighbor node features, feature normalization, and nonlinear transformation operations. Graph convolution, graph attention, or other structured information fusion mechanisms can be used to jointly model and extract information of the target UAV node and its neighbor nodes. After information fusion and feature propagation through a multi-layer graph neural network, the structural perception state feature vector of each UAV is finally obtained. The structural perception state feature vector integrates information of the target UAV and the local structure of the formation at multiple adjacency levels, and can dynamically adapt to changes in the formation structure, thereby achieving a high-dimensional expression of formation coordination and local environmental state. The structural perception state feature vector of each UAV is output and then input into the Actor network and Critic network in the UAV control strategy network for intelligent decision-making and formation optimization control.

[0026] In this embodiment, the step of inputting the structural perception state feature vector of each UAV into the Actor network and the Critic network respectively, and outputting the UAV's continuous action decision, first value function and second value function specifically includes: The structural perception state feature vector of each UAV is input into the Actor network and Critic network in the UAV control policy network; The Actor network takes the structure-aware state feature vector as input and sequentially passes it through several layers of fully connected neural networks and activation functions to generate continuous action decision vectors. The continuous action decision vectors include action components for flight heading adjustment, action components for speed control, and action components for formation relative position changes. The Critic network takes the concatenation result of the structure-aware state feature vector and the continuous action decision vector as input. The Critic network includes two sets of value function sub-networks with the same structure but independent parameters. They are passed through several layers of fully connected neural networks and activation functions in sequence, and output the first Q value and the Q value respectively. Both the Actor network and the Critic network employ a multi-layer fully connected structure adapted to the dimensions of the structure-aware state feature vector and the continuous action decision dimension, enabling joint high-dimensional feature modeling and control output of individual UAV state information and local collaborative state information.

[0027] In this embodiment, the training and updating of the UAV control policy network based on the TD3 reinforcement learning mechanism specifically includes: The UAV control strategy network is trained based on the improved TD3 algorithm. During the training process, the experience replay pool is first initialized, and the sampling batch size, learning rate, discount factor, and noise perturbation coefficient are set as training parameters. In each training iteration, a batch of samples are randomly sampled from the experience replay pool. Each sample includes the current structure-aware state feature vector, the current continuous action decision, the immediate reward, the new structure-aware state feature vector, and the task termination marker. The target Actor network calculates the next consecutive action decision based on the state feature vector of each new structure: ; in, For the decision of the next continuous action, it means that after the output of the target Actor network is added with noise perturbation, the... Action decision-making for drones under new structural perception conditions. Let be the target Actor network, representing a network with the same structure as the current Actor network but whose parameters are obtained through soft updates, used to generate continuous action decisions for the next time step. Let be the new structure-aware state feature vector, representing the th The structural perception state features collected by the drone after it performs its actions. For noise disturbance, Number the drone; The formula is derived from the Twin Delayed Deep Deterministic Policy Gradient (TD3) algorithm in the field of deep reinforcement learning. This patent uses a structured perception state feature vector instead of the traditional "next state" based on the structured perception of multi-UAV formations to reflect the latest perception state of a single UAV in the global and local formation structure. At the same time, the noise perturbation setting can adaptively adjust its variance according to the connectivity of the UAV formation structure. left side of the formula and the right side All of these represent action decision vectors, with physical dimensions of "continuous control quantities in the UAV action space", namely component 1: heading angle (unit: radians or degrees), component 2: velocity (unit: meters per second), and component 3: spatial relative position transformation (unit: meters). The first and second value functions are calculated separately using the target Critic network, and the smaller value of the two is taken as the target Q value: ; in, For the target Q value, For instant rewards, As a discount factor, As a termination marker, Indicates taking the first The novel structural perception state feature vector of the UAV and the next-moment continuous action decision are used as inputs, and the smaller value among the value functions calculated by the first target Critic network and the second target Critic network is used. The first Q-value of the target Critic network is the output of the state feature vector perceived by the new structure and the continuous action decision in the next time step. The second Q-value of the target Critic network is the output of the state feature vector perceived by the new structure and the continuous action decision of the next time step. The instant reward represents the first The instantaneous reward value returned by the environment to the drone in its current structure-aware state and actions; the termination marker indicates the... Does the drone reach the mission termination state after the current action is performed? ,otherwise ; The target Q-value formula is derived from the target value update method of the TD3 algorithm in the field of deep reinforcement learning. Based on the TD3 target Q-value calculation formula, this application generalizes the state-action pair into multi-UAV structure perception feature vectors and continuous actions, and adds a task termination marker to improve the engineering practicality of multi-agent systems. Both the left and right sides of the formula represent "Q-values," which are the cumulative expected rewards of a state-action pair. Essentially, they are dimensionless real numbers, representing immediate rewards and the Q-values ​​output by the target Critic network. , Both the target Q value and the cumulative expected reward are numerical representations with the same dimensions. The discount factor and the termination mark are dimensionless coefficients. Therefore, both sides of the formula have the same dimensions and are dimensionless values ​​of the cumulative expected reward. Calculate the mean squared loss based on the target Q value and the first and second value functions output by the current Critic network: ; in, This is the loss function of the Critic network, used to measure the mean square error between the current output Q-value and the target Q-value of the Critic network, and used to guide the optimization of the Critic network parameters. Batch size represents the number of data samples taken from the experience replay pool in each training round. For the current Critic network, the first The first Q-value is the input-output structure perception state feature vector of the UAV and the current continuous action decision. For the current Critic network, the first The second Q-value is input and output using the structure-aware state feature vector of the UAV and the current continuous action decision. This is the structure-aware state feature vector. For the current continuous action decision, The target Q value; left side of the formula All items on the right are "mean squared error" results, which are "the average of the sum of squared errors between the current Q value and the target Q value". , , The physical dimensions of all are cumulative expected rewards (usually dimensionless real numbers or reward units), and the dimensions of the squared error are consistent. Therefore, the dimensions of both sides of the formula are consistent, and both are the mean square error of the cumulative reward. The loss function formula for the Critic network is derived from the mean squared error definition of the dual-Critic network in the TD3 algorithm of deep reinforcement learning. In TD3, the loss function is typically defined as: ; in, It is the current Q-value output by the Critic network. It is the target Q value. Batch size; This patent addresses the structural perception scenario of multi-UAV cooperative formation. It employs a dual-Critic network structure to calculate two Q values ​​separately, and then performs a weighted summation of the two values ​​with the target Q value after applying mean square error, thereby improving estimation accuracy and convergence stability. After every few updates to the Critic network parameters, the Actor network takes the structure-aware state feature vector as input and aims to maximize the first Q-value of the Critic network output. ; in, The loss function of the Actor network measures the value of the action output by the current Actor network policy under the first current Critic network. A structural cooperation regularization term is introduced to improve the consistency and cooperation of multi-UAV formations. The first current Critic network is the first The Q-value is calculated by taking the structure-aware state feature vector of the UAV and the continuous action decisions output by the current Actor network based on the structure-aware state feature vector as input. These are the regularization weight coefficients. For structural coordination regularization; The formula for the loss function of the Actor network is derived from the negative Q-value optimization strategy of the TD3 algorithm in the field of deep reinforcement learning. Combined with the innovation of this application for multi-UAV formation control, a structural cooperative regularization term is introduced to improve the consistency and cooperation of group decision-making. Both the left and right sides of the formula represent "loss function values," which are essentially a weighted sum of the negative mean of the Q-value (i.e., the cumulative reward) and the regularization term. Structural Co-regulation All are dimensionless real numbers or reward units. Since the coefficients are dimensionless, both sides of the formula have the same dimension, which are dimensionless real values ​​of the loss function. The parameters of the target Actor network and the target Critic network are iterated using a soft update method. That is, the parameters of the current Actor network and the current Critic network are weighted and merged with the parameters of the corresponding target Actor network and the target Critic network respectively. The updated target network parameters are equal to the soft update coefficient multiplied by the current network parameters, plus one and minus the soft update coefficient multiplied by the original target network parameters. The soft update coefficient ranges from zero to one. During each iteration of the training process, the adjacency matrix and structure-aware state features are dynamically updated to ensure that the current network and target network parameters and decision-making strategies adapt to the formation structure and environmental state in real time, thereby continuously improving the formation cooperative control capability and complex environment adaptation capability of the multi-UAV system.

[0028] In this embodiment, each UAV performs autonomous flight control based on continuous action decisions, including formation maintenance, formation reconfiguration, path replanning, and task allocation, specifically including: Each UAV periodically acquires its own structure perception state feature vector and inputs it into the current Actor network to obtain the UAV's continuous action decisions, which include flight heading adjustment, speed control, and formation relative position transformation. Based on continuous action decision-making, the UAV flight control system adjusts the flight heading angle in real time. Specifically, the heading adjustment value in the action decision component is superimposed with the current heading to obtain a new flight heading. Based on continuous action decision-making, the UAV flight control system dynamically adjusts the flight speed by superimposing the speed control value in the action decision component with the current speed to obtain a new flight speed. Based on continuous action decisions, the UAV flight control system adjusts the relative positions of the formation. Specifically, the position transformation value in the action decision component is used to correct the spatial coordinates of the UAVs relative to the formation centroid or reference point, so as to achieve formation maintenance and reconstruction among the UAVs. During flight control, the connectivity and coordination of the overall formation structure are monitored in real time. Based on changes in UAV formation, environmental dynamics and mission requirements, path replanning and task reallocation are dynamically triggered to optimize the collaborative division of labor among UAVs. The drone records and stores the status, actions, and environmental feedback data of each autonomous flight, providing a data foundation for experience replay pool sampling and policy network optimization.

[0029] In this embodiment, the step of adjusting the formation structure diagram based on the current available information of the adjacency matrix of the formation structure diagram under the condition of communication restriction or link interruption, and the autonomous decision-making of each UAV based on the adjusted continuous action decision specifically includes: In the event of communication restrictions or link interruption, each UAV periodically detects the communication status and signal quality between itself and neighboring UAVs, and dynamically updates the adjacency matrix of the formation structure diagram based on the detection results, retaining only the edge relationships that can currently communicate normally or meet the physical distance threshold and signal quality threshold conditions. Based on the adjacency matrix of the adjusted formation structure graph, each UAV redetermines its node connection relationship in the adjusted formation structure graph, and updates its own structure perception state feature vector with its own original state feature data and the adjacency matrix of the UAVs it can currently communicate with. The updated structure-aware state feature vector is input into the current Actor network to obtain the adjusted continuous action decision, which includes flight heading adjustment, speed control and formation relative position transformation. Each UAV independently executes flight control based on the adjusted continuous action decisions, including heading adjustment, speed regulation, and relative position transformation, to achieve decentralized collaborative decision-making and adaptive adjustment of formation structure under communication-limited conditions.

[0030] In this embodiment, during the autonomous flight of the UAV based on continuous action decisions, task execution data is periodically collected, stored in an experience replay pool, and the UAV control strategy network is batch trained, specifically including: During the process of UAVs performing autonomous flight and cooperative control based on continuous action decisions, each UAV periodically collects task execution data, which includes the current structure perception state feature vector, the current continuous action decision, the immediate reward, the task termination marker, and the new structure perception state feature vector after the action is executed. The collected task execution data is stored in the experience replay pool in chronological order. The experience replay pool is used to save the historical status, actions, rewards, next status and termination marker data of multiple drone nodes and multiple time steps. Periodically or on-demand random sampling from the experience replay pool, with a batch size of [missing information]. Task execution data, The set sampling batch size; Using the batch task execution data obtained from sampling, the target Q value is calculated, the Critic network loss function is calculated, the Actor network loss function is calculated, and the network parameters are updated in sequence. Through continuous batch training and parameter optimization, the UAV control strategy network can adaptively adjust to changes in the environment and formation tasks, thereby improving the collaborative formation control performance of the UAV multi-agent system and its autonomous operation capability in complex scenarios.

[0031] Example 1: To verify the feasibility of this invention in practice, it was applied to an emergency search and rescue drill in a complex coastal waterway. This scenario included multiple target water areas, involving dynamic weather changes, complex obstacle distribution, and multi-task coordination requirements. The drill assumed that several drones formed a formation and needed to complete full-coverage patrols of the target water areas, identification and location of floating objects, search for distressed targets, rapid delivery of supplies, and dynamic task reallocation within a short period. Traditional solutions struggle to maintain efficient coordination when communication is poor or the environment changes abruptly; formations are easily lost, and responses to obstacles and sudden environmental changes are not flexible enough, often resulting in missed searches, duplicate searches, and response delays.

[0032] The method of this invention first involves each UAV autonomously collecting its own and neighboring UAVs' state and environmental data to dynamically construct an adjacency matrix reflecting the formation structure, which is then input into the control strategy network in real time. The front end of the UAV control strategy network integrates a graph neural network module, which can effectively fuse the state characteristics of the UAV itself and neighboring formation members, and optimize continuous action decisions based on an improved TD3 algorithm. All UAVs can autonomously adjust their heading, speed, and formation according to changes in the current environment and mission, and can still complete formation cooperation using locally available information even when communication is restricted. During the exercise, by monitoring the UAVs' performance in real time, the system automatically records the coverage efficiency, target recognition rate, mission response latency, and formation integrity at each stage.

[0033] Statistics show that the method of this invention achieves an average coverage rate of 98.7% for the target area by the formation of UAVs, a target recognition accuracy rate of 96.4%, a formation reconstruction time of less than 7 seconds, and a material delivery response delay of less than 5 seconds. Compared with deep learning solutions based on traditional centralized control or unstructured perception, this invention improves the formation maintenance rate by 15.6% in complex environments and the task completion rate by 12.3% in communication interference environments. Furthermore, after adopting this invention, the average energy consumption per UAV is reduced by 9.2%, and the collaborative path conflict rate between UAVs decreases to 3.8%, which is superior to traditional distributed rule algorithms. Specific data statistics for each stage are shown in the table below. This application example fully verifies the efficiency, intelligence, and system robustness of this invention in real-world complex maritime emergency scenarios, providing a solid foundation for subsequent promotion and application.

[0034] Table 1. Comparison of Control Performance of Unmanned Aerial Vehicle (UAV) Maritime Emergency Formation

[0035] As shown in Table 1 above, the UAV maritime emergency formation control method based on deep reinforcement learning proposed in this invention demonstrates advantages over traditional centralized control schemes and unstructured perception deep learning schemes in several key performance indicators. First, in terms of area coverage, the proposed method achieves a high level of 98.7%, higher than the 87.3% of the traditional centralized control scheme and the 92.1% of the unstructured perception deep learning scheme, fully demonstrating that this method can achieve efficient full coverage of complex target waters. Second, in terms of target recognition accuracy, the proposed method reaches 96.4%, also superior to the traditional scheme and the unstructured perception scheme, improving by 11.8 percentage points and 7.1 percentage points respectively, showing the effective integration of structured perception mechanisms and deep reinforcement learning to enhance the intelligent perception capabilities of UAVs.

[0036] This invention also demonstrates significant improvements in two key indicators reflecting emergency response efficiency: average formation reconfiguration time and task response latency. The average formation reconfiguration time is reduced to 6.8 seconds, a reduction of 6.7 seconds compared to traditional centralized control and 4.4 seconds compared to unstructured sensing methods. This reflects that the invention's solution can quickly complete formation reconfiguration under conditions of limited communication or dynamic environmental changes, ensuring the continuity and stability of formation tasks. Simultaneously, the task response latency is only 4.8 seconds, a reduction of 5.3 seconds compared to traditional solutions and 2.7 seconds compared to unstructured sensing solutions, indicating that this method has a faster response capability under sudden events. In terms of formation maintenance rate and task completion rate under communication interference environments, two indicators reflecting system robustness and coordination, this invention achieves 93.2% and 90.5% respectively, representing improvements of 15.6% and 12.3% compared to traditional centralized solutions, significantly enhancing the task execution capability of multi-UAV systems in complex environments.

[0037] Furthermore, this invention has also achieved significant results in energy management and collaborative path conflict control. Average energy consumption was reduced by 9.2%, and the collaborative path conflict rate among UAVs decreased to 3.8%, outperforming the comparative schemes. These results demonstrate that the multi-UAV control method based on structure perception and the improved TD3 algorithm not only enhances the overall system's intelligent coordination and emergency response capabilities but also optimizes energy conservation, consumption reduction, and safety, greatly meeting the high requirements of actual maritime emergency missions for the autonomy, efficiency, and reliability of UAV formations. The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A method for controlling unmanned aerial vehicle (UAV) maritime emergency formations based on deep reinforcement learning, characterized in that, Includes the following steps: Each drone collects its own state information and the state information of neighboring drones to obtain raw state characteristic data; Based on the original state feature data and according to the preset physical distance threshold, a drone formation structure diagram is dynamically constructed, and an adjacency matrix of the formation structure diagram is generated. The original state feature data and the adjacency matrix of the formation structure graph are input into the UAV control strategy network. The front end of the UAV control strategy network integrates a graph neural network module and adopts an improved TD3 algorithm to output continuous action decisions. Each UAV performs autonomous flight control based on continuous action decisions, including formation maintenance, formation reconfiguration, path replanning, and task allocation. In the event of communication restrictions or link interruption, the continuous action decision is adjusted, and each UAV makes autonomous decisions based on the adjusted continuous action decision. During the autonomous flight of the UAV based on continuous action decisions, task execution data is periodically collected, stored in the experience replay pool, and used for batch training of the UAV control strategy network.

2. The method for controlling unmanned aerial vehicle (UAV) maritime emergency formations based on deep reinforcement learning according to claim 1, characterized in that, The original state characteristic data includes the current position coordinates, speed, remaining battery power, and heading angle of each UAV, the current position coordinates, speed, remaining battery power, and heading angle of neighboring UAVs, the ambient wind speed, obstacle positions, and communication signal quality.

3. The method for controlling unmanned aerial vehicle (UAV) maritime emergency formations based on deep reinforcement learning according to claim 1, characterized in that, The step of dynamically constructing a UAV formation structure diagram based on the original state feature data and according to preset dynamic formation adjustment rules, and generating the adjacency matrix of the formation structure diagram, specifically includes: Based on the original state feature data, extract the current position coordinates, communication signal quality, set of neighboring drones, and current position coordinates of each neighboring drone for each drone; For each pair of drones, the spatial distance between them is calculated in real time using the Euclidean spatial distance formula, and the communication signal quality between them at the current moment is obtained. Based on the preset dynamic formation adjustment rules, combined with the spatial distance and communication signal quality between each pair of drones, each pair of drones is judged. If the spatial distance between a pair of drones is less than or equal to the set physical distance threshold, and the communication signal quality between them is higher than or equal to the set signal quality threshold, then it is determined that there is a connection between the two drones; otherwise, no connection is established. Within each refresh cycle, each drone updates the edge determination result synchronously based on its own data and that of neighboring drones, and generates a local substructure edge set in a distributed and dynamic manner. After all local substructure edge sets are aggregated, a global drone formation structure diagram is formed. The results of the edge determination between all pairs of drone nodes are collected to form an adjacency matrix of the formation structure graph.

4. The UAV maritime emergency formation control method based on deep reinforcement learning according to claim 1, characterized in that, The process of inputting the original state feature data and the adjacency matrix of the formation structure diagram into the UAV control strategy network, and using the improved TD3 algorithm to output continuous action decisions specifically includes: The original state feature data and the adjacency matrix of the formation structure graph are used as inputs and synchronously input into the UAV control strategy network. At the front end of the UAV control strategy network, there is a graph neural network module. Each UAV is a node in the graph structure, and the adjacency matrix of the formation structure graph is the connection relationship. The original state feature data of each UAV and the original state feature data of neighboring UAVs are fused to generate the structure perception state feature vector of each UAV. The structural perception state feature vector of each UAV is input into the Actor network and the Critic network respectively to output the UAV's continuous action decision vector, first Q value and second Q value; The UAV control strategy network is trained and updated based on the TD3 reinforcement learning mechanism, and outputs continuous action decisions for each UAV.

5. The UAV maritime emergency formation control method based on deep reinforcement learning according to claim 4, characterized in that, At the front end of the UAV control strategy network, a graph neural network module is provided. Using the adjacency matrix of the formation structure graph as the connection relationship, the generation of structure-aware state feature vectors specifically includes: At the front end of the UAV control strategy network, a feature fusion module containing a multi-layer graph neural network is set up, with each UAV as a node in the graph structure, and the edge relationship between nodes is determined by the adjacency matrix of the formation structure graph. For each layer of the graph neural network, the original state feature data of each drone is used as the initial node feature, and the original state feature data of all neighboring drones that have an edge relationship with the drone are used as the neighbor node feature. At the same time, the information transmission weight between nodes is determined according to the adjacency matrix. In each layer of the graph neural network, the features of each node are iteratively updated by weighted aggregation of the features of neighboring nodes; Through information fusion and feature propagation via a multi-layer graph neural network, the structural perception state feature vector of each UAV is finally obtained.

6. The UAV maritime emergency formation control method based on deep reinforcement learning according to claim 4, characterized in that, The step of inputting the structural perception state feature vector of each UAV into the Actor network and the Critic network respectively, and outputting the UAV's continuous action decision, first value function and second value function specifically includes: The structural perception state feature vector of each UAV is input into the Actor network and Critic network in the UAV control policy network; The Actor network takes the structure-aware state feature vector as input and passes it through several layers of fully connected neural networks and activation functions to generate continuous action decision vectors. The Critic network takes the concatenation result of the structure-aware state feature vector and the continuous action decision vector as input. The Critic network includes two sets of value function sub-networks with the same structure but independent parameters. They are passed through several layers of fully connected neural networks and activation functions in sequence to output the first Q value and the second Q value, respectively. Both the Actor network and the Critic network employ a multi-layer fully connected structure adapted to the dimensions of the structure-aware state feature vector and the continuous action decision dimension.

7. The UAV maritime emergency formation control method based on deep reinforcement learning according to claim 4, characterized in that, The unmanned aerial vehicle (UAV) control strategy network is trained and updated based on the TD3 reinforcement learning mechanism, specifically including: The UAV control strategy network is trained based on the improved TD3 algorithm. During the training process, the experience replay pool is first initialized, and the sampling batch size, learning rate, discount factor, and noise perturbation coefficient are set as training parameters. In each training iteration, a batch of samples are randomly sampled from the experience replay pool. Each sample includes the current structure-aware state feature vector, the current continuous action decision, the immediate reward, the new structure-aware state feature vector, and the task termination marker. The target Actor network is used to calculate the next continuous action decision based on the state feature vector of each new structure. The first Q-value and the second Q-value are calculated using the target Critic network, and the smaller of the two values ​​is taken as the target Q-value. The mean square loss is calculated based on the target Q value and the first and second value functions output by the current Critic network. After every few updates of the Critic network parameters, the Actor network takes the structure-aware state feature vector as input and optimizes by maximizing the first Q value of the Critic network output. The parameters of the target Actor network and the target Critic network are iterated using a soft update method, and the adjacency matrix and structure-aware state features are dynamically updated in each iteration.

8. The method for controlling unmanned aerial vehicle (UAV) maritime emergency formations based on deep reinforcement learning according to claim 1, characterized in that, Each UAV performs autonomous flight control based on continuous action decisions, including formation maintenance, formation reconfiguration, path replanning, and task allocation. Specifically, this includes: Each UAV periodically acquires its own structure perception state feature vector and inputs it into the current Actor network to obtain the UAV's continuous action decisions, which include flight heading adjustment, speed control, and formation relative position transformation. Based on the flight heading adjustment made by continuous action decisions, the UAV flight control system adjusts the flight heading angle in real time. Speed ​​control based on continuous action decisions is achieved by dynamically adjusting the flight speed of the UAV flight control system. Based on the changes in the relative positions of the formation according to the continuous action decision, the UAV flight control system adjusts the relative positions of the formation. During flight control, the connectivity and coordination of the overall formation are monitored in real time, and path replanning and task reallocation are dynamically triggered based on changes in UAV formation, environmental dynamics and mission requirements.

9. The method for controlling unmanned aerial vehicle (UAV) maritime emergency formations based on deep reinforcement learning according to claim 1, characterized in that, The process of adjusting continuous action decisions in the event of communication constraints or link interruption, and enabling each UAV to make autonomous decisions based on the adjusted continuous action decisions, specifically includes: In the event of communication restrictions or link interruption, each UAV periodically detects the communication status and signal quality between itself and neighboring UAVs, and dynamically updates the adjacency matrix of the formation structure graph based on the detection results, retaining only the edge relationships that currently meet the physical distance threshold and signal quality threshold conditions. Based on the adjacency matrix of the adjusted formation structure graph, each UAV redetermines its node connection relationship in the adjusted formation structure graph and updates its own structure perception state feature vector. The updated structure-aware state feature vector is input into the current Actor network to obtain the adjusted continuous action decision. Each UAV independently executes flight control based on the adjusted continuous action decision.

10. The UAV maritime emergency formation control method based on deep reinforcement learning according to claim 1, characterized in that, During the autonomous flight of the UAV based on continuous action decisions, task execution data is periodically collected, stored in an experience replay pool, and used for batch training of the UAV control strategy network. Specifically, this includes: During the process of autonomous flight and cooperative control executed by UAVs based on continuous action decisions, each UAV periodically collects task execution data; The collected task execution data is stored in the experience replay pool in chronological order. Periodically or on-demand random sampling from the experience replay pool, with a batch size of [missing information]. Task execution data, The set sampling batch size; Using the batch task execution data obtained from sampling, the target Q-value is calculated, the Critic network loss function is calculated, the Actor network loss function is calculated, and the network parameters are updated in sequence.

Images