Multi-auv cooperative target pursuit method based on relationship abstraction mechanism

Abstract

The application belongs to the technical field of underwater autonomous vehicle control, and proposes a multi-underwater autonomous vehicle cooperative target pursuit method based on a relationship abstraction mechanism, which guides AUV search in the target detection stage, assists strategy planning in the pursuit stage, optimizes the cooperative action among multiple AUVs by introducing a relationship abstraction mechanism, captures key interaction information among AUVs by using hard attention and soft attention mechanisms, and adjusts the action decision of each AUV in the pursuit process according to the position, state and behavior information of other AUVs. A critic network is constructed to optimize the action network under the distributed training and centralized execution framework, complete the strategy and control in the multi-AUV cooperation task, and realize efficient multi-AUV cooperative pursuit and optimized training convergence performance.

Description

Technical Field

[0001] This invention belongs to the field of underwater autonomous vehicle control technology. Background Technology

[0002] With the increasing demands for marine resource development and maritime security, autonomous underwater vehicles (AUVs) are playing a vital role in marine exploration, target tracking, and underwater operations. Multi-AUV cooperative systems can accomplish complex tasks that are difficult for a single AUV to perform through distributed collaboration, demonstrating significant advantages, particularly in strategic tasks such as underwater target pursuit, containment, and attack.

[0003] Traditional multi-AUV cooperative target pursuit methods are mainly based on heuristic algorithms and deterministic pursuit strategies, which require the pre-establishment of accurate environmental and target motion models. However, the complexity and uncertainty of the underwater environment make accurate modeling difficult. In recent years, deep reinforcement learning technology has provided new ideas for solving multi-agent cooperative decision-making problems, but existing technologies still have the following technical problems: Existing algorithms usually set the motion state of the escaping target as a deterministic strategy, i.e., a non-intelligent escaping target, which is inaccurate in actual pursuit tasks, as real escaping targets often possess intelligent decision-making capabilities; secondly, the agent scalability of heuristic algorithms and deterministic pursuit strategies is weak, making it difficult to adapt to the needs of large-scale multi-AUV cooperative operations; most existing centralized training decentralized execution (CTDE) methods, when the AUV observation space is large and the cluster size is large, will cause an explosion of input dimensions in the critic network, resulting in slow or non-convergent convergence of the policy network.

[0004] In summary, most existing technologies are applicable to two-dimensional environments and ignore the ability of escape targets to maneuver at varying depths. This greatly simplifies the complexity of the environment and results in weak scalability of the intelligent agent, failing to fully utilize the motion performance and maneuverability advantages of AUVs in three-dimensional underwater space. Summary of the Invention

[0005] To address the technical challenges in existing technologies, such as difficulties in environmental modeling, simplification of target motion strategies, weak agent scalability, the curse of dimensionality, and limited motion performance, and to achieve efficient multi-AUV cooperative pursuit and optimized training convergence performance, a game-theoretic method for multi-AUV cooperative target pursuit using a relational abstraction mechanism is proposed. The specific technical solution is as follows: Construct a target trajectory prediction network (ETPNet) to predict the location of escaping targets and guide AUVs in search and pursuit; The predicted features output by the target trajectory prediction network are input into the action network and concatenated with local observation information to form a joint input vector. The depth spatiotemporal features related to the pursuit task are extracted and processed to output three parameters: the thrust generated by the thruster, the horizontal rudder angle of the AUV rudder, and the vertical rudder angle, in order to drive the AUV. A critic network is constructed to optimize the action network within a distributed training and centralized execution framework, enabling policy and control in multi-AUV collaborative tasks. Iterative training involves cyclical interaction and collection of sample experience. The target trajectory prediction network, action network, and commentator network are periodically updated until the maximum number of training steps is reached, at which point the final collaborative target tracking model is output.

[0006] Technical effects: This invention addresses the problems of dimensionality disaster and adaptability to complex underwater environments in multi-AUV collaboration. It optimizes the collaborative actions among multiple AUVs by introducing a relationship abstraction mechanism, and uses hard attention and soft attention mechanisms to capture key interaction information between AUVs. By dynamically modeling the spatial positional relationships between AUVs, each AUV adjusts its action decisions in real time based on the target's position, status, and the behavior information of other AUVs during the pursuit process, thereby improving the collaborative efficiency of the multi-AUV system.

[0007] A target trajectory prediction network is designed, utilizing a Long Short-Term Memory (LSTM) network to learn historical target trajectories. Based on LSTM processing of temporal correlation features, it guides AUV search during the target detection phase and assists in policy planning during the pursuit phase. The prediction results guide the AUV in target search. When a target is detected, ETPNet is used to predict its future trajectory to optimize the AUV's pursuit path. A multi-agent reinforcement learning algorithm is employed to iteratively train parameters to optimize the commentator network. A dual-agent network is used. Q Network structure to avoid Q The estimate is too high. The collaborative strategy is optimized within a centralized training and decentralized execution framework; the task difficulty is gradually increased using curriculum learning techniques to improve training stability. This invention effectively solves the problem of collaborative pursuit among multiple AUVs in partially observable dynamic underwater environments, exhibiting good robustness and adaptability. Attached Figure Description

[0008] Figure 1 This is an overall flowchart of an embodiment of the present invention.

[0009] Figure 2 The diagram shows the internal structure of the target trajectory prediction network and the action network.

[0010] Figure 3 A diagram illustrating the internal structure of the relational abstraction mechanism in a critic network.

[0011] Figure 4 This is a framework diagram of the multi-AUV underwater target pursuit cyclic training process in an embodiment of the present invention. Detailed Implementation

[0012] The reinforcement learning training environment is initialized with the following parameters: task completion radius is 5 meters, minimum safe distance is 4 meters, maximum detection radius is 50 meters, total training steps are 1 million steps, experience memory capacity is 800,000 steps, number of parallel environments is 8, discount factor is 0.99, actor network learning rate is 0.001, critic network learning rate is 0.001, temperature coefficient is 0.01, gradient cutoff is 0.5, soft update rate is 0.001, training frequency is once every 20 steps, and the learning rate of the ETPNet network is 0.0005, with a training frequency of once every 50 steps. These parameters are for illustrative purposes only and are not limited to these specific parameters.

[0013] A target trajectory prediction network (ETPNet) is constructed to predict the location of escape targets by learning from their historical information, thereby guiding AUVs in search and pursuit.

[0014] Furthermore, such as Figure 2 As shown in the figure express t The position coordinates of all AUVs at any given time. This indicates that ETPNet is in t To predict the location of evasive AUVs in real time, the target trajectory prediction network ETPNet employs a Long Short-Term Memory (LSTM) architecture with 128 hidden layers and a sequence length of 20 time steps. By learning features from the location coordinates of the evasive AUV (EAUV) over the past 20 frames, it outputs the predicted trajectory for the next 5 steps. This prediction logic predicts the potential area of ​​the target based on historical patterns when it is not detected, guiding the efficient search of pursuit-oriented AUVs (PAUVs). After the target is captured, the real-time updated predicted trajectory provides a forward-looking decision-making basis for PAUV collaborative interception and pursuit path optimization.

[0015] In the action network, the predicted features output by the target trajectory prediction network With local observation information The input vector is concatenated to form a joint input vector. This vector is mapped through two layers of a multilayer perceptron (MLP) to extract the depth and spatiotemporal features relevant to the pursuit task and output the Gaussian distribution parameters corresponding to the control actions. This Gaussian distribution is sampled and processed by a Tanh function to limit the output value to the range of 0 to 1, thus outputting the thrust generated by the thruster. T、 Horizontal rudder angle of AUV rudder and vertical rudder angle The above three parameters directly drive the movement and navigation of the AUV.

[0016] Furthermore, the local observation information This includes the position, speed, and orientation of each AUV, as well as environmental information detected by the sensors.

[0017] A critic network is constructed to optimize the action network within a distributed training and centralized execution framework, outputting better three parameters to achieve policy and control in multi-AUV cooperative tasks. A relational abstraction mechanism is introduced into the critic network to reduce the dimensionality of the numerical vector formed by concatenating local observation information from multiple AUVs. This mechanism includes hard attention and soft attention networks, respectively used to capture spatial interaction information between AUVs and to weightedly integrate the contributions of each AUV to other agents. By dynamically modeling the spatial positional relationships between AUVs, each AUV adjusts its action decisions in real time based on the target's position, state, and the behavior information of other AUVs during pursuit, thereby outputting better thrust. T、 and vertical rudder angle This improves the collaborative efficiency of multiple AUV systems.

[0018] like Figure 3 As shown, joint local observation information of multiple AUV clusters First, the input is fed into a multilayer perceptron (MLP) to extract encoded features. The encoded features are input into a hard attention network and a soft attention network, respectively, and output AUVs. i With AUV j The corresponding hard attention weights and soft attention weights .

[0019] Furthermore, the hard attention weights are represented using a one-hot encoding format, which encodes the features. and Rows are concatenated to form a single feature vector, which is then used as input to a gated recurrent unit (GRU) module. ,in, This represents a fully connected layer FCN, which uses bidirectional gated recurrent units (Bi-GRU) to process the observation data of the historical sliding window to extract temporally dependent features. By aggregating forward and backward context information through a bidirectional processing mechanism, the model's sensitivity to the order of the input sequence is effectively reduced, thereby ensuring the robustness of the feature representation.

[0020] In calculating AUV i With AUV j When considering the interaction relationship weights between them, AUV should be taken into account comprehensively. kThe correlation effect. To address the non-differentiability issue in the sampling process and achieve end-to-end backpropagation, the Gumbel-Softmax function is introduced to approximate the discrete distribution as continuous. Based on this, the calculation process for the hard attention weight coefficients is as follows: , Based on the aforementioned hard attention network, the interaction subgraph corresponding to each AUV can be calculated. Accordingly, AUV... i Only need to integrate AUVs that have an interaction with it. j The observation characteristics. Subsequently, regarding AUV... i Each AUV with interaction j The corresponding weight coefficients are calculated using a soft attention network: , in, and The AUV is determined by the element-wise product of two learnable parameters. i For AUV j Assigned feature fusion weights.

[0021] By calculating the hard attention weight coefficient With soft attention weight coefficient The element-wise product (matrix multiplication) determines the AUV. i Assigned to AUV j The feature fusion weights are then used to obtain the fused cooperator observation feature terms. .

[0022] Preferably, AUV is used for non-cooperating entities. i To address the training instability issue caused by concatenating zero vectors during feature fusion, a residual network is introduced to compute AUV separately. i Self-characteristics With collaborative items The self-attention weights are controlled by a learnable parameter gate, which regulates the weight coefficients of the self-features and the co-occurrence terms. Finally, this two-layer attention residual network outputs AUV (Active UV). i The comprehensive integration characteristics represent And input it into a multilayer perceptron (MLP), outputting... The Used in the loss function to update the critic network.

[0023] In one possible implementation, the loss function is: , In the formula, for Figure 4 The Q-value of the target network, expresst Joint local observation information of all AUVs at any given time. express t The coordinated action of all AUVs at all times. The definition is as follows: , In the formula, Indicates that based on AUV in t Local observation status at time +1 Below, the current strategy The output action value.

[0024] Iterative training involves iterative interaction and collection of sample experience, periodically updating the target trajectory prediction network, action network, and commentator network until the maximum number of training steps is reached, at which point the final model is output.

[0025] In one possible implementation, a multi-agent reinforcement learning algorithm, RA-MASAC, based on Multi-Agent SoftActor-Critic (MASAC), is established to iteratively train parameters to optimize the critic network. This is achieved using a dual-agent... Q The network structure includes two independent Q network Q 1 and Q 2. Avoid by taking the minimum of the two. Q The problem of overestimating values.

[0026] In one possible implementation, the RA-MASAC algorithm is trained using a centralized training and decentralized execution (CTDE) framework, which ensures that AUV systems can collaborate efficiently in large-scale environments by sharing observation information and implementing independent strategies.

[0027] In one possible implementation, efficient exploration of the AUV system is ensured by maximizing the policy entropy, with the objective function being a weighted sum of expected reward and policy entropy.

[0028] In one possible implementation, by temperature parameter The balance between exploration and exploitation is adjusted, with an initial value set to 0.2 and dynamically updated through an adaptive adjustment mechanism to prevent the strategy from getting stuck in local optima and ensure that the optimal pursuit strategy is learned in a dynamic underwater environment.

[0029] like Figure 4As shown, during the training process, a learning mechanism is first set up for PAUV and EAUV respectively, and the attack range is gradually adjusted according to the current training level to change the training difficulty of the pursuit mission. Subsequently, PAUV and EAUV generate control actions through their respective action networks based on their observation states and apply them to the pursuit game environment. The environment updates the system state according to the actions executed by each AUV, and returns interactive information such as the next time step state, reward, and mission termination flag. The state transition samples generated by PAUV and EAUV are stored in the PAUV experience memory and EAUV experience memory respectively.

[0030] During training, a certain batch of interaction samples is randomly drawn from the corresponding experience memory to update the respective action network and critic network. The critic network calculates value estimates and corresponding gradients based on the sampled data, and uses the gradient information to optimize the action network parameters, thereby improving the policy's decision-making ability in the current pursuit / escape state. Through the aforementioned cyclical training process of "course difficulty adjustment—policy action generation—environmental interaction—experience storage—sample sampling—network parameter update," the PAUV and EAUV policies are continuously optimized under progressively increasing pursuit / escape task difficulty, ultimately yielding cooperative pursuit strategies and autonomous escape strategies suitable for multi-AUV pursuit / escape game tasks.

[0031] Furthermore, both PAUV and EAUV's action network and commentator network include a main network and a target network. The main network learns parameters based on sampled data, while the target network achieves periodic iteration through soft updates of the main network's parameters, thereby improving the stability of the training process.

[0032] Furthermore, the course implementation is divided into four stages, gradually increasing the difficulty of the tasks. The first stage involves training in a simple environment with no obstacles and no ocean currents, with the initial distance between the AUV and the target set at 100 meters. The second stage introduces static obstacles, with an obstacle density of 10% of the environmental area. The third stage incorporates the influence of ocean currents, with current speeds set at 0.5~1.5 m / s and directions varying randomly. The fourth stage presents a complete and complex environment, including dynamic obstacles and time-varying ocean currents, and gradually reduces the AUV's pursuit range from 100 meters to 50 meters, increasing the difficulty of the pursuit task and enhancing the stability and generalization ability of the strategy.

[0033] In partially observable underwater environments, the observation range of each AUV is limited to a spherical region of radius R. Dynamic obstacles move with random speeds and directions, requiring the AUVs to continuously optimize their cooperative strategies through reinforcement learning to adapt to environmental changes. The system exhibits strong robustness, capable of handling practical issues such as sensor noise and communication latency, and demonstrates good adaptability in complex three-dimensional underwater environments.

[0034] In a preferred embodiment, ETPNet is trained using the mean squared error loss function, with a learning rate of 0.001, a batch size of 64, and a training cycle of 1000 rounds.

[0035] In a preferred embodiment, the attention weights in the relation abstraction mechanism are calculated using a multi-head attention mechanism with 8 heads and 64 dimensions for each head.

[0036] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A cooperative target acquisition method for multiple underwater autonomous vehicles based on a relational abstraction mechanism, characterized in that, Construct a target trajectory prediction network to predict the location of escaping targets and guide AUVs in search and pursuit; The predicted features output by the target trajectory prediction network are input into the action network and concatenated with local observation information to form a joint input vector. The depth spatiotemporal features related to the pursuit task are extracted and processed to output three parameters: the thrust generated by the thruster, the horizontal rudder angle of the AUV rudder, and the vertical rudder angle, in order to drive the AUV. A critic network is constructed to optimize the action network within a distributed training and centralized execution framework, enabling policy and control in multi-AUV collaborative tasks. Iterative training involves cyclical interaction and collection of sample experience. The target trajectory prediction network, action network, and commentator network are periodically updated until the maximum number of training steps is reached, at which point the final collaborative target tracking model is output.

2. The multi-underwater autonomous vehicle cooperative target acquisition method based on relational abstraction mechanism according to claim 1, characterized in that, The target trajectory prediction network adopts a Long Short-Term Memory (LSTM) architecture with a hidden layer dimension of 128 and a sequence length of 20 time steps. By learning the features of the position coordinates of the evasive AUV in the past 20 frames, it outputs the predicted trajectory of the next 5 steps to guide the pursuit AUV in the search.

3. The multi-underwater autonomous vehicle cooperative target acquisition method based on relational abstraction mechanism according to claim 1, characterized in that, The joint input vector in the action network is mapped through two layers of multilayer perceptron (MLP) to extract the depth spatiotemporal features related to the pursuit task and output the Gaussian distribution parameters of the corresponding control actions. The parameters are sampled and processed by the Tanh function to limit the output value to the range of 0 to 1.

4. The multi-underwater autonomous vehicle cooperative target acquisition method based on relational abstraction mechanism according to claim 1, characterized in that, A relational abstraction mechanism is introduced into the commentator network to reduce the dimensionality of the numerical vector formed by splicing local observation information from multiple AUVs. The relational abstraction mechanism includes hard attention and soft attention networks, which are used to capture key interaction information between AUVs. By dynamically modeling the spatial positional relationship between AUVs, each AUV adjusts its action decision in real time based on the target's position, state, and the behavior information of other AUVs during the pursuit process.

5. The multi-underwater autonomous vehicle cooperative target acquisition method based on relational abstraction mechanism according to claim 4, characterized in that, Joint local observation information of multiple AUV clusters First, the input is fed into a multilayer perceptron (MLP) to extract encoded features. The encoded features are input into a hard attention network and a soft attention network, respectively, and output AUVs. i With AUV j The corresponding hard attention weights and soft attention weights Hard attention weights Encode features and Row concatenation is performed to form a single feature vector, which is then used as the input to the gated recurrent unit module: ,in, This indicates a fully connected layer FCN; Bi-GRU is a bidirectional gated recurrent unit. For AUV i Each AUV with interaction j The corresponding weight coefficients are calculated using a soft attention network: ， in, and The AUV is determined by the element-wise product of two learnable parameters. i For AUV j Assigned feature fusion weights; Hard attention weights and soft attention weights The matrix multiplication operation is passed through a two-layer attention residual network and a multilayer perceptron (MLP) to output parameters for the loss function, thereby updating the critic network.

6. The multi-underwater autonomous vehicle cooperative target acquisition method based on relational abstraction mechanism according to claim 1, characterized in that, The local observation information includes each AUV's own position, velocity, azimuth angle, and environmental information detected by the sensors.

7. The multi-underwater autonomous vehicle cooperative target acquisition method based on relational abstraction mechanism according to claim 1, characterized in that, A multi-agent reinforcement learning algorithm RA-MASAC based on a multi-agent soft actor-critic is established to iteratively train parameters to make the critic network optimal.

8. The multi-underwater autonomous vehicle cooperative target acquisition method based on relational abstraction mechanism according to claim 1, characterized in that, During training, a learning mechanism is first set up for both pursuit-type and evasion-type AUVs, and the attack range is gradually adjusted according to the current training level to change the training difficulty of the pursuit mission. Then, pursuit-type and evasion-type AUVs generate control actions through corresponding action networks based on their respective observation states and apply them to the pursuit game environment. The pursuit game environment updates the system state according to the actions performed by each AUV, returns the interaction information for the next moment, and stores the state transition samples generated by pursuit-type and evasion-type AUVs into their respective experience memories.

Images