Distributed unmanned ship formation control method based on attention mechanism SAC, electronic equipment and storage medium

By adopting a distributed unmanned surface vessel (USV) formation control method based on attention mechanism SAC, the problems of insufficient dynamic performance and inter-vessel collision in USV formation control are solved, and autonomous navigation and stable formation in unknown waters are realized.

CN119045484APending Publication Date: 2024-11-29SHANGHAI UNIV +1
View PDF 0 Cites 4 Cited by

Patent Information

Application Number
CN202411136384.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-19
Publication Date
2024-11-29

AI Technical Summary

Technical Problem

In existing unmanned surface vessel (USV) formation control technologies, underactuated systems result in insufficient dynamic performance, are susceptible to sea conditions, have complex formation control, and are prone to inter-vessel collisions and position drift, making it difficult to achieve autonomous navigation and close formation in unknown waters.

Method used

A distributed unmanned surface vessel (USV) formation control method based on attention mechanism SAC is adopted. By forming a chain formation structure through isomorphic intelligent agent USVs, an attention mechanism is introduced to process neighborhood information, and Markov decision process and neighborhood information encoding module are designed to achieve stable formation control.

Benefits of technology

It reduces the communication burden on the navigator, achieves a stable formation structure, can navigate autonomously in unknown waters, avoids collisions between vessels, and possesses dynamic navigation strategies and close formation capabilities.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119045484A_ABST
    Figure CN119045484A_ABST
Patent Text Reader

Abstract

The invention belongs to the technical field of unmanned ship formation control, and particularly discloses a distributed unmanned ship formation control method based on an attention mechanism SAC, electronic equipment and a storage medium, and the method comprises the steps: S1, employing isomorphic agent unmanned ships to build an unmanned ship formation; s2, constructing the unmanned ship formation into an unmanned ship chained formation structure based on a navigator-follower framework; s3, introducing an attention mechanism into the unmanned ship chain formation structure to help the unmanned ship chain formation structure to process neighborhood information so as to realize safe and stable formation control; s4, designing a corresponding Markov decision process for each follower unmanned ship in the unmanned ship chain formation structure; s5, constructing a neighborhood information coding module and a decision network based on an attention mechanism; and S6, training the unmanned ship chained formation structure in a simulation environment.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of unmanned boat formation control, and specifically relates to a distributed unmanned boat formation control method, electronic equipment and storage medium based on an attention mechanism SAC. Background Art

[0002] Unmanned surface vessels (UAVs) play an important role in maritime escort, maritime search and rescue, patrol and surveillance, environmental monitoring, minesweeping and anti-submarine missions, etc., due to their shallow draft, low cost, high flexibility, modular assembly, and the ability to replace humans in high-risk environments. Cruising and exploration in unknown waters is the most common application of UAV formations. The core technical difficulty lies in realizing the autonomous navigation planning of UAVs and the tight control of the formation. However, most of the current UAV systems are under-actuated systems. The so-called under-actuation means that the dimension of the actual control input of the surface vessel is smaller than the dimension of the boat's motion space, which results in insufficient dynamic performance of the UAV. It is necessary to achieve operations such as heading keeping and path tracking through coupled motion, and it is easily affected by various sea conditions, which greatly increases the difficulty of avoiding dynamic obstacles.

[0003] In addition, due to the close formation and small distance between boats, the unmanned boats have to face coupling interference from the surrounding environment and adjacent boats, which can easily induce the boat's position drift and collision between boats, threatening navigation safety and increasing the complexity of formation control. Summary of the invention

[0004] Based on the above problems, the present invention provides a distributed unmanned boat formation control method, electronic device and storage medium based on the attention mechanism SAC that can autonomously navigate in unknown waters.

[0005] Based on the above purpose, the present invention is achieved through the following technical solutions:

[0006] The first aspect of the present invention provides a distributed unmanned boat formation control method based on the attention mechanism SAC, comprising the following steps:

[0007] S1. Use homogeneous intelligent agent unmanned boats to form an unmanned boat fleet; all unmanned boats in the homogeneous intelligent agent unmanned boats are of the same type, and all unmanned boats have the same state space S and action space and the same reward function r.

[0008] S2. Construct the unmanned boat formation into an unmanned boat chain formation structure based on a leader-follower framework; the unmanned boats in the unmanned boat chain formation structure are divided into follower unmanned boats and leader unmanned boats.

[0009] S3. An attention mechanism is introduced into the unmanned boat chain formation structure to help the unmanned boat chain formation structure process neighborhood information to achieve safe and stable formation control.

[0010] S4. Design a corresponding Markov decision process for each follower unmanned boat in the unmanned boat chain formation structure. S5. Construct a neighborhood information encoding module and decision network based on the attention mechanism. The neighborhood information encoding module is used to effectively encode the collective influence of the unmanned boats in the neighborhood and accurately calculate the importance of each neighborhood unmanned boat relative to the currently controlled unmanned boat. With the help of the neighborhood information encoding module, the decision network can analyze and integrate neighborhood information more deeply, and evaluate the impact of the relative state of the controlled unmanned boat and its neighborhood on the formation control, and formulate an efficient and accurate formation control strategy.

[0011] S6. Train the unmanned boat chain formation structure in a simulation environment.

[0012] According to the above-mentioned distributed unmanned boat formation control method based on the attention mechanism SAC, preferably, in step S2, the unmanned boat chain formation structure is a leader-follower model with a chain structure, specifically: each follower unmanned boat only needs to pay attention to and track its direct preceding leader unmanned boat, rather than all follower unmanned boats jointly tracking the same leader unmanned boat, thereby reducing the communication pressure of the formation leader unmanned boat, and achieving a stable formation structure by minimizing the tracking error of each unmanned boat; the specific steps of constructing the unmanned boat chain formation structure are:

[0013] First, a local coordinate system OX based on the bow direction of the pilot unmanned boat is constructed. B Y B ; The local coordinate system OX B Y B With the world coordinate system OX W Y W share the same origin, and Y B The direction of the axis is consistent with the heading of the direct leader unmanned boat of each follower unmanned boat, and the local coordinate system OX B Y B With the world coordinate system OX W Y W Coordinate transformation relationship:

[0014]

[0015] in, W x i , W y i They represent the follower unmanned boat U i In the world coordinate system OX W Y W The coordinate position of B x i , B yi They represent the follower unmanned boat U i In the local coordinate system OX B Y B The new coordinates in .

[0016] In the local coordinate system OX B Y B The following represents the velocity vector [u i ,v i ]:

[0017]

[0018] in, W u i and W v i They represent the unmanned boat along the world coordinate system X W Axis and Y W Speed ​​in the axis direction; and They represent the unmanned boat along the local coordinate system X B Axis and Y B The speed in the axis direction.

[0019] The coordinate relationship of all follower unmanned boats is converted to each pair of specific local coordinate systems, and the originally complex unmanned boat formation control problem is converted into the problem of each follower maintaining the position relationship between it and its direct leader; using the queue matrix F s Describe the relationship between formations:

[0020]

[0021] Among them, the first line indicates the number of each unmanned boat in the formation; and Represents the local coordinate system of each follower unmanned boat and its direct leader The horizontal and vertical distance offsets required to be maintained; IL(U i ) indicates that the follower unmanned boat U i The number of the direct pilot.

[0022] According to the queue matrix F s Configure the formation, with the central unmanned boat as the leader of the cluster, and set the distance offset between the follower unmanned boat and its direct leader The follower unmanned boats can form a chain tracking one by one, and a tight formation structure can be achieved;

[0023] Based on the above chain leader-follower structure, the control target is:

[0024]

[0025] Among them, U follower represents the set of follower unmanned boats; H i Indicates the preset position of the follower unmanned boat in the formation; H′ i Indicates the actual position of the follower in the formation, which is the actual coordinate position p′ of the follower unmanned boat i Coordinate position with the pilot Definition of deviation; Indicates follower unmanned boat U i The coordinate position of the direct pilot unmanned boat.

[0026] According to the above-mentioned distributed unmanned boat formation control method based on the attention mechanism SAC, preferably, in step S3, the attention mechanism first adopts the attention mechanism to construct a key-value pair attention model:

[0027] (K,V)=[(k1,v1),(k2,v2),…,(k n ,v n )];

[0028] Among them, K represents the key, which is used to calculate the attention distribution; V represents the value, which is used to calculate the final aggregate information based on the attention distribution.

[0029] The final aggregate information is expressed as:

[0030]

[0031] The calculation of the key-value pair attention model is divided into three stages: using the scoring function to calculate the similarity between the query vector q and the current input key K (key); normalizing different scores through the softmax function to obtain the attention weight of each part; and performing information aggregation based on the attention distribution and the corresponding value.

[0032] The attention mechanism then uses the key-value pair attention model to build a multi-head attention model to improve the expression and generalization capabilities of the attention mechanism. Specifically, through multiple query vectors Q = [q1, q2, ..., q m ] in parallel from the input information set (K, V) = [(k1, v1), (k2, v2), …, (k n ,v n )] to retrieve multiple sets of related information; during the query process, each query vector q iIt will focus on different parts of the input information, and can deeply analyze and understand the current input information from different dimensions or perspectives, so as to learn more comprehensive feature information; finally, the final output is obtained by splicing the results of all query vectors:

[0033]

[0034] Among them, the symbol Represents a vector concatenation operation.

[0035] According to the above-mentioned distributed unmanned boat formation control method based on the attention mechanism SAC, it is preferably characterized in that the unmanned boat chain formation structure adopts a decentralized training strategy, each unmanned boat in the formation is an independent unit with the ability to communicate with each other, and is equipped with a flexible actor-critic network based on the attention mechanism, each unmanned boat takes action independently based on its own observation information and the communication information of the neighborhood, and each unmanned boat learns to update its own value network and strategy network by interacting with the environment together, and finally learns an effective formation control strategy; the unmanned boats in the unmanned boat chain formation structure are WAM-V unmanned boats, and they all perform the same task, and the follower unmanned boats in the formation share the same observation, action space and reward function.

[0036] According to the above-mentioned distributed unmanned boat formation control method based on the attention mechanism SAC, preferably, in step S4, in the unmanned boat chain formation structure, the Markov decision process of the follower unmanned boat includes: S401, obtaining the follower unmanned boat U i The complete observation space of the follower unmanned boat U i S403, obtain the follower unmanned boat U i The complete reward function.

[0037] According to the above-mentioned distributed unmanned boat formation control method based on the attention mechanism SAC, preferably, in step S401,

[0038] By Followers Unmanned Boat U i Direct pilot unmanned boat Position in local coordinate system With the queue matrix F s , get the follower unmanned boat U i Target position in formation Further obtain the target location With the follower unmanned boat U i The relative position deviation It can help the follower unmanned boat to clearly identify its tracking target.

[0039] Keep following UAV iThe stable relative position relationship between the follower and its direct leader can realize the rapid formation and long-term maintenance of the formation, making it easier for the follower to accurately control its own catching-up speed and obtain the follower unmanned boat U i The task observation space is:

[0040]

[0041] in, Indicates follower unmanned boat U i The speed relationship with its direct leader.

[0042] Measurement Follower Unmanned Boat U i The motion state in the observation space is expressed as the coordinate position in the local coordinate system ( B x i , B y i ),speed( B u i , B v i ) and heading angle B ψ i Add to follower UAV i In the observation space, it can effectively manipulate and avoid collisions:

[0043]

[0044] Collect follower unmanned boat U i Neighborhood status information As its neighborhood observation space:

[0045]

[0046] in, and For the follower unmanned boat U i The relative position and relative speed of the unmanned boats in the neighborhood; K≤N is the number of neighbors that each unmanned boat can observe.

[0047] The environmental observation information is added to the observation space of the follower unmanned boat. Specifically, the external environmental data is collected by using the lidar, and the redundant information is filtered using the minimum pooling method; and the point cloud dataset is Convert to local coordinate system Next relative to the accused follower unmanned boat U i The coordinate position of the follower unmanned boat enables it to sense external environmental obstacles:

[0048]

[0049] Thus, the follower unmanned boat U iEnvironmental observation space:

[0050]

[0051] Finally, the follower unmanned boat U i The complete observation space of:

[0052]

[0053] In step S402,

[0054] The observed state of the follower unmanned boat is used to obtain the distribution of the actual control actions of the follower unmanned boat:

[0055]

[0056] Among them, n iτ 、n iw Follower unmanned boat U i The propulsion command and rudder angle steering command of the thruster; considering that when the formation has not yet been formed or is in an unstable state, the follower unmanned boat needs to be able to quickly adjust its speed state to catch up with the leader, the follower unmanned boat has a wider propulsion command control range, that is, [-1.0, 1.0], to ensure that the follower unmanned boat can have a faster speed and stronger braking ability. At the same time, a larger rudder angle steering range is set, that is, [-25°, 25°], so that the follower can have a smaller turning radius.

[0057] According to the above-mentioned distributed unmanned boat formation control method based on the attention mechanism SAC, preferably, in step S403,

[0058] Each follower unmanned boat in the formation maintains a preset offset distance with its direct leader in each time step, that is, the follower unmanned boat should minimize the distance from the desired target position. The relative position deviation Reduce the difference between the expected formation and the expected formation; quantify the effect of the formation maintenance strategy and introduce the follower unmanned boat U i Location Constraint Rewards

[0059]

[0060] In the process of formation control, in order to avoid collisions between unmanned boats and between unmanned boats and obstacles, the system should give certain penalties to unmanned boats that collide in the formation during the formation coordinated cruising process; in order to quantify this penalty and guide the unmanned boats to avoid collisions, a sparse collision penalty term for the unmanned boats that collide is constructed.

[0061]

[0062] Among them, d s can represents the distance information detected by the laser radar, D collision Indicates the distance threshold at which a collision occurs.

[0063] In order to accelerate the learning of the avoidance strategy of the unmanned boat and take avoidance actions in advance before a potential collision occurs, an additional reward based on the safe neighborhood is constructed. Specifically, the follower unmanned boat needs to perceive all the information of the unmanned boats or obstacles in its neighborhood, and its detection range covers all directions of the unmanned boat; at the same time, in order to give the follower unmanned boat a certain motion offset, the warning range D′ of the follower unmanned boat safe Set the distance D slightly smaller than the ideal formation distance of the unmanned boat formation When other unmanned boats or obstacles enter the warning area, the accused unmanned boat will be given an additional penalty. Used to warn the unmanned boat to make evasive action in advance; additional penalty defined for:

[0064]

[0065] Among them, D′ safe Indicates the warning range of the follower unmanned boat; D formation Indicates the ideal formation distance of unmanned boats.

[0066] Based on the above rewards, get the follower unmanned boat U i The complete reward function is:

[0067]

[0068] Among them, λ p and λ s are constant coefficients used to scale the two rewards; set λ p and λ s They are 0.02 and 1 respectively.

[0069] According to the above-mentioned distributed unmanned boat formation control method based on the attention mechanism SAC, preferably, in step S5, the neighborhood information encoding module uses a multi-layer perceptron to embed the state of a single neighborhood unmanned boat, and calculates the average embedding vector of all neighborhood unmanned boat states; then, the average embedding vector is combined with the embedding representation of a single neighborhood unmanned boat to calculate the attention score of each neighborhood unmanned boat; finally, based on the attention score, the embedding representation of each neighborhood unmanned boat is weighted and summed to obtain the joint representation of all neighborhood information states, thereby aggregating several neighborhood information into an embedding vector of fixed length;

[0070] The neighborhood information encoding module transforms its own observation space into With each neighborhood observation space Concatenate them and use a multi-layer perceptron to generate several fixed-length neighborhood embedding vectors

[0071]

[0072] Among them, j∈[1,K] represents the number of the neighboring unmanned boat, W e is a multi-layer perceptron ψ e network parameters.

[0073] Then embed the vector Passed into the subsequent multi-layer perceptron ψ h , and obtain the pairwise interaction feature vectors between the controlled unmanned boat and its neighborhood:

[0074]

[0075] The average pooling method is used to obtain the average embedding vector of all neighborhood individuals.

[0076]

[0077] The average embedding vector With each neighborhood embedding vector After concatenation, it is passed into a new multi-layer perceptron ψ α , get the attention score of each neighborhood embedding vector

[0078]

[0079] Use the softmax function for normalization:

[0080]

[0081] Based on the normalized attention score For all neighborhood interaction vectors Weighted summation is performed to obtain an embedding vector that aggregates all neighborhood information:

[0082]

[0083] Therefore, it is possible to transform any number of neighborhood information inputs into fixed-size embedding outputs;

[0084] Follower Unmanned Boat U i Mission Observation Environmental Observation Use a multi-layer perceptron φ respectively t ,φ s Encode to capture its internal feature information:

[0085]

[0086] Based on task information feature encoding Neighborhood Information Encoding And environmental information encoding Get the complete status code information:

[0087]

[0088] Value Network Q i With the policy network π i According to the complete status code information Get follower unmanned boat U i final decision.

[0089] A second aspect of the present invention provides an electronic device, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, any step of the distributed unmanned boat formation control method based on the attention mechanism SAC as described in the first aspect is implemented.

[0090] The third aspect of the present invention provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a computer processor, it implements any step in the distributed unmanned boat formation control method based on the attention mechanism SAC as described in the first aspect.

[0091] Compared with the prior art, the present invention has the following beneficial effects:

[0092] 1. The present invention adopts an unmanned boat chain formation structure based on the leader-follower framework, which can decouple the relationship between the leader unmanned boat and the follower unmanned boat, realize parameterized number of unmanned boats and formation shape, and thus reduce the communication pressure of the leader unmanned boat; each follower unmanned boat only needs to pay attention to and track its direct preceding leader unmanned boat, rather than all follower unmanned boats jointly tracking the same leader unmanned boat, thereby reducing the communication pressure of the formation leader unmanned boat, and by minimizing the tracking error of each unmanned boat, a stable formation structure can be achieved, and at the same time, in order to avoid collisions within the formation, an attention mechanism is introduced, so that the formation control algorithm can better understand the correlation between the state of the neighboring unmanned boats and the controlled unmanned boats themselves, and achieve better synergy effect.

[0093] 2. The present invention adopts a decentralized training strategy, treating each unmanned boat in the formation as an independent unit with the ability to communicate with each other, and is equipped with a flexible actor-critic network based on the attention mechanism; each unmanned boat takes independent action based on its own observation information and the communication information of the neighborhood, thereby realizing scalable follower unmanned boat formation distributed control. Therefore, each unmanned boat can learn to update its own value network and strategy network by interacting with the environment together, and finally form an effective formation control strategy.

[0094] 3. The WAM-V unmanned boats adopted in the present invention perform the same tasks, so that the follower unmanned boats in the formation can share the same observation, action space and reward function; based on the distributed training method and the chain leader-follower structure, the present invention designs a corresponding Markov decision process for each follower unmanned boat, which can realize the dynamic navigation strategy of multi-boat close formation, thereby realizing the autonomous navigation of the unmanned boat formation in unknown waters.

[0095] 4. In the present invention, the controlled unmanned boat not only needs to closely follow its direct leader to ensure that the ideal formation can be maintained, but also has the ability to predict the behavior of other unmanned boats in its neighborhood, so as to avoid potential collision risks. The present invention adopts a neighborhood information encoding module based on the attention mechanism, which is designed to effectively encode the collective influence of unmanned boats in the neighborhood and accurately calculate the importance of each neighborhood unmanned boat relative to the current controlled unmanned boat; with the help of this module, the decision network can analyze and integrate neighborhood information more deeply, and evaluate the influence of the relative state of the controlled unmanned boat and its neighborhood on the formation control, and then formulate an efficient and accurate formation control strategy.

[0096] 5. Based on the basic principle of the attention mechanism, the present invention uses a multi-layer perceptron to embed the state of a single neighborhood unmanned boat, and calculates the average embedding vector of all neighborhood unmanned boat states. Then, the average embedding vector is combined with the embedding representation of a single neighborhood unmanned boat to calculate the attention score of each neighborhood unmanned boat. Finally, based on the attention score, the embedding representation of each neighborhood unmanned boat is weightedly summed to obtain the joint representation of all neighborhood information states, thereby aggregating several neighborhood information into an embedding vector of fixed length.

[0097] 6. Attention Mechanism is an important mechanism of the human brain when processing information. It enables humans to selectively focus on certain key information while filtering out irrelevant content. This mechanism is not only the key to humans' efficient processing of information and completion of tasks, but also an important cornerstone of human cognitive activities. In the field of deep learning, researchers have designed an "attention mechanism" algorithm by imitating the characteristics of human selective attention. This algorithm enables neural networks to selectively focus on information related to the current task like humans when processing complex input data, thereby effectively filtering out the most critical content for the current task from the complex data and improving the performance and generalization ability of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0098] Figure 1 It is a schematic diagram of the process of the present invention;

[0099] Figure 2 It is a local coordinate schematic diagram of the unmanned boat chain formation structure;

[0100] Figure 3 Schematic diagram of the calculation process of the key-value pair attention model;

[0101] Figure 4 It is a schematic diagram of the follower's unmanned boat environment interaction process;

[0102] Figure 5 Schematic diagram of the neighborhood information encoding module based on the attention mechanism;

[0103] Figure 6 Schematic diagram of decision framework based on full state coding;

[0104] Figure 7 This is a pseudo code diagram of the formation control algorithm based on attention SAC;

[0105] Figure 8 This is a diagram of the ideal formation;

[0106] Fig. 9 This is a schematic diagram of the ROS communication network topology in the formation control task environment;

[0107] Fig.10 This is a schematic diagram of the second phase training environment for the formation control mission;

[0108] Fig.11 This is a schematic diagram for comparing reward curves;

[0109] Fig.12 This is a schematic diagram of the formation distance error comparison;

[0110] Fig.13This is a schematic diagram of the formation control test environment;

[0111] Fig.14 Schematic diagram of the formation operation trajectory and position deviation of the two algorithms in an obstacle-free environment;

[0112] Fig.15 Schematic diagram of the formation operation trajectory and position deviation of the two algorithms in several obstacle environments. DETAILED DESCRIPTION

[0113] The present invention is further described in detail below through specific examples, but the scope of the present invention is not limited thereto.

[0114] Example 1

[0115] A distributed unmanned boat formation control method based on attention mechanism SAC, the process is as follows Figure 1 As shown, the following steps are included:

[0116] S1. Use homogeneous intelligent agent unmanned boats to form an unmanned boat formation; all unmanned boats in the homogeneous intelligent agent unmanned boats are of the same type, all unmanned boats have the same state space S and action space A, and the same reward function r.

[0117] S2. Construct the unmanned boat formation into an unmanned boat chain formation structure based on a leader-follower framework; the unmanned boats in the unmanned boat chain formation structure are divided into follower unmanned boats and leader unmanned boats.

[0118] In step S2, the unmanned boat chain formation structure is a leader-follower model with a chain structure, specifically: each follower unmanned boat only needs to pay attention to and track its direct leading unmanned boat, rather than all follower unmanned boats jointly tracking the same leader unmanned boat, thereby reducing the communication pressure of the formation leader unmanned boat, and achieving a stable formation structure by minimizing the tracking error of each unmanned boat; the specific steps of constructing the unmanned boat chain formation structure are as follows: Figure 2 As shown:

[0119] First, a local coordinate system OX based on the bow direction of the pilot unmanned boat is constructed. B Y B ; The local coordinate system OX B Y B With the world coordinate system OX W Y W share the same origin, and Y B The direction of the axis is consistent with the heading of the direct leader unmanned boat of each follower unmanned boat, and the local coordinate system OX B Y B With the world coordinate system OX W Y WCoordinate transformation relationship:

[0120]

[0121] in, W x i , W y i They represent the follower unmanned boat U i In the world coordinate system OX W Y W The coordinate position of B x i , B y i They represent the follower unmanned boat U i In the local coordinate system OX B Y B The new coordinates in .

[0122] In the local coordinate system OX B Y B The following represents the velocity vector [u i ,v i ]:

[0123]

[0124] in, W u i and W v i They represent the unmanned boat along the world coordinate system X W Axis and Y W Speed ​​in the axis direction; and They represent the unmanned boat along the local coordinate system X B Axis and Y B The speed in the axis direction.

[0125] The coordinate relationship of all follower unmanned boats is converted to each pair of specific local coordinate systems, and the originally complex unmanned boat formation control problem is converted into the problem of each follower maintaining the position relationship between it and its direct leader; using the queue matrix F s Describe the relationship between formations:

[0126]

[0127] Among them, the first line indicates the number of each unmanned boat in the formation; and Represents the local coordinate system of each follower unmanned boat and its direct leader The horizontal and vertical distance offsets required to be maintained; IL(U i) indicates that the follower unmanned boat U i The number of the direct pilot.

[0128] According to the queue matrix F s Configure the formation, with the central unmanned boat as the leader of the cluster, and set the distance offset between the follower unmanned boat and its direct leader The follower unmanned boats can form a chain tracking one by one, and a tight formation structure can be achieved;

[0129] Based on the above chain leader-follower structure, the control target is:

[0130]

[0131] Among them, U follower represents the set of follower unmanned boats; H i Indicates the preset position of the follower unmanned boat in the formation; H′ i Indicates the actual position of the follower in the formation, which is the actual coordinate position p′ of the follower unmanned boat i Coordinate position with the pilot Definition of deviation; Indicates follower unmanned boat U i The coordinate position of the direct pilot unmanned boat.

[0132] In step S3, the attention mechanism first uses the attention mechanism to construct a key-value pair attention model, such as Figure 3 As shown:

[0133] (K,V)=[(k1,v1),(k2,v2),…,(k n ,v n )];

[0134] Among them, K represents the key, which is used to calculate the attention distribution; V represents the value, which is used to calculate the final aggregate information based on the attention distribution.

[0135] The final aggregate information is expressed as:

[0136]

[0137] The calculation of the key-value pair attention model is divided into three stages: using the scoring function to calculate the similarity between the query vector q and the current input key K (key); normalizing different scores through the softmax function to obtain the attention weight of each part; and performing information aggregation based on the attention distribution and the corresponding value.

[0138] The attention mechanism then uses the key-value pair attention model to build a multi-head attention model to improve the expression and generalization capabilities of the attention mechanism. Specifically, through multiple query vectors Q = [q1, q2, ..., q m ] in parallel from the input information set (K, V) = [(k1, v1), (k2, v2), …, (k n ,v n )] to retrieve multiple sets of related information; during the query process, each query vector q i It will focus on different parts of the input information, and can deeply analyze and understand the current input information from different dimensions or perspectives, so as to learn more comprehensive feature information; finally, the final output is obtained by splicing the results of all query vectors:

[0139]

[0140] Among them, the symbol Represents a vector concatenation operation.

[0141] S3. An attention mechanism is introduced into the unmanned boat chain formation structure to help the unmanned boat chain formation structure process neighborhood information to achieve safe and stable formation control.

[0142] The unmanned boat chain formation structure adopts a decentralized training strategy. Each unmanned boat in the formation is an independent unit with mutual communication capabilities and is equipped with a flexible actor-critic network based on the attention mechanism. Each unmanned boat takes independent actions based on its own observation information and the communication information of the neighborhood. Each unmanned boat learns to update its own value network and strategy network by interacting with the environment together, and finally learns an effective formation control strategy, such as Figure 4 As shown; the unmanned boats in the unmanned boat chain formation structure are WAM-V unmanned boats, and they all perform the same task, and the follower unmanned boats in the formation share the same observation, action space and reward function.

[0143] S4. Design a corresponding Markov decision process for each follower unmanned boat in the unmanned boat chain formation structure.

[0144] In step S4, in the unmanned boat chain formation structure, the Markov decision process of the follower unmanned boat includes: S401, obtaining the follower unmanned boat U i The complete observation space of; in step S401,

[0145] By Followers Unmanned Boat U i Direct pilot unmanned boat Position in local coordinate system With the queue matrix F s, get the follower unmanned boat U i Target position in formation Further obtain the target location With the follower unmanned boat U i The relative position deviation It can help the follower unmanned boat to clearly identify its tracking target.

[0146] Keep following UAV i The stable relative position relationship between the follower and its direct leader can realize the rapid formation and long-term maintenance of the formation, making it easier for the follower to accurately control its own catching-up speed and obtain the follower unmanned boat U i The task observation space is:

[0147]

[0148] in, Indicates follower unmanned boat U i The speed relationship with its direct leader.

[0149] Measurement Follower Unmanned Boat U i The motion state in the observation space is expressed as the coordinate position in the local coordinate system ( B x i , B y i ),speed( B u i , B v i ) and heading angle B ψ i Add to follower UAV i In the observation space, it can effectively manipulate and avoid collisions:

[0150]

[0151] Collect follower unmanned boat U i Neighborhood status information As its neighborhood observation space:

[0152]

[0153] in, and For the follower unmanned boat U i The relative position and relative speed of the unmanned boats in the neighborhood; K≤N is the number of neighbors that each unmanned boat can observe.

[0154] The environmental observation information is added to the observation space of the follower unmanned boat. Specifically, the external environmental data is collected by using the lidar, and the redundant information is filtered using the minimum pooling method; and the point cloud dataset is Convert to local coordinate system Next relative to the accused follower unmanned boat U i The coordinate position of the follower unmanned boat enables it to sense external environmental obstacles:

[0155]

[0156] Thus, the follower unmanned boat U i Environmental observation space:

[0157]

[0158] Finally, the follower unmanned boat U i The complete observation space of:

[0159]

[0160] S402, obtain the follower unmanned boat U i action space; in step S402,

[0161] The observed state of the follower unmanned boat is used to obtain the distribution of the actual control actions of the follower unmanned boat:

[0162]

[0163] Among them, n iτ 、n iw Follower unmanned boat U i The propulsion command and rudder angle steering command of the thruster; considering that when the formation has not yet been formed or is in an unstable state, the follower unmanned boat needs to be able to quickly adjust its speed state to catch up with the leader, the follower unmanned boat has a wider propulsion command control range, that is, [-1.0, 1.0], to ensure that the follower unmanned boat can have a faster speed and stronger braking ability. At the same time, a larger rudder angle steering range is set, that is, [-25°, 25°], so that the follower can have a smaller turning radius.

[0164] S403, obtain the follower unmanned boat U i The complete reward function of; in step S403,

[0165] Each follower unmanned boat in the formation maintains a preset offset distance with its direct leader in each time step, that is, the follower unmanned boat should minimize the distance from the desired target position. The relative position deviation Reduce the difference between the expected formation and the expected formation; quantify the effect of the formation maintenance strategy and introduce the follower unmanned boat U i Location Constraint Rewards

[0166]

[0167] In the process of formation control, in order to avoid collisions between unmanned boats and between unmanned boats and obstacles, the system should give certain penalties to unmanned boats that collide in the formation during the formation coordinated cruising process; in order to quantify this penalty and guide the unmanned boats to avoid collisions, a sparse collision penalty term for the unmanned boats that collide is constructed.

[0168]

[0169] Among them, d s can represents the distance information detected by the laser radar, D collision Indicates the distance threshold at which a collision occurs.

[0170] In order to accelerate the learning of the avoidance strategy of the unmanned boat and take avoidance actions in advance before a potential collision occurs, an additional reward based on the safe neighborhood is constructed. Specifically, the follower unmanned boat needs to perceive all the information of the unmanned boats or obstacles in its neighborhood, and its detection range covers all directions of the unmanned boat; at the same time, in order to give the follower unmanned boat a certain motion offset, the warning range D′ of the follower unmanned boat safe Set the distance D slightly smaller than the ideal formation distance of the unmanned boat formation When other unmanned boats or obstacles enter the warning area, the accused unmanned boat will be given an additional penalty. Used to warn the unmanned boat to make evasive action in advance; additional penalty defined for:

[0171]

[0172] Among them, D′ safe Indicates the warning range of the follower unmanned boat; D formation Indicates the ideal formation distance of unmanned boats.

[0173] Based on the above rewards, get the follower unmanned boat U i The complete reward function is:

[0174]

[0175] Among them, λ p and λ s are constant coefficients used to scale the two rewards; set λ p and λ s They are 0.02 and 1 respectively.

[0176] S5. Construct a neighborhood information encoding module and decision network based on the attention mechanism. The neighborhood information encoding module is used to effectively encode the collective influence of the unmanned boats in the neighborhood and accurately calculate the importance of each neighborhood unmanned boat relative to the currently controlled unmanned boat. With the help of the neighborhood information encoding module, the decision network can analyze and integrate neighborhood information more deeply, and evaluate the impact of the relative state of the controlled unmanned boat and its neighborhood on the formation control, and formulate an efficient and accurate formation control strategy.

[0177] In step S5, the neighborhood information encoding module uses a multi-layer perceptron to embed the state of a single neighborhood unmanned boat and calculates the average embedding vector of all neighborhood unmanned boat states; then, the average embedding vector is combined with the embedding representation of a single neighborhood unmanned boat to calculate the attention score of each neighborhood unmanned boat; finally, based on the attention score, the embedding representation of each neighborhood unmanned boat is weighted and summed to obtain the joint representation of all neighborhood information states, thereby aggregating several neighborhood information into an embedding vector of fixed length;

[0178] like Figure 5 As shown in the figure, the neighborhood information encoding module transforms its own observation space into With each neighborhood observation space Concatenate them and use a multi-layer perceptron to generate several fixed-length neighborhood embedding vectors

[0179]

[0180] Among them, j∈[1,K] represents the number of the neighboring unmanned boat, W e is a multi-layer perceptron ψ e network parameters.

[0181] Then embed the vector Passed into the subsequent multi-layer perceptron ψ h , and obtain the pairwise interaction feature vectors between the controlled unmanned boat and its neighborhood:

[0182]

[0183] The average pooling method is used to obtain the average embedding vector of all neighborhood individuals.

[0184]

[0185] The average embedding vector With each neighborhood embedding vector After concatenation, it is passed into a new multi-layer perceptron ψ α , get the attention score of each neighborhood embedding vector

[0186]

[0187] Use the softmax function for normalization:

[0188]

[0189] Based on the normalized attention score For all neighborhood interaction vectors Weighted summation is performed to obtain an embedding vector that aggregates all neighborhood information:

[0190]

[0191] Therefore, it is possible to transform any number of neighborhood information inputs into fixed-size embedding outputs;

[0192] Follower Unmanned Boat U i Mission Observation Environmental Observation Use a multi-layer perceptron φ respectively t ,φ s Encode to capture its internal feature information:

[0193]

[0194] Based on task information feature encoding Neighborhood Information Encoding And environmental information encoding Get the complete status code information:

[0195]

[0196] Value Network Q i With the policy network π i According to the complete status code information Get follower unmanned boat U i The final decision framework, such as Figure 6 shown.

[0197] S6. Train the unmanned boat chain formation structure in a simulation environment.

[0198] All follower unmanned boats in the unmanned boat chain formation structure are homogeneous agents, that is, all unmanned boats are of the same type, and all unmanned boats have the same state space S and action space A, as well as the same reward function r; let all unmanned boats share a strategy network π φ and value network Q θ; During the training process, each unmanned boat independently makes decisions and executes its own control instructions based on its own state observation and neighborhood information communication; Based on the interaction of all unmanned boats, the unmanned boat chain formation formation is realized; During the training process, there is an overall formation leader that sails with a certain strategy or navigation trajectory, and at the beginning of each round, the direct leader of each follower is designated to train the chain formation control strategy of the unmanned boat; The pseudo code of the formation control algorithm for training the unmanned boat chain formation structure is as follows Figure 7 shown.

[0199] In order to train and test the proposed formation control algorithm, this embodiment uses a formation consisting of three unmanned boats to conduct algorithm experiments; one of the unmanned boats is used as a leader, responsible for leading the direction of travel of the entire formation, while the other two unmanned boats are used as followers, each using the distributed formation control algorithm proposed in this embodiment to perform collaborative operations.

[0200] 1. Network parameter settings:

[0201] In the experiment of this embodiment, since the unmanned boat formation consists of three units, in order to ensure that each follower can maintain communication and cooperation with at least one other unmanned boat, this embodiment sets the number of neighborhoods K to 2; the neighborhood encoder parts ψ e , h , α And the task information encoder φ t and the environmental information encoder φ s The number of hidden layer units is (256, 256); the complete information encoder φ o The number of hidden layer units is 256; finally, the strategy network π is used φ and value network Q θ The output layer obtains the action strategy and value estimation; other parameter settings in the training process are similar to those in Chapter 3, see Table 1 for details.

[0202] Table 1 Network parameter settings

[0203] parameter Numeric The number of neighbors K 2 <![CDATA[ψ e Number of hidden layer units]]> 256,256 <![CDATA[ψ h Number of hidden layer units]]> 256,256 <![CDATA[ψ α Number of hidden layer units]]> 256,256 <![CDATA[φ t Number of hidden layer units]]> 256,256 <![CDATA[φ s Number of hidden layer units]]> 256,256 <![CDATA[φ o Number of hidden layer units]]> 256 Hidden layer activation function Relu Optimizer Adam Learning Rate <![CDATA[1×10 -4 ]]> Batch size 512 Reward discount factor γ 0.99 Experience replay pool size <![CDATA[1×10 6 ]]> Network gradient update frequency 1 Target network update frequency 1 Target network weighted average parameter τ 0.01 Regularization coefficient α change strategy automatic

[0204] 2. Training environment settings:

[0205] In order to train and test the distributed formation control algorithm described in this embodiment, this embodiment constructs three ideal formations of unmanned boats required for the experiment according to the chain formation generation method, such as Figure 8 Specifically, in this embodiment, each follower unmanned boat is set to be 7.5m offset from its direct leader in the horizontal direction and 13m offset in the vertical direction in the local coordinate system, and the formation matrix F is obtained. s for:

[0206]

[0207] In order to establish a communication connection between the unmanned boat formation control algorithm and the simulation training environment, this embodiment designs an environment interaction module dedicated to the unmanned boat formation control task and three corresponding unmanned boat control modules; wherein, in addition to being used to realize the initialization, conditional termination, random reset and other operations of the unmanned boat formation training scene, the environment interaction module is also equivalent to the control core and information interaction hub of the unmanned boat formation; specifically, the environment interaction module can calculate and set the formation constraint target of each unmanned boat to ensure that the formation can complete the cruising task according to the predetermined formation; at the same time, the environment interaction module can obtain the relative position and relative speed of the boat and its neighborhood in real time according to the specified unmanned boat number, and provide necessary data support for the unmanned boat control module; and the environment interaction module is also responsible for collecting and integrating the reward information of each unmanned boat to help the subsequent reinforcement learning algorithm to optimize learning; and the unmanned boat control module is responsible for controlling the independent movement of each unmanned boat according to the controller instructions, and calculating the reward value obtained by each unmanned boat based on the observation data.

[0208] Similarly, this embodiment encapsulates the above four modules into the " / gym" node to obtain the ROS communication network in the unmanned boat formation control task environment, such as Fig. 9 As shown; this embodiment sets a corresponding numbered namespace for each unmanned boat to realize the reuse of the unmanned boat model control function and the sensor data reading function code in the VRX environment; at the same time, each unmanned boat control module in the " / gym" node respectively sends and receives the unmanned boat propeller control topic and the lidar data topic, driving the unmanned boat to perform reinforcement learning exploration training in the VRX environment.

[0209] In addition, in order to train and test the formation control algorithm of this embodiment, this embodiment also sets the simulation environment to a micro-wave water scene, and adopts a phased training plan: in the early stage of training, in order to allow the unmanned boat formation to focus on learning the target tracking ability, the unmanned boat formation is placed in an obstacle-free environment for training, and a navigator is set to guide the followers to the randomly generated target point, thereby strengthening the formation constraint ability of the unmanned boat alone and temporarily ignoring the impact of obstacles on the formation movement; as the training deepens, the unmanned boat formation is placed in a more challenging environment, such as Fig.10 As shown, the scene has four large static obstacles and four static obstacles at random positions. At the beginning of each round of training, the initial positions of the unmanned boat formation and its target point will be randomly set in two opposite yellow solid line boxes to ensure that the formation needs to cross the central obstacle area, thereby further strengthening and improving the unmanned boat formation's ability to adapt to complex environments on the basis of achieving target tracking.

[0210] 3. Algorithm training and analysis:

[0211] Based on the above training environment and training scheme, this embodiment is an adaptive navigation algorithm obtained by the leader unmanned boat according to the training to guide the formation to the target point correctly; then the unmanned boat formation control algorithm described in this embodiment is trained; the unmanned boat control frequency is set to 10Hz, and the maximum training step size is 2×10 7 Steps, the maximum step length per round is 1000 steps. At the same time, in order to fully verify the effectiveness of the algorithm described in this embodiment, this embodiment selects the SAC algorithm that uses a multilayer perceptron to process neighborhood information as a comparison algorithm, and performs control training in the same experimental environment.

[0212] Fig.11 The average rewards accumulated by all follower drones in each training round under the two algorithms are shown in the figure. Fig.11 It can be seen that the attention-based SAC (AM-SAC) algorithm proposed in this embodiment shows a convergence trend after about 4800 training cycles when trained in an obstacle-free environment. Fig.10 When training in the obstacle scene shown in the figure, the strategy obtained in the early training cannot effectively avoid these obstacles; despite this, the strategy can still drive the follower unmanned boat to track and approach the ideal formation position as much as possible; as the training gradually deepens, convergence in the obstacle scene is achieved after about 12,500 training cycles, and the reward value enters a relatively stable dynamic equilibrium state; while the SAC algorithm that does not use the attention mechanism obtains rewards similar to those of the AM-SAC algorithm in the early training stage, its performance is significantly reduced when facing a complex environment containing obstacles, and the reward value obtained at the final convergence is significantly less than that of the AM-SAC algorithm, highlighting the advantages of the algorithm in this embodiment. Fig.12 The average distance deviation between all followers and the ideal formation under the two algorithms during the training process is presented, which reflects the ability of the follower unmanned boat to maintain the stability of the formation during the training process. Fig.12 It can be more intuitively seen that the algorithm of this embodiment has better formation constraint performance.

[0213] In order to further verify the actual performance of the formation control algorithm improved based on the attention mechanism in this embodiment, this embodiment also recreated two new test environments, one is a simple environment without obstacles, and the other is an unstructured environment with multiple obstacles, such as Fig.13As shown in the figure, in an obstacle-free environment, the target point position is set at the side and rear of the unmanned boat formation, aiming to simulate a scenario that requires the formation to perform a large-scale turning movement, so as to test whether the unmanned boat formation can maintain good formation constraint performance when performing large turning actions; in another set of obstacle environment tests, several static obstacles are arranged to simulate the complex obstacles that may be encountered in actual navigation, so as to verify whether the unmanned boat formation control algorithm can effectively avoid obstacles on the route while maintaining the stability of the formation.

[0214] like Fig.14 The navigation trajectory and position deviation of the two formation control algorithms driving the unmanned boat formation to perform large-scale steering in an obstacle-free environment are demonstrated; it can be seen that in the absence of obstacle interference, the unmanned boat formation can achieve a good formation constraint effect, but relatively speaking, the algorithm of this embodiment shows better performance in formation error control; Fig.15 The two algorithms are shown to move in formation in a scene with several obstacles. Fig.15 It can be observed that in complex scenarios, the distributed formation control algorithm proposed in this embodiment not only shows a more significant formation constraint effect, but also shows excellent performance in obstacle avoidance and collision avoidance between boats; while the SAC algorithm without the attention mechanism cannot well judge the impact of neighboring unmanned boats on their own navigation safety, resulting in collisions when passing through narrow areas due to the reduction of formation spacing.

[0215] 4. Summary of this embodiment

[0216] This embodiment aims at the follower unmanned boat formation control problem and proposes a distributed unmanned boat formation control method based on the attention mechanism SAC; firstly, in order to realize a distributed formation control form that is easy to expand and has a low communication burden, this embodiment improves the traditional leader-follower formation architecture, proposes a chain leader-follower formation organizational structure, and simplifies the formation control problem; on this basis, according to the task requirements, a corresponding Markov decision process is constructed for the follower unmanned boat formation control problem, and the design of the observation space, action space and reward function is analyzed and introduced; secondly, in view of the difficulty in processing neighborhood information in the collaborative interaction process of multiple agents, this embodiment uses the attention encoding module to aggregate the neighborhood information state into a fixed-length feature vector, thereby improving the model's ability to extract important information; finally, based on simulation experiments, it is verified that the formation control algorithm proposed in this embodiment has good formation constraint performance and can effectively avoid collisions between formations.

[0217] Example 2

[0218] An electronic device comprises a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, any step in a distributed unmanned boat formation control method based on an attention mechanism SAC as described in Example 1 is implemented.

[0219] Further, the distributed unmanned boat formation control method process based on the attention mechanism SAC described in Example 1 can be implemented as a computer software program. For example, this embodiment includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains a program code for executing the method. In such an embodiment, the computer program can be downloaded and installed from the network, and / or installed from a removable medium. When the computer program is executed by the processor, the above functions defined in the method of the present application are executed.

[0220] Example 3

[0221] A computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, any step in a distributed unmanned boat formation control method based on an attention mechanism SAC as described in Example 1 is implemented.

[0222] The computer-readable medium described in the present application may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to, an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0223] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0224] In summary, the present invention effectively overcomes the deficiencies in the prior art and has a high industrial utilization value. The above embodiments serve to illustrate the substantive content of the present invention, but do not limit the protection scope of the present invention. Those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the essence and protection scope of the technical solution of the present invention.

[0225] The above embodiments are specific implementation methods of the present invention, but the implementation methods of the present invention are not limited to the above embodiments. Any other combination, change, modification, substitution and simplification that does not exceed the design concept of the present invention shall fall within the protection scope of the present invention.

Claims

1. A distributed unmanned boat formation control method based on attention mechanism SAC, characterized in that: The following steps are involved: S1. Use homogeneous intelligent agent unmanned boats to form an unmanned boat fleet; all unmanned boats in the homogeneous intelligent agent unmanned boats are of the same type, and all unmanned boats have the same state space S and action space and the same reward function r; S2. The unmanned boat formation is constructed as an unmanned boat chain formation structure based on a leader-follower framework; the unmanned boats in the unmanned boat chain formation structure are divided into follower unmanned boats and leader unmanned boats; S3. Introducing the attention mechanism into the unmanned boat chain formation structure to help the unmanned boat chain formation structure process neighborhood information to achieve safe and stable formation control; S4. Design a corresponding Markov decision process for each follower unmanned boat in the unmanned boat chain formation structure; S5. Construct a neighborhood information encoding module and decision network based on the attention mechanism. The neighborhood information encoding module is used to effectively encode the collective influence of the unmanned boats in the neighborhood and accurately calculate the importance of each neighborhood unmanned boat relative to the currently controlled unmanned boat; S6. Train the unmanned boat chain formation structure in a simulation environment.

2. The distributed unmanned boat formation control method based on the attention mechanism SAC according to claim 1 is characterized in that: In step S2, the specific steps of constructing the unmanned boat chain formation structure are: First, a local coordinate system OX based on the bow direction of the pilot unmanned boat is constructed. B Y B ; The local coordinate system OX B Y B With the world coordinate system OX W Y W share the same origin, and Y B The direction of the axis is consistent with the heading of the direct leader unmanned boat of each follower unmanned boat, and the local coordinate system OX B Y B With the world coordinate system OX W Y W Coordinate transformation relationship: in, They represent the follower unmanned boat U i In the world coordinate system OX W Y W The coordinate position of B x i , B y i They represent the follower unmanned boat U i In the local coordinate system OX B Y B The new coordinates in ; In the local coordinate system OX B Y B The following represents the velocity vector [u i ,v i ]: in, W u i and W v i They represent the unmanned boat along the world coordinate system X W Axis and Y W Speed ​​in the axis direction; and They represent the unmanned boat along the local coordinate system X B Axis and Y B Speed ​​in the axis direction; The coordinate relationship of all follower unmanned boats is converted to each pair of specific local coordinate systems; using the queue matrix F s Describe the relationship between formations: Among them, the first line indicates the number of each unmanned boat in the formation; and Represents the local coordinate system of each follower unmanned boat and its direct leader The horizontal and vertical distance offsets required to be maintained; IL(U i ) indicates that the follower unmanned boat U i The number of the direct pilot; According to the queue matrix F s Configure the formation, with the central unmanned boat as the leader of the cluster, and set the distance offset between the follower unmanned boat and its direct leader The follower unmanned boats can form a chain tracking one by one, and a tight formation structure can be achieved; Based on the above chain leader-follower structure, the control target is: Among them, U follower represents the set of follower unmanned boats; H i Indicates the preset position of the follower unmanned boat in the formation; H′ i Indicates the actual position of the follower in the formation, which is the actual coordinate position p′ of the follower unmanned boat i Coordinate position with the pilot Definition of deviation; Indicates follower unmanned boat U i The coordinate position of the direct pilot unmanned boat.

3. The distributed unmanned boat formation control method based on the attention mechanism SAC according to claim 1 is characterized in that: In step S3, the attention mechanism first uses the attention mechanism to build a key-value pair attention model: (K,V)=[(k1,v1),(k2,v2),…,(k n ,v n )]; Among them, K represents the key, which is used to calculate the attention distribution; V represents the value, which is used to calculate the final aggregate information based on the attention distribution; The final aggregate information is expressed as: The calculation of the key-value pair attention model is divided into three stages: using the scoring function to calculate the similarity between the query vector q and the current input key K; normalizing different scores through the softmax function to obtain the attention weight of each part; and aggregating information based on the attention distribution and the corresponding value; The attention mechanism then uses the key-value pair attention model to build a multi-head attention model to improve the expression and generalization capabilities of the attention mechanism. Specifically, through multiple query vectors Q = [q1, q2, ..., q m ] in parallel from the input information set (K, V) = [(k1, v1), (k2, v2), …, (k n ,v n )] to retrieve multiple sets of related information; during the query process, each query vector q i It will focus on different parts of the input information, and can deeply analyze and understand the current input information from different dimensions or perspectives, so as to learn more comprehensive feature information; finally, the final output is obtained by splicing the results of all query vectors: Among them, the symbol Represents a vector concatenation operation.

4. The distributed unmanned boat formation control method based on the attention mechanism SAC according to claim 1 is characterized in that: The unmanned boat chain formation structure adopts a decentralized training strategy. Each unmanned boat in the formation is an independent unit with the ability to communicate with each other, and is equipped with a flexible actor-critic network based on the attention mechanism. Each unmanned boat takes action independently based on its own observation information and the communication information of the neighborhood. Each unmanned boat learns to update its own value network and strategy network by interacting with the environment together, and finally learns an effective formation control strategy.

5. The distributed unmanned boat formation control method based on the attention mechanism SAC according to claim 1 is characterized in that: In step S4, in the unmanned boat chain formation structure, the Markov decision process of the follower unmanned boat includes: S401, obtaining the follower unmanned boat U i The complete observation space of the follower unmanned boat U i S403, obtain the follower unmanned boat U i The complete reward function.

6. The distributed unmanned boat formation control method based on the attention mechanism SAC according to claim 5 is characterized in that: In step S401, By Followers Unmanned Boat U i Direct pilot unmanned boat Position in local coordinate system With the queue matrix F s , get the follower unmanned boat U i Target position in formation Further obtain the target location With the follower unmanned boat U i The relative position deviation Keep following UAV i The stable relative position relationship between the follower and its direct leader can realize the rapid formation and long-term maintenance of the formation, making it easier for the follower to accurately control its own catching-up speed and obtain the follower unmanned boat U i The task observation space is: in, Indicates follower unmanned boat U i speed relationship with its immediate leader; Measurement Follower Unmanned Boat U i The motion state in the observation space is expressed as the coordinate position in the local coordinate system ( B x i , B y i ),speed( B u i , B v i ) and heading angle B ψ i Add to follower UAV i In the observation space: Collect follower unmanned boat U i Neighborhood status information As its neighborhood observation space: in, and For the follower unmanned boat U i The relative position and relative speed of the unmanned boats in the neighborhood; K≤N is the number of neighbors that each unmanned boat can observe; The environmental observation information is added to the observation space of the follower unmanned boat. Specifically, the external environmental data is collected by using the lidar, and the redundant information is filtered using the minimum pooling method; and the point cloud dataset is Convert to local coordinate system Next relative to the accused follower unmanned boat U i The coordinate position of the follower unmanned boat enables it to sense external environmental obstacles: Thus, the follower unmanned boat U i Environmental observation space: Finally, the follower unmanned boat U i The complete observation space of: In step S402, The observed state of the follower unmanned boat is used to obtain the distribution of the actual control actions of the follower unmanned boat: Among them, n iτ 、n iw Follower unmanned boat U i Propulsion commands for thrusters and rudder angle steering commands.

7. The distributed unmanned boat formation control method based on the attention mechanism SAC according to claim 6 is characterized in that: In step S403, Each follower unmanned boat in the formation maintains a preset offset distance with its direct leader in each time step, quantifies the effect of the formation maintenance strategy, and introduces the follower unmanned boat U i Location Constraint Rewards Constructing sparse collision penalty terms for unmanned boats that collide Among them, d s can represents the distance information detected by the laser radar, D collision Indicates the distance threshold for collision; Construct additional rewards based on safe neighborhoods and define additional penalties for: Among them, D′ safe Indicates the warning range of the follower unmanned boat; D formation Indicates the ideal formation distance of unmanned boats; Based on the above rewards, get the follower unmanned boat U i The complete reward function is: Among them, λ p and λ s are constant coefficients used to scale the two reward sizes.

8. The distributed unmanned boat formation control method based on the attention mechanism SAC according to claim 7 is characterized in that: In step S5, The neighborhood information encoding module transforms its own observation space into With each neighborhood observation space Concatenate them and use a multi-layer perceptron to generate several fixed-length neighborhood embedding vectors Among them, j∈[1,K] represents the number of the neighboring unmanned boat, We is the network parameter of the multi-layer perceptron ψe; Then embed the vector Passed into the subsequent multi-layer perceptron ψ h , and obtain the pairwise interaction feature vectors between the controlled unmanned boat and its neighborhood: The average pooling method is used to obtain the average embedding vector of all neighborhood individuals. The average embedding vector With each neighborhood embedding vector After concatenation, it is passed into a new multi-layer perceptron ψ α , get the attention score of each neighborhood embedding vector Use the softmax function for normalization: Based on the normalized attention score For all neighborhood interaction vectors Weighted summation is performed to obtain an embedding vector that aggregates all neighborhood information: Therefore, it is possible to transform any number of neighborhood information inputs into fixed-size embedding outputs; Follower Unmanned Boat U i Mission Observation Environmental Observation Use a multi-layer perceptron φ respectively t ,φ s Encode to capture its internal feature information: Based on task information feature encoding Neighborhood Information Encoding And environmental information encoding Get the complete status code information: Value Network Q i With the policy network π i According to the complete status code information Get follower unmanned boat U i final decision.

9. An electronic device comprising a memory and a processor, wherein a computer program is stored in the memory, wherein: When the processor executes the computer program, it implements any step in the distributed unmanned boat formation control method based on the attention mechanism SAC as described in any one of claims 1-8.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a computer processor, it implements any step in the distributed unmanned boat formation control method based on the attention mechanism SAC as described in any one of claims 1-8.

Citation Information

Cited By

  • Unmanned ship intelligent collision avoidance method based on mask attention mechanism and SAC

    CN120010526A

  • An intelligent collision avoidance method for unmanned boats based on masked attention mechanism and SAC

    CN120010526B

  • Evaporative waveguide unmanned ship buffer relay cooperative control method and system

    CN122909510A

  • A method and system for coordinated control of evaporation waveguide unmanned surface vessel buffer relay

    CN122909510B