Underwater glider formation control method based on multi-agent reinforcement learning

Abstract

This application provides a multi-agent reinforcement learning-based underwater glider formation control method, which can be applied to the field of underwater glider control technology. The method is applied to a control terminal capable of communicating with an underwater glider group, and includes: in an observation scenario where the underwater glider group is in a preset formation, determining the state spaces of the leader underwater glider and at least one follower underwater glider, and determining the action spaces of all underwater gliders in the group; predicting the leader underwater glider's state at time t; predicting the leader underwater glider's action at time t that enables it to reach its destination; outputting the follower actions at time t for each of the at least one follower underwater glider through a follower agent; and sending the follower actions at time t so that each of the at least one follower underwater glider maintains a preset formation with the leader underwater glider based on its follower actions at time t.

Description

Technical Field

[0001] This application relates to the field of underwater glider control technology, and more specifically, to an underwater glider formation control method based on multi-agent reinforcement learning. Background Technology

[0002] Underwater gliders can glide in a sawtooth-shaped trajectory in the underwater environment by adjusting their net buoyancy and attitude angle, in conjunction with the lift generated by their wings. Each sawtooth motion (one dive and one surfacing motion) of the underwater glider constitutes a profile. During underwater gliding, underwater gliders can achieve wide-area, long-term, and detailed continuous observation and monitoring of underwater environments such as the ocean. However, the control of underwater gliders in related technologies is simple and cannot meet the observation requirements. Summary of the Invention

[0003] In view of this, embodiments of this application provide an underwater glider formation control method based on multi-agent reinforcement learning.

[0004] One aspect of this application provides an underwater glider formation control method based on multi-agent reinforcement learning, applied to a control terminal capable of communicating with an underwater glider group. The underwater glider group includes a leader underwater glider with a preset formation and at least one follower underwater glider. The control terminal is deployed with a leader agent corresponding to the leader underwater glider and follower agents corresponding to each of the at least one follower underwater glider. The leader agent is used to output a leadership action that enables the leader underwater glider to navigate to the destination, and the follower agents are used to output a following action that enables each of the at least one follower underwater glider to maintain a preset formation with the leader underwater glider. The method includes: in an observation scenario where the underwater glider group is in the preset formation, determining the state of each of the leader underwater glider and at least one follower underwater glider based on the velocity components along different directions of each coordinate point in the inertial coordinate system of depth-averaged ocean current prediction data, the heading angle deviation and distance deviation of the underwater gliders. The action space of all underwater gliders in the underwater glider group is determined based on the heading angles of the underwater gliders at the water entry point at time t and time t+1, where t≥1. Based on the leader's underwater glider's leadership state at time t-1 and its output leadership action at time t-1 received by the leader agent, the leader's underwater glider's predicted leadership state at time t is predicted. Based on the leader's predicted leadership state at time t, the leader's predicted leadership action at time t, which enables the leader's underwater glider to reach its destination, is predicted. Based on the leader's predicted leadership state at time t, the leader's predicted leadership action at time t, and the follower's state at time t received by the follower agent, at least one follower underwater glider outputs its own follower action at time t. The follower action at time t is sent to at least one follower underwater glider so that each follower underwater glider maintains a preset formation with the leader underwater glider according to its follower action at time t.

[0005] According to an embodiment of this application, in an observation scenario with a preset formation of underwater gliders, the leader agent corresponding to the leader underwater glider outputs a leadership action to enable the leader underwater glider to navigate to the destination. The follower agents corresponding to at least one follower underwater glider output a following action to ensure that each follower underwater glider maintains a preset formation with the leader underwater glider. The predicted leadership state of the leader underwater glider at time t is obtained by predicting the actual leadership state and leadership action of the leader underwater glider at time t-1. Based on this predicted leadership state at time t, the predicted leadership state at time t is then used to predict the t-th... The system predicts the leader's actions at all times, so that even if the leader underwater glider has not yet emerged from the water at time t, the system can determine the follower underwater glider's actions at time t based on the predicted leader state, predicted leader action, and follower state at time t. This ensures that at least one follower underwater glider maintains a preset formation with the leader underwater glider based on its follower action at time t. This solves the problem of not being able to obtain the leader's state and output actions when the follower underwater glider emerges from the water before the leader underwater glider, ensuring the continuous operation of the underwater glider group and maintaining the preset formation. Attached Figure Description

[0006] The above and other objects, features and advantages of this application will become clearer from the following description of embodiments with reference to the accompanying drawings, in which:

[0007] Figure 1 An exemplary system architecture for applying a multi-agent reinforcement learning-based underwater glider formation control method according to an embodiment of this application is shown.

[0008] Figure 2 A flowchart of an underwater glider formation control method based on multi-agent reinforcement learning according to an embodiment of this application is shown.

[0009] Figure 3 A schematic diagram showing the current heading angle and desired heading angle of a leader underwater glider according to an embodiment of this application is provided.

[0010] Figure 4 A schematic diagram of the movement in the action space of an underwater glider according to an embodiment of this application is shown.

[0011] Figure 5 A flowchart of an underwater glider formation control method based on multi-agent reinforcement learning according to another embodiment of this application is shown.

[0012] Figure 6 A schematic diagram of an underwater glider unit in a preset formation according to an embodiment of this application is shown.

[0013] Figure 7 A block diagram of an underwater glider formation control device based on multi-agent reinforcement learning according to an embodiment of this application is shown. Detailed Implementation

[0014] The embodiments of this application will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of this application. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of this application for ease of explanation. However, it will be apparent that one or more embodiments may be implemented without these specific details. Furthermore, descriptions of well-known structures and technologies are omitted in the following description to avoid unnecessarily obscuring the concepts of this application.

[0015] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of this application. The terms “comprising,” “including,” etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0016] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0017] When using expressions such as "at least one of A, B and C", they should generally be interpreted in accordance with the meaning that is commonly understood by those skilled in the art (e.g., "a system having at least one of A, B and C" should include, but is not limited to, a system having A alone, a system having B alone, a system having C alone, a system having A and B, a system having A and C, a system having B and C, and / or a system having A, B and C, etc.).

[0018] In the embodiments of this application, the collection, updating, analysis, processing, use, transmission, provision, disclosure, and storage of data (e.g., including but not limited to user personal information) comply with relevant laws and regulations, are used for legitimate purposes, and do not violate public order and good morals. In particular, necessary measures have been taken to prevent unauthorized access to user personal information data and to safeguard user personal information security and network security.

[0019] In the embodiments of this application, the user's authorization or consent was obtained before obtaining or collecting the user's personal information.

[0020] Due to limitations such as small coverage area and single mission, single-aircraft underwater glider observation can no longer meet the needs for multi-dimensional, high-precision, and large-scale continuous observation of the marine environment. Therefore, it is necessary to use underwater glider groups formed by multiple underwater gliders to observe the marine environment.

[0021] Underwater gliders have a wide observation coverage, can complete tasks with high complexity, increase the richness of the ocean information acquired, and the simultaneous observation by multiple underwater gliders can also improve the reliability of underwater glider missions.

[0022] However, when underwater gliders perform navigation missions, the following problems exist: During underwater navigation, different underwater gliders are affected by varying ocean currents, leading to asynchronous surfacing. Communication can only be established after surfacing, so this asynchronous surfacing prevents timely synchronization of information between multiple gliders, affecting formation. Reinforcement learning-based formation methods suffer from inherent instability problems due to the state transitions of all underwater gliders being affected by their actions, making convergence difficult. Furthermore, reinforcement learning-based formation methods face the problem of dimensionality explosion as the number of underwater gliders increases, hindering their scaling to large-scale formations.

[0023] In view of this, embodiments of this application provide an underwater glider formation control method based on multi-agent reinforcement learning. Under an observation scenario with a preset formation, the leader agent corresponding to the leader glider outputs a leadership action to enable the leader glider to reach its destination. The follower agents corresponding to at least one follower glider output a following action to ensure that each follower glider maintains a preset formation with the leader glider. The predicted leadership state of the leader glider at time t is obtained by predicting the actual leadership state and leadership action of the leader glider at time t-1. Based on this predicted leadership state... The leader's predicted state at time t is used to predict the leader's predicted action at time t. This allows the follower's underwater glider to determine its following action at time t, even if the leader's underwater glider has not yet emerged from the water. This is achieved by using the predicted leader's predicted state, predicted action, and follower state at time t. This ensures that at least one follower underwater glider maintains a preset formation with the leader's underwater glider based on its following action at time t. This solves the problem of not being able to obtain the leader's state and output actions when the follower underwater glider emerges from the water before the leader, ensuring the continuous operation of the underwater glider group and maintaining its preset formation.

[0024] Figure 1 An exemplary system architecture for underwater glider formation control based on multi-agent reinforcement learning, according to embodiments of this application, is shown. It should be noted that... Figure 1 The examples shown are merely examples of system architectures that can be applied to the embodiments of this application, in order to help those skilled in the art understand the technical content of this application, but do not mean that the embodiments of this application cannot be used in other devices, systems, environments or scenarios.

[0025] like Figure 1 As shown, the system architecture 100 according to this embodiment may include a control terminal 104 and an underwater glider unit. The underwater glider unit includes a leader underwater glider 101, a first follower underwater glider 102, and a second follower underwater glider 103. The leader underwater glider 101, the first follower underwater glider 102, the second follower underwater glider 103, and the control terminal 104 can communicate via a network.

[0026] The control terminal 104 acquires the state and actions of the leader underwater glider 101, the first follower underwater glider 102, and the second follower underwater glider 103 at each moment, thereby executing the underwater glider formation control method based on multi-agent reinforcement learning according to the embodiments of this application.

[0027] It should be understood that Figure 1 The number of underwater gliders and control terminals in the underwater glider unit shown is merely illustrative. Depending on the implementation requirements, any number of underwater gliders, networks, and terminal devices can be included.

[0028] Figure 2 A flowchart of an underwater glider formation control method based on multi-agent reinforcement learning according to an embodiment of this application is shown.

[0029] like Figure 2 As shown, the method is applied to a control terminal capable of communicating with an underwater glider group. The underwater glider group includes a leader underwater glider with a preset formation and at least one follower underwater glider. The control terminal is equipped with a leader agent corresponding to the leader underwater glider and follower agents corresponding to each of the at least one follower underwater glider. The leader agent is used to output a leadership action that enables the leader underwater glider to sail to the destination, and the follower agent is used to output a following action that enables each of the at least one follower underwater glider to maintain a preset formation with the leader underwater glider. The method may include operations S210 to S250.

[0030] In operation S210, under the observation scenario where the underwater glider group is in a preset formation, the state space of the leader underwater glider and at least one follower underwater glider is determined based on the velocity components of each coordinate point in the inertial coordinate system along different directions, the heading angle deviation and distance deviation of the underwater gliders, according to the depth-averaged ocean current prediction data. The action space of all underwater gliders in the underwater glider group is determined based on the heading angle of the underwater glider at the water entry point at time t and the heading angle of the underwater entry point at time t+1.

[0031] In operation S220, based on the leader's underwater glider's leadership state at time t-1 received by the leader agent and the leader's action at time t-1 output, the leader's predicted leadership state of the underwater glider at time t is predicted.

[0032] In operation S230, based on the leader's predicted state at time t, the leader's predicted action at time t that enables the underwater glider to reach the destination is predicted.

[0033] In operation S240, based on the leader prediction state at time t, the leader prediction action at time t, and the follower agent's follow state at time t, the follower agent outputs the follow action at time t for each of the follower underwater gliders.

[0034] In operation S250, a follow action at time t is sent to at least one follower underwater glider so that each of the at least one follower underwater glider maintains a preset formation with the leader underwater glider according to the follow action at time t.

[0035] Underwater gliders can perform navigation missions in the ocean. Since underwater gliders cannot receive or send signals while navigating underwater, they can only do so after surfacing. Therefore, a control terminal can be set up on the surface of the water to communicate with the underwater glider after it surfaces. The control terminal receives and sends signals back to the underwater glider.

[0036] An underwater glider group can include multiple underwater gliders. The underwater gliders in the group can be configured into a preset formation. The leader underwater glider can be either the one at the very front of the formation or the one in the center. Each underwater glider corresponds to an agent at the control terminal, which can output the glider's actions. The leader agent outputs leadership actions to guide the leader underwater glider to its destination, while follower agents output following actions to ensure that at least one follower underwater glider maintains a preset formation with the leader underwater glider.

[0037] The observation scenario for the pre-set formation can be an underwater glider group performing a navigation mission. The depth-averaged ocean current prediction data, in the inertial coordinate system, includes the velocity components of each coordinate point along different directions, the heading angle deviation, and the distance deviation of the underwater gliders. It can also include data related to the gliding states of the leader and follower underwater gliders during their navigation missions, thus determining their respective state spaces. The action space of the underwater glider at the entry point of the profile is the change in its heading angle, specifically the difference between the heading angle at the entry point at time t and at time t+1. Each time point corresponds to one profile, meaning time t and time t+1 each correspond to one profile. The entry point at time t+1 is also the exit point at time t. Here, t ≥ 1.

[0038] The leadership state at time t-1 can be obtained through communication between the control terminal and the leader underwater glider after it emerges from the water at time t-2, where t ≥ 2. Here, time t-1 can be a time preceding and adjacent to time t, and time t-2 can be a time preceding and one time apart from time t. The leadership action at time t-1 can be the action obtained by inputting the leadership state at time t-1 into the leader agent. Therefore, the leadership state at time t-1 is the actual state of the leader underwater glider, and the leadership action at time t-1 is the actual action of the leader underwater glider.

[0039] Based on the leader's state and action at time t-1, the predicted leader state at time t can be predicted after the leader's underwater glider performs the action at time t-1 based on the leader's state at time t-1. The predicted leader state at time t can be the predicted state of the leader's underwater glider at the water entry point at time t, and the predicted leader action at time t can be predicted based on this state. The predicted leader action at time t can be the action the leader's underwater glider is expected to perform in the predicted leader state at time t, and can be predicted with the goal of the leader's underwater glider reaching its destination.

[0040] The following state at time t can be obtained through communication between the control unit and the follower underwater glider after it emerges from the water at time t-1. Since the follower underwater glider needs to maintain a preset formation with the leader underwater glider, the follower agent corresponding to the follower underwater glider also needs to obtain the state and action of the leader underwater glider when outputting its follower action. However, the leader underwater glider may emerge from the water later than the follower underwater glider. Therefore, the following action at time t can be output based on the predicted leader state and action at time t, and the following state of the follower underwater glider at time t. Even if the leader underwater glider emerges from the water earlier than the follower underwater glider, the following action at time t can still be output based on the predicted leader state and action at time t, and the following state of the follower underwater glider at time t.

[0041] The follow action at time t can be sent to the follower underwater glider, so that at least one follower underwater glider maintains a preset formation with the leader underwater glider according to the follow action at time t.

[0042] According to an embodiment of this application, in an observation scenario with a preset formation of underwater gliders, the leader agent corresponding to the leader underwater glider outputs a leadership action to enable the leader underwater glider to navigate to the destination. The follower agents corresponding to at least one follower underwater glider output a following action to ensure that each follower underwater glider maintains a preset formation with the leader underwater glider. The predicted leadership state of the leader underwater glider at time t is obtained by predicting the actual leadership state and leadership action of the leader underwater glider at time t-1. Based on this predicted leadership state at time t, the predicted leadership state at time t is then used to predict the t-th... The system predicts the leader's actions at all times, so that even if the leader underwater glider has not yet emerged from the water at time t, it can obtain the follow-up actions of the follower underwater gliders at time t based on the predicted leader state, predicted leader actions, and follower state at time t. This ensures that at least one follower underwater glider maintains a preset formation with the leader underwater glider based on its follow-up actions at time t. This reduces the problem of not being able to obtain the leader's state and output actions when the follower underwater glider emerges from the water before the leader underwater glider, ensuring the continuous operation of the underwater glider group and ensuring that the underwater glider group maintains a preset formation while sailing.

[0043] According to an embodiment of this application, in the state space of the leader underwater glider, the velocity component represents the eastward velocity component of the ocean current at each coordinate point in the leader's sea area centered on the leader underwater glider, and the northward velocity component of the ocean current at each coordinate point in the inertial coordinate system. The heading angle deviation of the underwater glider represents the deviation between the current heading angle and the desired heading angle of the leader underwater glider. Both the current heading angle and the desired heading angle of the leader underwater glider are the angles between the fuselage direction and the longitude direction of the leader underwater glider. The desired heading angle of the leader underwater glider represents the target heading angle under the path planning condition of the leader underwater glider. The distance deviation represents the difference between the distance between the current position and the destination of the leader underwater glider and the distance between the starting point and the destination of the leader underwater glider.

[0044] According to an embodiment of this application, in the state space of the follower underwater glider, the velocity component represents the eastward velocity component of the ocean current at each coordinate point in the follower's sea area centered on the follower underwater glider, and the northward velocity component of the ocean current at each coordinate point in the inertial coordinate system. The heading angle deviation of the underwater glider represents the deviation between the current heading angle of the follower underwater glider and the current heading angle of the leader underwater glider. The current heading angle of the follower underwater glider is the angle between the fuselage direction of the follower underwater glider and the longitude direction. The distance deviation represents the difference between the current position of the follower underwater glider and the current position of the leader underwater glider, and the distance between the desired position of the follower underwater glider and the current position of the leader underwater glider.

[0045] In an observation scenario where the underwater glider group is in a preset formation, the states of the leader underwater glider and the follower underwater gliders at each moment can be determined in a pre-constructed state space. The state space can be divided into the state space of the leader underwater glider and the state space of the follower underwater gliders. The state space of the leader underwater glider is expressed as follows (1).

[0046] (1);

[0047] in, This represents the state space of the leader's underwater glider. This represents the depth-average ocean current forecast data in the Leader's Sea Area, centered on the Leader underwater glider. This represents the eastward velocity component of the ocean current at each coordinate point in the inertial coordinate system within the depth-averaged ocean current prediction data. This represents the northward velocity component of the ocean current at each coordinate point in the inertial coordinate system within the depth-meaning ocean current prediction data. This indicates the deviation of the leader's underwater glider's heading angle. This indicates the distance deviation of the leader's underwater glider.

[0048] The depth-meaning ocean current prediction data can be normalized data, representing the average velocity component within the leader sea area centered on the current position of the underwater glider. The leader sea area can be a square sea area, with its four sides parallel to the longitude and latitude directions, respectively. The normalized expression of the depth-meaning ocean current prediction data is as follows (2).

[0049] (2);

[0050] in, This represents the eastward velocity components of ocean currents at various coordinate points in the leader's sea area, centered on the leader's underwater glider, in an inertial coordinate system. This represents the maximum eastward velocity component of the ocean current at each coordinate point in the inertial coordinate system. This represents the minimum eastward velocity component of the ocean current at each coordinate point in the inertial coordinate system. The velocity components of the ocean currents in the leader's sea area, centered on the leader's underwater glider, along the northward direction, are represented in the inertial coordinate system. This represents the maximum northward velocity component of the ocean current at each coordinate point in the inertial coordinate system. This represents the minimum northward velocity component of the ocean current at each coordinate point in the inertial coordinate system.

[0051] The heading angle deviation of the leader's underwater glider can be normalized and expressed as Equation (3).

[0052] (3);

[0053] in, This indicates the leader's desired heading angle for the underwater glider. Let represent the heading angle of the leader's underwater glider at time t, where time t can be the current time, i.e. This indicates the current heading angle of the leader's underwater glider.

[0054] Figure 3 A schematic diagram showing the current heading angle and desired heading angle of a leader underwater glider according to an embodiment of this application is provided.

[0055] like Figure 3 As shown, the origin of the coordinate system represents the current position of the leader's underwater glider, and the desired heading angle of the leader's underwater glider is... The current heading angle is In this application's embodiments, the heading angles are all the angles between the underwater glider's fuselage direction and the X-axis of the inertial coordinate system. The inertial coordinate system... An axis can represent the direction of longitude.

[0056] The distance deviation of the leader's underwater glider can be an indicator of the distance between the current position of the leader's underwater glider and the mission endpoint, and can be expressed as the following formula (4).

[0057] (4);

[0058] in, This indicates the distance between the leader's current underwater glider position and the mission endpoint. It indicates the distance between the starting point and the ending point of the leader's underwater glider mission.

[0059] There can be N follower underwater gliders, where N is an integer greater than or equal to 1. The following uses the nth follower underwater glider as an example to illustrate the embodiment of this application, where n is an integer greater than or equal to 1. The state space of the nth follower underwater glider is represented by the following formula (5).

[0060] (5);

[0061] in, Let represent the state space of the nth follower underwater glider. This represents the velocity components along different directions at each coordinate point in the inertial coordinate system based on the depth-averaged ocean current prediction data of the nth follower underwater glider. This represents the heading angle deviation of the nth follower underwater glider. This represents the heading angle deviation of the nth follower underwater glider. The depth-mean ocean current prediction data for the follower underwater glider can also be normalized data from the following sea area centered on the follower underwater glider. The normalization method can refer to the depth-mean ocean current prediction data based on the leader underwater glider, which will not be elaborated here.

[0062] The heading angle deviation of the nth follower underwater glider can be normalized and expressed as Equation (6).

[0063] (6);

[0064] in, This represents the heading angle of the nth follower underwater glider at time t, which is also the current heading angle of the nth follower underwater glider. This indicates the current heading angle of the leader's underwater glider.

[0065] The heading angle deviation of the follower underwater glider is an indicator of the distance between the follower underwater glider and the leader underwater glider, and can be expressed as the following formula (7).

[0066] (7);

[0067] in, This represents the heading angle deviation of the nth follower underwater glider. This represents the distance between the current position of the nth follower's underwater glider and the current position of the leader's underwater glider. This represents the distance between the desired position of the nth follower's underwater glider and the current position of the leader's underwater glider.

[0068] Figure 4 A schematic diagram showing the current heading angle and desired heading angle of a follower underwater glider according to an embodiment of this application is provided.

[0069] like Figure 4 As shown, the current heading angle of the follower underwater glider is... The leader's underwater glider is currently on the following course. The distance between the current position of the nth follower's underwater glider and the current position of the leader's underwater glider is... The distance between the desired position of the nth follower underwater glider and the current position of the leader underwater glider is . .

[0070] The actions of the leader and follower underwater gliders at each moment can be determined in a pre-constructed action space, which represents the heading angle adjustment of the underwater gliders.

[0071] The action space can be represented as shown in equation (8).

[0072] (8);

[0073] Where A represents the action space, This represents the action of the underwater glider at time t. , Let represent the heading angle of the underwater glider at time t. This represents the heading angle of the underwater glider at time t+1.

[0074] Furthermore, reward functions for underwater gliders can be constructed. The reward functions for the leader underwater glider and the follower underwater glider are expressed as equations (9) and (10) respectively.

[0075] (9);

[0076] (10);

[0077] in, This represents the reward function for the leader's underwater glider. Let represent the reward function for the nth follower underwater glider. In equation (9), the distance from the current position of the leader underwater glider to the mission endpoint ≤ 2km can be determined if the leader underwater glider does not collide with the obstacle. Since the end of the navigation mission is determined based on the distance between the leader underwater glider and the mission endpoint, in order to maintain the position of the follower underwater glider, the reward value of the reward function can be 0 if the distance from the current position of the follower underwater glider to the desired position ≤ 1km.

[0078] According to an embodiment of this application, the underwater glider formation control method based on multi-agent reinforcement learning may further include: outputting a leadership action at time t through a leader agent based on the leadership state of the leader underwater glider at time t; and sending the leadership action at time t to the leader underwater glider so that the leader underwater glider sails to the destination according to the leadership action at time t.

[0079] Upon receiving the leader's underwater glider's leadership state at time t from the control terminal, the leader agent can output the leader's action at time t and send the leader's action at time t to the leader's underwater glider, so that the leader's underwater glider can navigate to the destination according to the leader's action at time t.

[0080] According to embodiments of this application, the underwater glider formation control method based on multi-agent reinforcement learning may further include: outputting the following action at time t for each of at least one follower underwater glider through a follower agent based on the leader state, the leader action at time t, and the follower state at time t; and sending the following action at time t to at least one follower underwater glider so that each of the at least one follower underwater glider maintains a preset formation with the leader underwater glider based on the following action at time t.

[0081] When the control terminal receives the leader's underwater glider's leadership state at time t, it can directly output the follower action at time t for each of the at least one follower underwater gliders based on the leader's leadership state, leader action, and follower state at time t, without predicting the leader's underwater glider's state at time t. The follower action at time t is then sent to the at least one follower underwater glider so that each of the at least one follower underwater glider maintains a preset formation with the leader underwater glider based on its follower action at time t.

[0082] According to an embodiment of this application, when the leader's underwater glider's state at time t is received at time t, the follower's underwater glider's action output can be directly based on the leader's underwater glider's actual state, thereby improving the accuracy of the follower's underwater glider's actions.

[0083] According to embodiments of this application, the underwater glider formation control method based on multi-agent reinforcement learning may further include: inputting the navigation task of the underwater glider group, a preset formation, and the number of underwater gliders in the underwater glider group; and determining, based on the navigation task of the underwater glider group, the preset formation, and the number of underwater gliders in the underwater glider group, a communication topology between a leader underwater glider and at least one follower underwater glider, and between the leader underwater glider and at least one follower underwater glider, so that the control terminal communicates with the leader underwater glider and at least one follower underwater glider respectively based on the communication topology.

[0084] The navigation mission can be an underwater observation mission targeting a specific area in the ocean. The preset formation can be generated based on the number of underwater gliders in the underwater glider group.

[0085] Based on the underwater glider group's navigation mission, preset formation, and the number of underwater gliders in the group, the underwater glider located at the center of the preset formation can be designated as the leader underwater glider, and the other underwater gliders as follower underwater gliders. A leader agent corresponding to the leader underwater glider and follower agents corresponding to the follower underwater gliders can be deployed at the control terminal. Communication between the control terminal and the leader underwater glider is conducted through the leader agent, and communication between the control terminal and at least one follower underwater glider is conducted through the follower agent.

[0086] According to an embodiment of this application, by inputting the navigation mission, preset formation and number of underwater gliders, the communication topology between the leader underwater glider and at least one follower underwater glider, as well as between the leader underwater glider and at least one follower underwater glider, can be determined, thereby enabling the control terminal to establish communication with the leader underwater glider and at least one follower underwater glider.

[0087] According to an embodiment of this application, based on the leader's underwater glider's leadership state at time t-1 received by the leader agent and the leader's action at time t-1 output, the leader's predicted leadership state at time t of the follower underwater glider is predicted. This includes: processing the leadership state and action at time t-1 using input format to obtain the leadership state / action input features at time t-1; performing fully connected processing on the leadership state / action input features at time t-1 to obtain the leadership state / action fully connected features at time t-1; and processing the output dimension of the leadership state / action fully connected features at time t-1 to obtain the leader's predicted leadership state at time t.

[0088] A leader state prediction network can be constructed to obtain the predicted leader state at time t. The leader state prediction network can be a fully connected neural network. The input of the leader state prediction network is the leader state at time t-1 of the previous time profile of the underwater glider and the leader action at time t-1 of the previous time profile. The output of the leader state prediction network is the predicted leader state at time t.

[0089] The leader state prediction network can consist of an input layer, a hidden layer, and an output layer connected sequentially. The input layer has 4 neurons, the hidden layer has 256 neurons, and the output layer has 2 neurons. The leader's state and action at time t-1 are input to the input layer for input format processing to obtain the leader's state-action input features at time t-1. These features are then input to the hidden layer for fully connected processing to obtain the leader's state-action fully connected features at time t-1. Finally, these features are input to the output layer for output dimension processing to obtain the predicted leader state at time t.

[0090] According to an embodiment of this application, based on the leader prediction state at time t, a leader prediction action at time t that enables the underwater glider to reach its destination is predicted is obtained, including: processing the leader prediction state at time t using its input format to obtain the leader prediction state input features at time t; performing fully connected processing on the leader prediction state input features at time t to obtain the leader state fully connected features at time t; and processing the leader state fully connected features at time t using its output dimension to obtain the leader prediction action at time t.

[0091] A leader action prediction network can be constructed to obtain the predicted leader action at time t. The leader action prediction network is a fully connected neural network, and the input to the leader action prediction network is the predicted leader state at time t. The output of the leader action prediction network is the predicted leader state at time t. The leader action prediction network can include an input layer, a hidden layer, and an output layer connected sequentially. The input layer has 4 neurons, the hidden layer has 256 neurons, and the output layer has 2 neurons. The predicted leader state at time t is input to the input layer for input format processing to obtain the predicted leader state input features at time t. The predicted leader state input features at time t are then input to the hidden layer for fully connected processing to obtain the fully connected leader state features at time t. Finally, the fully connected leader state features at time t are input to the output layer for output dimension processing to obtain the predicted leader action at time t.

[0092] The system can predict the state of the leader's underwater glider based on a leader state prediction network, predict its actions based on a leader action prediction network, and output leadership actions from the leader agent and follower actions from the follower agents. Specifically, the leader agent can output leadership actions through a leadership policy network, and the follower agents can output follower actions through a follower policy network. The leadership policy network can be trained based on a leader evaluation network, and the follower policy network can be trained based on a follower evaluation network.

[0093] The leadership strategy network can employ a fully connected neural network, comprising: a sequentially connected input layer, two hidden layers, and an output layer. The activation function from the input layer to the hidden layers can be a rectified linear unit (ReLU) function, and the activation function from the hidden layers to the output layer can be a hyperbolic tangent function (Tanh). The input layer has 2 neurons, both hidden layers have 256 neurons each, and the output layer has 1 neuron.

[0094] The leadership evaluation network can use a fully connected neural network, consisting of an input layer, two hidden layers, and an output layer connected sequentially. The activation function from the input layer to the hidden layer can be the ReLU function, and there is no activation function from the hidden layer to the output layer. The input layer can have 3 neurons, the two hidden layers each have 256 neurons, and the output layer has 1 neuron.

[0095] The follower policy network can be a fully connected neural network, consisting of an input layer, two hidden layers, and an output layer connected sequentially. The activation function from the input layer to the hidden layer is the ReLU function, and the activation function from the hidden layer to the output layer is the Tanh function. The input layer has 5 neurons, the two hidden layers each have 256 neurons, and the output layer has 1 neuron.

[0096] The evaluation network can be a fully connected neural network, consisting of an input layer, two hidden layers, and an output layer connected sequentially. The activation function from the input layer to the hidden layer is the ReLU function, and there is no activation function from the hidden layer to the output layer. The input layer has 6 neurons, the two hidden layers each have 256 neurons, and the output layer has 1 neuron.

[0097] The input to the leadership strategy network is the state of the leader's underwater glider, and the output is the leader's underwater glider's actions. The input to the leadership evaluation network is the state and actions of the leader's underwater glider, and the output is the evaluation value of the leader's underwater glider's leadership actions.

[0098] The input to the follower policy network is the state of the follower underwater glider, the state of the leader underwater glider, and the action of the leader underwater glider. The output is the action of the follower underwater glider. The state and action of the leader underwater glider can be provided by the leader agent, or by the leader state prediction network and the leader action prediction network, respectively.

[0099] The input to the leader evaluation network includes the state of the follower's underwater glider, the action of the follower's underwater glider, the state of the leader's underwater glider, and the action of the leader's underwater glider. The output is the evaluation value of the follower's underwater glider's following action.

[0100] First, training samples can be obtained, and then the leader state prediction network, leader action prediction network, leadership strategy network, and follower strategy network can be trained based on these samples. The training samples are obtained as follows.

[0101] The underwater gliders can be configured based on their formation, number, and mission objectives. A leader glider and at least one follower glider are determined based on the formation, with the leader glider's initial position serving as the mission starting point. The control unit can receive the leader glider's moment-1 leadership status. The leader's actions at time 1 were calculated based on the leader's leadership state in the underwater glider at time 1. The control unit receives the first-time sample of the following state of the nth follower underwater glider. Based on the leader state, leader action, and follower state at time 1, the follower underwater glider's follower action at time 1 is calculated. The leader underwater glider and the follower underwater glider each perform their corresponding actions, and the leader's underwater glider obtains the second-time sample leadership state. The second time-following state of the underwater glider and its follower And the first-moment leadership reward corresponding to the leader's underwater glider. The follower reward at moment 1 corresponding to the nth follower's underwater glider. .

[0102] The leader's underwater glider's time-1 sample leadership state, time-1 sample leadership action, time-2 sample leadership state, and corresponding reward are used as one sample. Similarly, the follower's underwater glider's time-1 sample following state, time-1 sample following action, time-2 sample following state, and time-1 corresponding reward are used as one sample. This process is repeated until a sufficient number of training samples are obtained, for example, 5,000,000. The sample corresponding to time t in the sample data can be represented as... ,in, Let represent the sample leadership state of the leader underwater glider and the sample following states of the 1st, 2nd, ..., Nth follower underwater gliders at time t, respectively. Let represent the sample leadership actions of the leader's underwater glider at time t, and the sample following actions of the 1st, 2nd, ..., Nth follower's underwater gliders, respectively. Let represent the sample leadership state of the leader underwater glider at time t+1, and the sample following states of the 1st, 2nd, ..., Nth follower underwater gliders, respectively. Let t represent the sample leadership reward of the leader's underwater glider and the sample following reward of the 1st, 2nd, ..., Nth follower's underwater glider, respectively.

[0103] According to an embodiment of this application, the leader agent outputs the leadership actions of the leader's underwater glider through a leadership policy network. The leadership policy network is trained through the following operations: based on the sample leader's underwater glider's leadership state at time t+1 and time t, the initial leadership policy network is used to obtain the leader's output action at time t+1 and time t, respectively; based on the sample leader's underwater glider's leadership state and action at time t, the initial leadership evaluation network is used to obtain the leader's evaluation value at time t; based on the sample leader's leadership state and action at time t, the initial leadership evaluation network is used to obtain the leader's output evaluation value at time t; based on the sample leader's leadership state and action at time t+1, the initial target leadership evaluation network is used to obtain the leader's predicted evaluation value at time t+1; based on the sample leader's evaluation value and action at time t+1, the parameters of the initial leadership evaluation network are updated to obtain the leadership evaluation network; based on the sample leader's output evaluation value at time t, the parameters of the initial leadership policy network are updated to obtain the leadership policy network.

[0104] The leader's underwater glider sample at time t+1 can be used to determine the leader's state. and the leadership state of the sample at time t Input the initial leadership policy network into the network, and output the leadership action at time t+1. And the leader's output action at time t of the sample The leader's leadership state at time t is sampled from the underwater glider. And the leader's actions at time t of the sample Input the initial leadership evaluation network to obtain the leadership evaluation value of the sample at time t. The leader evaluation value at time t can characterize the reward for the underwater glider of the sample leader at time t to perform the leader action at time t based on the leader state at time t.

[0105] The leadership status at time t+1 of the sample And the leader's output action at time t+1 of the sample Input the initial target leadership evaluation network to obtain the predicted leadership evaluation value of the sample at time t+1. The initial target leadership evaluation network can be a network with the same network structure as the initial leadership evaluation network. The predicted leadership evaluation value at time t+1 can characterize the predicted reward of the underwater glider of the sample leader at time t+1, based on the leadership state at time t+1, when it performs the output action of the leader at time t+1. The loss function of the leadership evaluation network can be constructed based on the leadership evaluation value at time t and the predicted leadership evaluation value at time t+1. It can be constructed based on the temporal difference method and expressed as the following equation (11).

[0106] (11);

[0107] in, This represents the loss function of the leader evaluation network. Expressing expectations, This represents the parameters of the initial leadership evaluation network. This represents the reward for the leader's underwater glider performing the leadership action at time t in the sample. For the leader decay factor, This represents the leader's entropy temperature coefficient. This represents the leadership strategy network. In this embodiment, .

[0108] The loss value of the initial leadership evaluation network is calculated based on equation (11), and the parameters of the initial leadership evaluation network are updated through backpropagation to obtain the leadership evaluation network, which is expressed as equation (12).

[0109] (12);

[0110] in, The parameters representing the leadership's evaluation of the network, This represents the parameters of the initial leadership evaluation network. This indicates the leader's evaluation of the network's learning rate. This represents the gradient of the loss function of the leader evaluation network.

[0111] The leadership state of the sample at time t can be And the leader's output action at time t of the sample Input the initial leadership evaluation network to obtain the leadership output evaluation value at time t. Based on the leader's output evaluation value at time t of the sample, the loss function of the leadership strategy network is constructed, and the loss function of the leadership strategy network is expressed as the following equation (13).

[0112] (13);

[0113] in, This represents the loss function of the leadership policy network. This represents the parameters of the initial leadership strategy network.

[0114] The loss value of the initial leadership strategy network is calculated based on equation (13), and the parameters of the initial leadership strategy network are updated through backpropagation to obtain the leadership strategy network, which is expressed as equation (14).

[0115] (14);

[0116] in, The parameters represent the leadership strategy network. The parameters represent the initial leadership strategy network. This represents the learning rate of the leadership strategy network. This represents the gradient of the loss function of the leadership policy network.

[0117] According to an embodiment of this application, the follower agent outputs the following action of the follower underwater glider through a following policy network. The following policy network is trained as follows: based on the leader's underwater glider's sample leadership state at time t+1, the leader's output action at time t+1, and the follower's underwater glider's sample following state at time t+1, the following output action at time t is obtained through an initial following policy network; based on the follower's underwater glider's sample following state at time t and the following action at time t, the following action at time t is obtained through an initial following evaluation network. Evaluation values ​​are calculated as follows: Based on the following state and following output action at time t, the following output evaluation value at time t is obtained through the initial following evaluation network; Based on the following state and following output action at time t+1, the following prediction evaluation value at time t+1 is obtained through the initial target following evaluation network; Based on the following evaluation value and following prediction evaluation value at time t, the parameters of the initial following evaluation network are updated to obtain the following evaluation network; Based on the following output evaluation value at time t, the parameters of the initial following strategy network are updated to obtain the following strategy network.

[0118] Taking the nth sample follower underwater glider as an example, the leadership state of the sample leader underwater glider at time t+1 can be obtained. The leader's output action at time t+1 of the sample The sample following the underwater glider at time t+1 is the following state. The input is fed into the initial follower policy network to obtain the follower output action of the sample at time t+1. The sample follower underwater glider's sample follow state at time t. and the sample at time t to follow the action Input the initial follower evaluation network to obtain the follower evaluation value of the sample at time t. The follow evaluation value at time t can characterize the reward of the underwater glider that performs the follow action at time t based on the follow state at time t.

[0119] The sample will follow the state at time t+1. The sample follows the output action at time t+1. Input the initial target follower evaluation network to obtain the follower prediction evaluation value of the sample at time t+1. The initial target follow evaluation network can be a network with the same network structure as the initial follow evaluation network. The follow prediction evaluation value at time t+1 can characterize the predicted reward of the underwater glider of the sample follower at time t+1 based on the follow state at time t+1. The loss function of the follow evaluation network can be constructed based on the follow evaluation value at time t and the follow prediction evaluation value at time t+1. It can be constructed based on the temporal difference method and expressed as the following formula (15).

[0120] (15);

[0121] in, This represents the loss function of the follow evaluation network. This represents the parameters of the initial follow-up evaluation network. This represents the reward for the underwater glider that performs the following action at time t of the sample. For follower decay factor, This represents the temperature coefficient of the follower entropy. This represents a follow-the-policy network. In this embodiment, =0.99.

[0122] The loss value of the initial follower evaluation network is calculated based on equation (15), and the parameters of the initial follower evaluation network are updated through backpropagation to obtain the follower evaluation network, which is expressed as equation (16).

[0123] (16);

[0124] in, The parameters on the right side of the equation represent the parameters of the evaluation network. This represents the parameters of the initial follow-up evaluation network. This indicates the learning rate of the evaluation network. This represents the gradient of the loss function that follows the evaluation network.

[0125] The sample can follow the state at time t. The sample follows the output action at time t. Input the initial follower evaluation network to obtain the follower output evaluation value of the sample at time t. Based on the sample's output evaluation value at time t, a loss function for the following policy network is constructed, which is expressed as equation (17).

[0126] (17);

[0127] in, This represents the loss function of the follow-policy network. This represents the parameters of the initial follow-the-policy network.

[0128] The loss value of the initial follower policy network is calculated based on equation (17), and the parameters of the initial follower policy network are updated through backpropagation to obtain the follower policy network, which is expressed as equation (18).

[0129] (18);

[0130] in, These represent the parameters of the follow-policy network. These represent the parameters of the initial follow-the-policy network. This represents the learning rate of the policy-following network. This represents the gradient of the loss function of the follower policy network.

[0131] The loss function of the leader state prediction network can be constructed based on the leader state and leader action at time t-1 of the sample underwater glider, as shown in Equation (19).

[0132] (19);

[0133] in, This represents the loss function of the leader state prediction network. This represents the predicted state output by the leader state prediction network at time t of the sample. This represents the network parameters of the leader state prediction network.

[0134] The loss function of the leader action prediction network can be constructed based on the leader prediction action at time t of the sample, as shown in equation (20).

[0135] (20);

[0136] in, This represents the loss function of the leader action prediction network. This indicates that the leader action prediction network predicts the leader's state based on the sample at time t. The predicted action of the leader at time t is given. This represents the parameters of the leader action prediction network.

[0137] The loss value of the leader state prediction network is determined based on the loss function of the leader state prediction network, and the parameters of the leader state prediction network are updated through backpropagation, as shown in Equation (21).

[0138] (twenty one);

[0139] in, This represents the parameters of the updated leader state prediction network. This represents the learning rate of the leader state prediction network. This represents the gradient of the loss function of the leader state prediction network.

[0140] The loss value of the leader action prediction network is determined based on the loss function of the leader action prediction network, and the parameters of the leader action prediction network are updated through backpropagation, as shown in Equation (22).

[0141] (twenty two);

[0142] in, This represents the parameters of the updated leader action prediction network. This represents the learning rate of the leader action prediction network. This represents the gradient of the loss function of the leader action prediction network.

[0143] Figure 5 A flowchart of an underwater glider formation control method based on multi-agent reinforcement learning according to another embodiment of this application is shown.

[0144] like Figure 5 As shown, the method of this embodiment includes operations S501 to S511.

[0145] In S501, input the navigation mission of the underwater glider group, the preset formation, and the number of underwater gliders in the underwater glider group.

[0146] In operation S502, based on the underwater glider group's navigation mission, preset formation, and the number of underwater gliders in the underwater glider group, the leader underwater glider and at least one follower underwater glider, as well as the communication topology between the leader underwater glider and at least one follower underwater glider, are determined from the underwater gliders.

[0147] When operating S503, based on the leader's state at time t of the underwater glider, the leader agent outputs the leader's action at time t.

[0148] In operation S504, determine whether the follower agent has received the leader state and leader action at time t. If yes, execute operation S508; otherwise, execute operation S505.

[0149] In operation S505, based on the leader's underwater glider's leadership state at time t-1 received by the leader agent and the leader's action at time t-1 output, the leader's underwater glider's predicted leadership state at time t is predicted.

[0150] In operation S506, based on the leader's predicted state at time t, the leader's predicted action at time t that enables the underwater glider to reach the destination is predicted.

[0151] In operation S507, based on the leader's predicted state at time t, the leader's predicted action at time t, and the follower's state at time t received by the follower agent, the follower agent outputs the follower action at time t for each of the follower underwater gliders.

[0152] In operation S508, based on the leader state at time t, the leader action at time t, and the follower state at time t, the follower agent outputs the follower action at time t for each of the follower underwater gliders.

[0153] In operation S509, a leader action at time t is sent to the leader underwater glider so that the leader underwater glider sails to the destination according to the leader action at time t, and a follow action at time t is sent to at least one follower underwater glider so that each of the at least one follower underwater glider maintains a preset formation with the leader underwater glider according to the follow action at time t.

[0154] In operation S510, determine whether the distance from the current position of the leader's underwater glider to the mission endpoint is less than a preset distance value. If yes, execute operation S511; otherwise, execute operation S503.

[0155] The task was terminated while operating S511.

[0156] Operations S501 to S511 can be referred to the description of other embodiments of this application, wherein operation S506 can also be executed if it is determined that the follower agent has received the leader state and the leader action at time t. As an example, the preset distance value can be set to 2km.

[0157] Figure 6 A schematic diagram of an underwater glider unit in a preset formation according to an embodiment of this application is shown.

[0158] like Figure 6As shown, in a real marine environment, the underwater glider formation control method based on multi-agent reinforcement learning, as described in this application, controls an underwater glider group to perform navigation tasks. The underwater glider group includes four underwater gliders, namely G1, G2, G3, and G4. The preset formation is a quadrilateral. Underwater glider G3 is determined as the leader underwater glider, and underwater gliders G1, G2, and G4 are determined as follower underwater gliders. The distances between underwater gliders G1 and G3, G1 and G4, G2 and G3, and G2 and G4 are all set to 30 km, while the distance between underwater gliders G3 and G4 is set to 40 km.

[0159] like Figure 6 As shown, during the navigation mission, the underwater glider group conducted navigation through 20 profiles (the position of each colored dot represents the entry point of the underwater glider in a profile), which means 20 descents and ascents. The average navigation deviation between underwater gliders G2 and G4 was 0.23 km, between G2 and G3 it was 0.32 km, and between G3 and G4 it was 0.70 km. Due to strong ocean currents in the area where underwater glider G1 was located, the average navigation deviation between G1 and G3 was excessively high at 2.95 km, and the average navigation deviation between G1 and G4 was 0.80 km. It can be seen that the underwater glider formation control method based on multi-agent reinforcement learning, as described in this application, can maintain the preset formation when controlling the underwater glider group to perform navigation missions.

[0160] Figure 7 A block diagram of an underwater glider formation control device based on multi-agent reinforcement learning according to an embodiment of this application is shown.

[0161] like Figure 7As shown, the underwater glider formation control device 700 based on multi-agent reinforcement learning is applied to a control terminal capable of communicating with an underwater glider group. The underwater glider group includes a leader underwater glider with a preset formation and at least one follower underwater glider. The control terminal is equipped with a leader agent corresponding to the leader underwater glider and follower agents corresponding to each of the at least one follower underwater glider. The leader agent is used to output a leadership action that enables the leader underwater glider to sail to the destination. The follower agent is used to output a following action that enables each of the at least one follower underwater glider to maintain a preset formation with the leader underwater glider. The underwater glider formation control device 700 based on multi-agent reinforcement learning includes a determination module 710, a first prediction module 720, a second prediction module 730, an output module 740, and a transmission module 750.

[0162] The determination module 710 is used to determine the state space of the leader underwater glider and at least one follower underwater glider in an observation scenario where the underwater glider group is in a preset formation. Based on the velocity components of each coordinate point in the inertial coordinate system along different directions, the heading angle deviation and distance deviation of the underwater gliders, according to the depth-averaged ocean current prediction data, the state space of the leader underwater glider and at least one follower underwater glider. The module also determines the action space of all underwater gliders in the underwater glider group based on the heading angle of the underwater gliders at the water entry point at time t and at time t+1, where t≥1.

[0163] The first prediction module 720 is used to predict the leader prediction state of the leader underwater glider at time t based on the leader state of the leader underwater glider at time t-1 received by the leader agent and the leader action at time t-1 output.

[0164] The second prediction module 730 is used to predict the leader's predicted action at time t, which will enable the leader's underwater glider to reach the destination, based on the leader's predicted state at time t.

[0165] The output module 740 is used to output the following actions of at least one follower underwater glider at time t based on the leader prediction state at time t, the leader prediction action at time t, and the follower state received by the follower agent at time t.

[0166] The sending module 750 is used to send a follow action at time t to at least one follower underwater glider, so that each of the at least one follower underwater glider maintains a preset formation with the leader underwater glider according to the follow action at time t.

[0167] Any one or more of the modules, submodules, units, and subunits according to the embodiments of this application, or at least part of the functions of any one or more of them, can be implemented in one module. Any one or more of the modules, submodules, units, and subunits according to the embodiments of this application can be implemented by dividing them into multiple modules. Any one or more of the modules, submodules, units, and subunits according to the embodiments of this application can be at least partially implemented as hardware circuits, such as field-programmable gate arrays (FPGAs), programmable logic arrays (PLAs), systems-on-a-chip, systems-on-a-substrate, systems-on-package, application-specific integrated circuits (ASICs), or implemented by hardware or firmware in any other reasonable manner by integrating or packaging circuits, or implemented in any one of software, hardware, and firmware, or in a suitable combination of any of these. Alternatively, one or more of the modules, submodules, units, and subunits according to the embodiments of this application can be at least partially implemented as computer program modules, which, when run, can perform corresponding functions.

[0168] For example, any plurality of the determining module 710, the first prediction module 720, the second prediction module 730, the output module 740, and the transmitting module 750 can be combined into one module / unit / subunit, or any one of these modules / units / subunits can be split into multiple modules / units / subunits. Alternatively, at least part of the functionality of one or more of these modules / units / subunits can be combined with at least part of the functionality of other modules / units / subunits and implemented in one module / unit / subunit. According to embodiments of this application, at least one of the determining module 710, the first prediction module 720, the second prediction module 730, the output module 740, and the transmitting module 750 can be at least partially implemented as hardware circuitry, such as a field-programmable gate array (FPGA), a programmable logic array (PLA), a system-on-a-chip, a system-on-a-substrate, a system-on-package, an application-specific integrated circuit (ASIC), or any other reasonable means of integrating or packaging the circuitry, or implemented in software, hardware, or firmware, or in any suitable combination of any of these three implementation methods. Alternatively, at least one of the determining module 710, the first prediction module 720, the second prediction module 730, the output module 740, and the sending module 750 may be implemented at least partially as a computer program module, which can perform corresponding functions when the computer program module is run.

[0169] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions. Those skilled in the art will understand that the features described in the various embodiments of this application can be combined and / or combined in various ways, even if such combinations are not explicitly described in this application. In particular, without departing from the spirit and teachings of this application, the features described in the various embodiments of this application can be combined and / or combined in various ways. All such combinations and / or combinations fall within the scope of this application.

[0170] The embodiments of this application have been described above. However, these embodiments are merely illustrative and not intended to limit the scope of this application. Although various embodiments have been described above, this does not mean that the measures in the various embodiments cannot be used advantageously in combination. Without departing from the scope of this application, those skilled in the art can make various substitutions and modifications, all of which should fall within the scope of this application.

Claims

1. A method for underwater glider formation control based on multi-agent reinforcement learning, characterized in that, A control terminal capable of communicating with an underwater glider group, the underwater glider group comprising a leader underwater glider with a preset formation and at least one follower underwater glider, the control terminal being deployed with a leader agent corresponding to the leader underwater glider and follower agents corresponding to each of the at least one follower underwater glider, the leader agent outputting a leadership action to enable the leader underwater glider to navigate to its destination, and the follower agents outputting a following action to enable each of the at least one follower underwater glider to maintain the preset formation with the leader underwater glider, the method comprising: In the observation scenario where the underwater glider group is in the preset formation, based on the velocity components of each coordinate point in the inertial coordinate system along different directions, the heading angle deviation and distance deviation of the underwater gliders, the state spaces of the leader underwater glider and at least one follower underwater glider are determined. Furthermore, based on the heading angle of the underwater gliders in the underwater glider group at the water entry point of the profile at time t and the heading angle of the water entry point of the profile at time t+1, the action space of all underwater gliders in the underwater glider group is determined, where t≥1. Based on the leader's underwater glider's leadership state at time t-1 received by the leader agent and the leader's action at time t-1 output, the leader's underwater glider's predicted leadership state at time t is predicted. Based on the leader prediction state at time t, the leader prediction action at time t that enables the underwater glider to reach the destination is predicted. Based on the leader prediction state at time t, the leader prediction action at time t, and the follower agent's follow state at time t, the follower agent outputs at least one follow action at time t for each of the follower underwater gliders. Send the follow action at time t to at least one of the follower underwater gliders so that each of the at least one follower underwater glider maintains the preset formation with the leader underwater glider according to the follow action at time t.

2. The method according to claim 1, characterized in that, In the state space of the leader underwater glider, the velocity component represents the eastward velocity component of the ocean current at each coordinate point in the leader's sea area centered on the leader underwater glider, and the northward velocity component of the ocean current at each coordinate point in the inertial coordinate system. The heading angle deviation of the underwater glider represents the deviation between the current heading angle and the desired heading angle of the leader underwater glider. Both the current heading angle and the desired heading angle of the leader underwater glider are the angles between the fuselage direction and the longitude direction of the leader underwater glider. The desired heading angle of the leader underwater glider represents the target heading angle under the path planning condition of the leader underwater glider. The distance deviation represents the relationship between the distance between the current position and the destination of the leader underwater glider and the distance between the starting point and the destination of the leader underwater glider.

3. The method according to claim 1, characterized in that, In the state space of the follower underwater glider, the velocity component represents the eastward velocity component of the ocean current at each coordinate point in the follower's sea area centered on the follower underwater glider, and the northward velocity component of the ocean current at each coordinate point in the inertial coordinate system. The heading angle deviation of the underwater glider represents the deviation between the current heading angle of the follower underwater glider and the current heading angle of the leader underwater glider. The current heading angle of the follower underwater glider is the angle between the fuselage direction and the longitude direction of the follower underwater glider. The distance deviation represents the relationship between the current position of the follower underwater glider and the current position of the leader underwater glider, and the distance between the desired position of the follower underwater glider and the current position of the leader underwater glider.

4. The method according to any one of claims 1 to 3, characterized in that, The method further includes: Based on the leadership state of the leader's underwater glider at time t, the leader agent outputs the leadership action at time t. Send the leadership action at time t to the leader underwater glider so that the leader underwater glider navigates to the destination according to the leadership action at time t.

5. The method according to claim 4, characterized in that, The method further includes: Based on the leadership state at time t, the leadership action at time t, and the follower state at time t, the follower agent outputs at least one follower action at time t for each of the follower underwater gliders; Send the follow action at time t to at least one of the follower underwater gliders so that each of the at least one follower underwater glider maintains the preset formation with the leader underwater glider according to the follow action at time t.

6. The method according to any one of claims 1 to 3, characterized in that, The method further includes: Input the navigation mission of the underwater glider group, the preset formation, and the number of underwater gliders in the underwater glider group; Based on the navigation mission of the underwater glider group, the preset formation, and the number of underwater gliders in the underwater glider group, a communication topology is determined from the underwater gliders, including the leader underwater glider and at least one follower underwater glider, as well as the communication topology between the leader underwater glider and at least one follower underwater glider, so that the control terminal can communicate with the leader underwater glider and at least one follower underwater glider respectively based on the communication topology.

7. The method according to any one of claims 1 to 3, characterized in that, The step of predicting the predicted leadership state of the follower's underwater glider at time t based on the leadership state of the leader's underwater glider at time t-1 received by the leader agent and the leadership action at time t-1 output by the leader agent includes: The leadership state and leadership action at time t-1 are processed to obtain the leadership state and action input features at time t-1. Perform fully connected processing on the leadership state action input features at time t-1 to obtain the fully connected features of the leadership state action at time t-1; The output dimension of the fully connected features of the leader's state action at time t-1 is processed to obtain the predicted leader state at time t.

8. The method according to any one of claims 1 to 3, characterized in that, The step of predicting the leader's predicted action at time t, based on the leader's predicted state at time t, to enable the leader's underwater glider to reach its destination, includes: The leadership prediction state at time t is processed to obtain the leadership prediction state input features at time t. The fully connected features of the leadership prediction state at time t are processed to obtain the fully connected features of the leadership state at time t. The output dimension of the fully connected features of the leadership state at time t is processed to obtain the predicted action of the leadership at time t.

9. The method according to any one of claims 1 to 3, characterized in that, The leader agent outputs the leadership actions of the underwater glider through a leadership policy network, which is trained through the following operations: Based on the leader's underwater glider state at time t+1 and time t, the leader's output action at time t+1 and time t is obtained through the initial leadership policy network. Based on the leadership state and leadership action of the underwater glider of the sample leader at time t, the leadership evaluation value at time t is obtained through the initial leadership evaluation network. Based on the leadership state and the leadership output action at time t of the sample, the leadership output evaluation value at time t of the sample is obtained through the initial leadership evaluation network. Based on the leader's state at time t+1 and the leader's output action at time t+1, the predicted evaluation value of the leader at time t+1 is obtained through the initial target leader evaluation network, wherein the initial target leader evaluation network has the same network structure as the initial leader evaluation network. Based on the leadership evaluation value at time t and the predicted leadership evaluation value at time t+1, the parameters of the initial leadership evaluation network are updated to obtain the leadership evaluation network. Based on the leader's output evaluation value at time t of the sample, the parameters of the initial leadership strategy network are updated to obtain the leadership strategy network.

10. The method according to any one of claims 1 to 3, characterized in that, The follower agent outputs the following actions of the underwater glider through a follower policy network, which is trained through the following operations: Based on the leader's underwater glider's state at time t+1, the leader's output action at time t+1, and the follower's underwater glider's state at time t+1, the follower's output action at time t+1 is obtained through the initial follower policy network. Based on the following state and following action of the underwater glider at time t, the following evaluation value at time t is obtained through the initial following evaluation network. Based on the following state and the following output action of the sample at time t, the following output evaluation value of the sample at time t is obtained through the initial following evaluation network; Based on the following state and following output action of the sample at time t+1, the following prediction evaluation value of the sample at time t+1 is obtained through the initial target following evaluation network, wherein the initial target following evaluation network has the same network structure as the initial following evaluation network. Based on the following evaluation value of the sample at time t and the following prediction evaluation value of the sample at time t, the parameters of the initial following evaluation network are updated to obtain the following evaluation network; Based on the evaluation value of the sample at time t, the parameters of the initial following strategy network are updated to obtain the following strategy network.

Images