A ship swarm formation control method based on collaborative exploration deep reinforcement learning

Abstract

The present invention discloses a ship cluster formation control method based on collaborative exploration deep reinforcement learning, which belongs to the technical field of ship cluster formation control, including: designing an expected formation based on a hierarchical virtual leader strategy, establishing a ship expected position library, and assigning indexes to expected positions in the formation; matching the position index in the formation with the ship, and when there is a collision risk between ships, updating the expected position matching relationship between ships through a local negotiation strategy of the ship formation to eliminate the risk; training a ship motion control decision control network based on collaborative exploration deep reinforcement learning, and controlling the ship to track the expected position navigation. The present invention can realize the autonomous collaborative formation of large-scale clusters of ships.

Description

Technical Field

[0001] The present invention belongs to the technical field of ship cluster formation control, and more specifically, relates to a large-scale ship cluster formation control method based on collaborative exploration deep reinforcement learning. Background Art

[0002] Ship clusters can perform tasks such as water surveys and data collection at lower costs and higher efficiency, and replace personnel in dangerous and challenging tasks such as maritime patrols, disaster prevention and rescue, and marine resource development, thus ensuring the safety of personnel. The research on large-scale cluster formation navigation technology of ships is of great significance, and has a profound impact on the sustainable development and innovation of ocean and water-related fields, which helps to improve efficiency, reduce risks, protect the environment, maintain national security, and promote scientific research.

[0003] At present, all the major maritime powers in the world have focused on the field of intelligent ships and unmanned surface vessels and invested huge resources. Many countries and regions such as Japan, South Korea and Europe regard it as a key development area and have launched a series of key technology research projects in the fields of formation navigation, including Japan's "SSAP" and "MEGURI 2040", South Korea's "KASS" and "HiNas Control 2.0", Europe's "Yara Birkeland" and "AEGIS". Many domestic research units have carried out a series of research, including "Intelligent Ship 1.0 R&D Special Project", "Zhuhai Cloud", "Zhifeihao", etc., and accumulated experience and data from real sea tests, but have not yet formed a good synergy in the field of unmanned surface vessel cluster formation control research.

[0004] In the field of unmanned surface vessel cluster control technology, breakthroughs in advanced communication and perception technologies have helped the rapid development of unmanned surface vessel cluster collaborative control technology. Researchers have developed various cluster collaboration algorithms and control algorithms to coordinate and control the behavior of unmanned surface vessel clusters. Collaborative algorithms include distributed control, centralized control, distributed leader-follower, etc., and control algorithms include sliding mode control, adaptive robust control, model predictive control, heuristic algorithms, artificial intelligence algorithms, etc. Through the combination of collaborative algorithms and control algorithms, it is ensured that the boats in the cluster can work together to perform specific tasks.

[0005] However, at the current stage, the cluster formation control algorithm has not effectively solved the problem of autonomous control of large-scale unmanned surface vessel clusters of more than 30 vessels. The autonomous and collaborative capabilities of the unmanned surface vessels within the cluster are limited, the formation control flexibility is poor, and it is difficult to change the formation quickly. When there is a collision conflict between boats, it is difficult to achieve collision avoidance in a limited space. The design of the unmanned surface vessel motion controller is complex, and it is difficult to make quick decisions. Summary of the invention

[0006] In view of the above defects or improvement needs of the prior art, the present invention designs a large-scale cluster formation control method of ships based on collaborative exploration deep reinforcement learning, aiming at the problem of large-scale cluster formation control of ships, which solves the problems of free transformation of unmanned surface vessel cluster formation, formation navigation, internal position conflict of cluster, collision avoidance between boats, etc., and improves the efficiency of exploring the navigation environment of unmanned surface vessel cluster formation based on the collaborative exploration deep reinforcement learning algorithm, and realizes dynamic team control of large-scale unmanned surface vessel clusters, which can provide important reference for the research on autonomous navigation of unmanned surface vessels and cluster collaborative control.

[0007] To achieve the above object, according to one aspect of the present invention, a method for controlling a fleet of ships based on collaborative exploration deep reinforcement learning is provided, comprising:

[0008] Design the desired formation based on the hierarchical virtual leader strategy, establish a library of desired ship positions, and assign indexes to the desired positions within the formation;

[0009] Match the position index within the formation with the ship. When there is a risk of collision between ships, the expected position matching relationship between ships is updated through the local negotiation strategy of the ship formation to eliminate the risk.

[0010] Based on collaborative exploration deep reinforcement learning, the ship motion control decision control network is trained to control the ship to track the desired position.

[0011] In some optional implementation schemes, the design of the desired formation based on the hierarchical virtual leader strategy, establishing a library of desired ship positions, and assigning indexes to the desired positions in the formation include:

[0012] According to the preset formation and formation size, the position layers are automatically divided and the position index is assigned. The position index includes the layer index and the intra-layer index to form a ship expected position library, so that the ship can query the expected position in the ship expected position library according to the layer index and the intra-layer index of the expected position, and when the formation changes, it is only necessary to update the ship expected position library and index value according to the formation.

[0013] In some optional implementation schemes, the expected position and actual position of the ship in the fleet formation design are respectively: Among them, L is the number of layers divided, represents the expected position of the ship, P(t)={(x i (t),y i (t)|i∈N * ,i<M} represents the actual position of the ship, Indicates the kth ship in the lth layer of the fleet at time t l The coordinates of the expected position of the ship, Indicates the kth ship in the lth layer of the fleet at time t lThe coordinates of the actual position of the ship, l represents the layer index of the position of the ship in the formation, k l Indicates the index within the layer, N * is a set of positive integers, and M is the number of ships in the fleet.

[0014] In some optional implementation schemes, the local negotiation strategy of the ship formation achieves conflict resolution by negotiating and exchanging the expected position indexes of locally conflicting ships.

[0015] In some optional implementation schemes, when some ships are out of position when forming a fleet or changing formations, and the distance between the actual position and the expected position is greater than a preset value, and there are other ships in between, the actual position of the ships is compared with the expected position, and a set of ships that conflict with the currently controlled ship is constructed, the ship with the greatest risk of collision is found, and the expected position indexes of the two ships are exchanged, so that the ships can sail to the nearby target positions.

[0016] In some optional implementation schemes, the collaborative exploration-based deep reinforcement learning includes the design of state space, action space, decision network and reward function.

[0017] In some optional embodiments, the state space includes the motion state of the ship, the relative angle and distance between the ship and the desired position; the action space includes the control input of the ship, including: thrust in the heave direction and torque in the bow roll direction.

[0018] In some optional implementation schemes, the reward function is designed as: Among them, e d 、e a , v represent the distance between the ship and the expected position, the error angle between the actual heading of the ship and the expected heading of the ship, and the speed of the ship, respectively, and w d 、w a 、w v are all weight coefficients greater than zero, and w d +w a +w v =1.

[0019] In some optional implementation schemes, during the environmental state exploration phase, the collaborative exploration-based deep reinforcement learning stores the exploration experience in a shared experience pool through parallel exploration of individuals in the ship formation for training the ship decision control network. At preset time intervals, the parameters of each decision control network are synchronized to balance the learning progress of the decision control network and promote the overall training progress.

[0020] According to another aspect of the present invention, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the steps of any one of the above methods are implemented.

[0021] In general, the above technical solutions conceived by the present invention can achieve the following beneficial effects compared with the prior art:

[0022] The top-down "formation design-real-time scheduling-motion control" architecture was adopted, and a ship formation construction strategy was designed based on hierarchical virtual leaders. Local coordination of ships was achieved through conflict negotiation, solving the problem of ship collision avoidance within the formation. The shared experience pool was used to design collaborative exploration of deep reinforcement learning, which improved the training efficiency of the decision-making control model and achieved accurate and flexible large-scale ship formation control. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 A large-scale ship formation control method framework based on collaborative exploration deep reinforcement learning is provided in an embodiment of the present invention;

[0024] Figure 2 A ship position allocation strategy based on a hierarchical virtual leader strategy provided by an embodiment of the present invention;

[0025] Figure 3 A local negotiation strategy for a large-scale fleet of ships provided by an embodiment of the present invention;

[0026] Figure 4 A ship decision control network training process based on collaborative exploration deep reinforcement learning is provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0027] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0028] The present invention provides a ship cluster formation control method based on collaborative exploration deep reinforcement learning, comprising:

[0029] Ship formation time-varying formation construction phase: used to assign target position points to ships according to the expected formation of the ship cluster formation;

[0030] Local negotiation phase of the formation: used to negotiate the movement of ships to avoid collisions when there are position conflicts or collision risks between ships within the cluster formation;

[0031] Ship motion control stage: used to control the ship to navigate to the target position.

[0032] Preferably, in the stage of constructing a time-varying formation of the ship fleet, the desired formation is designed based on a hierarchical virtual leader strategy, a library of desired ship positions is established, and indexes are assigned to the desired positions within the formation.

[0033] Preferably, the hierarchical virtual leader strategy automatically divides the position layers and assigns position indexes according to the preset formation and formation size, and the position index includes a layer index and an intra-layer index to form a ship expected position library; the ship can query the expected position in the ship expected position library according to the layer index and the intra-layer index of its expected position; when the formation changes, it is only necessary to update the ship expected position library and the index value according to the formation; the expected position and actual position of the ship in the formation design can be expressed as:

[0034]

[0035]

[0036] Among them, L is the number of layers divided, Indicates the desired position of the ship, Indicates the actual position of the ship. Indicates the kth ship in the lth layer of the fleet at time t l The coordinates of the desired position of the ship, Indicates the kth ship in the lth layer of the fleet at time t l The coordinates of the actual position of the ship, l represents the layer index of the position of the ship in the formation, k l Indicates the index within the layer, N * is a set of positive integers, and M is the number of ships in the fleet.

[0037] Preferably, in the local negotiation stage of the formation, the position index within the cluster formation is matched with the ship, and when there is a risk of collision between ships, the expected position matching relationship between the ships is updated through the local negotiation strategy of the ship formation to eliminate the risk.

[0038] Preferably, the local negotiation strategy of the ship formation achieves conflict resolution by negotiating and exchanging the expected position indexes of locally conflicting ships.

[0039] Preferably, in the ship motion control stage, a ship motion control decision control network is trained based on a collaborative exploration deep reinforcement learning method to control the ship to track the target position and navigate.

[0040] Preferably, the collaborative exploration-based deep reinforcement learning includes the design of state space, action space, decision network and reward function.

[0041] Preferably, in the environmental state exploration phase, the collaborative exploration-based deep reinforcement learning stores the exploration experience in a shared experience pool through parallel exploration of individuals in the ship formation for training the ship decision control network. At regular intervals, the parameters of each decision control network are synchronized to balance the learning progress of the decision control network and promote the overall training progress.

[0042] In one embodiment of the present invention, Figure 1 The large-scale ship formation control algorithm based on collaborative exploration deep reinforcement learning is divided into three main steps, namely, the construction of time-varying formation of ship formation, local negotiation of formation and ship motion control. The expected formation is designed based on the hierarchical virtual leader strategy, and the expected position library of ships is established, and indexes are assigned to the expected positions in the formation. Then, in the local negotiation stage of the ship formation, the position index in the formation is matched with the ship. When there is a risk of collision between ships, the matching relationship of the expected positions between ships is updated through local negotiation to eliminate the risk. Then, the ship motion control decision control network is trained based on collaborative exploration deep reinforcement learning to control the ship to track the expected position. Finally, the ship formation navigation is realized by looping the above steps.

[0043] In one embodiment of the present invention, Figure 2 The ship formation hierarchical design strategy automatically divides the position layer and assigns the position index according to the preset formation and formation size. The position index includes the layer index and the layer index to form the ship expected position library. The ship can query the expected position in the ship expected position library according to the layer index and layer index of its expected position.

[0044] In one embodiment of the present invention, Figure 3 , some ships are out of position when forming a fleet and changing formations. The actual position is far from the expected position and there are other ships in between. At this time, if the out-of-position ship is controlled to sail to the expected position, the sailing distance is long and there is a risk of collision with other ships, which is difficult. First, by comparing the actual position of the ship with the expected position, a set of ships that conflict with the currently controlled ship is constructed, and the ship with the greatest collision risk is found. The expected position indexes of the two ships are exchanged to make the ship sail to a suitable nearby position, thereby eliminating the risk of collision and improving the efficiency of the formation.

[0045] In one embodiment of the present invention, Figure 4 The collaborative exploration-based deep reinforcement learning includes the design of state space, action space, decision network and reward function.

[0046] In one embodiment of the present invention, the state space includes the motion state of the ship, the relative angle and distance between the ship and the desired position.

[0047] In one embodiment of the present invention, the action space includes the control input τ of the ship = (τu ,τ r ), that is, the thrust τ in the heave direction u and the yaw direction moment τ r .

[0048] In one embodiment of the present invention, the decision network includes an online decision control network and a target decision control network, which use the same fully connected network structure, the number of network nodes is designed to be (400×300×2), and the activation function of each layer is (Relu, Relu, Tanh). The value network includes an online value network and a target value network, which use the same fully connected network structure, the number of network nodes is designed to be (400×300×1), and the activation function of each layer is (Relu, Relu, None).

[0049] In one embodiment of the present invention, the reward function is designed as:

[0050]

[0051] Among them, e d 、e a , v represent the distance between the ship and the target position, the error angle between the actual heading of the ship and the expected heading of the ship, and the speed of the ship, respectively, and w d 、w a 、w v are all weight coefficients greater than zero, and w d +w a +w v =1.

[0052] It should be pointed out that, according to the needs of implementation, the various steps / components described in this application can be split into more steps / components, and two or more steps / components or partial operations of steps / components can be combined into new steps / components to achieve the purpose of the present invention.

[0053] It will be easily understood by those skilled in the art that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A ship cluster formation control method based on collaborative exploration deep reinforcement learning, characterized in that: include: Design the desired formation based on the hierarchical virtual leader strategy, establish a library of desired ship positions, and assign indexes to the desired positions within the formation; Match the position index within the formation with the ship. When there is a risk of collision between ships, the expected position matching relationship between ships is updated through the local negotiation strategy of the ship formation to eliminate the risk. Based on collaborative exploration deep reinforcement learning, the ship motion control decision control network is trained to control the ship to track the desired position. The collaborative exploration-based deep reinforcement learning includes the design of state space, action space, decision network and reward function, wherein the reward function is designed as follows: Among them, e d 、e a , v represent the distance between the ship and the expected position, the error angle between the actual heading of the ship and the expected heading of the ship, and the speed of the ship, respectively, and w d 、w a 、w v are all weight coefficients greater than zero, and w d +w a +w v =1.

2. The method according to claim 1, characterized in that: The method of designing an expected formation based on a hierarchical virtual leader strategy, establishing an expected position library for ships, and assigning indexes to expected positions in the formation includes: According to the preset formation and formation size, the position layers are automatically divided and the position index is assigned. The position index includes the layer index and the intra-layer index to form a ship expected position library, so that the ship can query the expected position in the ship expected position library according to the layer index and the intra-layer index of the expected position, and when the formation changes, it is only necessary to update the ship expected position library and index value according to the formation.

3. The method according to claim 2, characterized in that The expected position and actual position of ships in the fleet formation design are: Among them, L is the number of layers divided, represents the expected position of the ship, P(t)={(x i (t),y i (t)|i∈N * ,i<M} represents the actual position of the ship, Indicates the kth ship in the lth layer of the fleet at time t l The coordinates of the expected position of the ship, Indicates the kth ship in the lth layer of the fleet at time t l The coordinates of the actual position of the ship, l represents the layer index of the position of the ship in the formation, k l Indicates the index within the layer, N * is a set of positive integers, and M is the number of ships in the fleet.

4. The method according to claim 2 or 3, characterized in that: The local negotiation strategy of the ship formation realizes conflict resolution by negotiating and exchanging the expected position indexes of the ships that have local conflicts.

5. The method according to claim 4, characterized in that When some ships are out of position when forming a fleet or changing formations, and the distance between the actual position and the expected position is greater than the preset value, and there are other ships in between, by comparing the actual position of the ship with the expected position, a set of ships that conflict with the currently controlled ship is constructed, and the ship with the greatest collision risk is found. The expected position indexes of the two ships are exchanged, so that the ships can sail to the nearby target position.

6. The method according to claim 5, characterized in that The state space includes the ship's motion state, the relative angle and distance between the ship and the desired position; the action space includes the ship's control input, including: thrust in the heave direction and moment in the bow roll direction.

7. The method according to claim 6, characterized in that In the environmental state exploration phase, the collaborative exploration-based deep reinforcement learning stores the exploration experience in a shared experience pool through parallel exploration of individuals in the ship formation for training the ship decision control network. At preset time intervals, the parameters of each decision control network are synchronized to balance the learning progress of the decision control network and promote the overall training progress.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images