A multi-task oriented ship swarm planning and control method

Abstract

The present invention discloses a multi-task swarm planning and control method, comprising the following steps: training simulated data of a sea area based on a reinforcement learning optimization algorithm, using multiple processes to perform multi-task parallel training during the training process; calculating the difference between its action value function and state value function for each ship to obtain an advantage value, combining the advantage value with directional entropy to encourage the ship to select the advantage value in the direction of entropy reduction as much as possible; calculating the error between the estimated advantage value and the advantage value based on directional entropy, and calculating the Q function based on the error; making decisions using a reinforcement learning method; inputting the actual state of the sea area into the trained reinforcement learning model, and outputting the action strategy of each ship. This application further adjusts the scheduling decision in each task by calculating the advantage value based on directional entropy for each ship and optimizing the deviation between each ship and the advantage value estimated by the command center.

Description

Technical Field

[0001] The present invention belongs to the field of intelligent command technology, and in particular relates to a multi-task oriented ship cluster planning and control method. Background Art

[0002] Swarm training requires coordination between ships, including information sharing, situational awareness, and joint operations. This requires each ship to possess a highly automated command system capable of rapid decision-making and coordinated action to cope with the complex and ever-changing maritime environment. Furthermore, swarm training also requires consideration of technical issues such as formation control, formation motion planning, target identification and tracking, and information sharing and fusion. Therefore, planning and control of swarms is a complex, multi-tasking system.

[0003] Although existing models have initially integrated knowledge and data in training data, they are overly sensitive to adversarial samples due to their use of purely supervised learning. Once non-random noise that is imperceptible to humans is added to the sample set, the model will give erroneous outputs with high confidence. This phenomenon is common in deep learning models and is even more fatal when applied to ship navigation problems. Summary of the Invention

[0004] The goal of this application is to use a decision-making system based on multi-ship deep reinforcement learning to solve the optimization process of a series of ship planning and control problems. Through the calculation of directional entropy advantage value in reinforcement learning, it is ultimately possible to achieve a ship cluster planning and control effect for multiple tasks (ship obstacle avoidance, cruising scenarios, etc.).

[0005] To achieve the above objectives, the multi-task swarm planning and control method disclosed in this application includes the following steps:

[0006] The simulation data of the sea area is trained based on the reinforcement learning optimization algorithm, and multiple processes are used for multi-task parallel training during the training process;

[0007] Each ship calculates the difference between its action value function and state value function to obtain the advantage value, and combines the advantage value with the directional entropy to encourage the ship to choose the advantage value in the direction of entropy reduction as much as possible;

[0008] Calculate the error between the advantage value estimated by the command center and the advantage value calculated for each ship based on directional entropy, and calculate the Q function based on the error;

[0009] Adopting the "actor-critic" method in reinforcement learning to make decisions, the Q function is optimized by gradient descent method, and the optimal parameters in reinforcement learning are obtained through training;

[0010] The actual state of the sea area is input into the trained reinforcement learning model, and the action strategy of each ship is output.

[0011] Preferably, the advantage value is calculated as follows:

[0012] A π (s, a) = Q π (s, a)-V π (s)

[0013] π is the strategy, Q π (s, a) is the expected reward of taking action a in state s, V π (s) is the expected return of following the current policy in state s;

[0014] The advantage value is divided into n directional levels, each directional level has a certain probability pi in all directional levels, and the advantage value based on directional entropy is calculated as follows:

[0015]

[0016] It represents the advantage value estimate of ship x for the behavior based on current knowledge.

[0017] Preferably, the error between the advantage value calculated by the ship based on directional entropy and the advantage value estimated by the command center is calculated as follows:

[0018]

[0019] in and They represent the advantage value based on directional entropy calculated by the ship and the advantage value estimated by the command center, i is the scene number, M is the scene set, and τ is a hyperparameter.

[0020] Preferably, the Q function is as follows:

[0021]

[0022] in, is the Q function value, is the action at time t+1, is the value at time t+1, is the total reward value of scene i at time t, and π(i) is the strategy under scene i.

[0023] Preferably, the reward function between ship x and ship y is:

[0024] R i =λ1r c0 +λ2r en

[0025] Among them, r co Reward for value improvement, ren is the reward for situation improvement, λ1 and λ2 are adjustment factors;

[0026] The value improvement reward is calculated as follows:

[0027]

[0028] is the state action value of ship x when ship y cooperates with ship x, t is the current moment, t-1 is the previous moment, is the state action value of ship x when ship y did not cooperate with ship x at the previous moment;

[0029] The situation improvement bonus is a bonus given for changes in the number of cooperating ships. The situation improvement bonus is calculated as follows:

[0030]

[0031] For the knowledge of the situation before the change, is the knowledge in the changed situation, θ is the knowledge parameter, are the state-value pairs of all ships in the situation.

[0032] Preferably, the total reward value of scene i at time t is calculated as follows:

[0033]

[0034] L is the total number of ships in scenario i, r l is the reward value of the l-th ship, α l is the weight of the reward value of the l-th ship.

[0035] The beneficial effects of this application are as follows:

[0036] This application aims to optimize the training needs of ship obstacle avoidance and cruising scenarios, and is based on building a ship cluster scheduling method that is more suitable for high-precision sea area scenario needs. By calculating the advantage value based on directional entropy for each ship and optimizing the deviation between the advantage value of each ship and the command center's estimate, the scheduling decision in each task can be further adjusted, and finally a ship cluster planning and control effect for multiple tasks (ship obstacle avoidance, cruising scenarios, etc.) can be achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 Flowchart of the ship cluster planning and control method of this application. DETAILED DESCRIPTION

[0038] The present invention will be further described below with reference to the accompanying drawings, but the present invention is not limited in any way. Any changes or substitutions made based on the teachings of the present invention fall within the scope of protection of the present invention.

[0039] Artificial intelligence (AI) refers to the theories, methods, techniques, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, to perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that seeks to understand the essence of intelligence and produce new intelligent machines that can respond in a manner similar to human intelligence. AI also involves studying the design principles and implementation methods of various intelligent machines, enabling them to possess the capabilities of perception, reasoning, and decision-making.

[0040] Natural language processing (NLP) is a key area of ​​research in computer science and artificial intelligence. It studies the theories and methods that enable effective communication between humans and computers using natural language. Natural language processing (NLP) integrates linguistics, computer science, and mathematics. Therefore, research in this field involves natural language—the language we use in everyday life—and is closely linked to the study of linguistics. Natural language processing technologies typically include text processing, semantic understanding, machine translation, robotic question answering, and knowledge graphs.

[0041] The technical solutions provided in the embodiments of this application involve technologies such as machine learning and natural language processing of artificial intelligence, and are specifically introduced and explained through the following embodiments.

[0042] Before introducing the embodiments of the present application, some terms involved in the present application are explained.

[0043] 1. Reinforcement Learning: Reinforcement learning (RL) is a branch of machine learning that aims to train intelligent agents to learn how to take actions to maximize a cumulative reward while interacting with their environment. The core idea of ​​RL is that through trial and error, agents learn the optimal strategy—the actions that yield the greatest long-term benefits in a given environment.

[0044] The basic elements of reinforcement learning include:

[0045] Agent: An agent that performs actions, such as a robot, software program, or decision-making system.

[0046] Environment: The external system in which the agent exists, which can be the physical world, a game interface, or an abstract state space.

[0047] State: A description of the environment at a certain moment, based on which the agent chooses actions.

[0048] Action: An action that the agent can perform in the environment, such as moving, rotating, or buying a stock.

[0049] Reward: The feedback received by the agent from the environment after taking an action. Rewards can be positive (encouragement) or negative (punishment) and are used to guide the agent's learning.

[0050] Policy: The rule by which an agent chooses actions, that is, what action to take in a given state.

[0051] Value Function: Evaluates the long-term return of a state or state-action pair and guides the optimization of the strategy.

[0052] The Q-function (also known as the state-action value function) is used to estimate the expected long-term reward of taking a specific action in a given state. The Q-function is often denoted as Q(s,a), where s represents the state and a represents the action. The value of the Q-function represents the expected cumulative reward obtained by following a policy, starting from state s and taking action a.

[0053] The reinforcement learning process: An agent performs actions in the environment, changing its state. The environment then responds with rewards based on the new state. The agent then updates its strategy based on the rewards and the new state, hoping to achieve higher rewards in the future. This process continues iteratively until the agent learns to take the optimal action in each state.

[0054] This application focuses on multi-task scenarios for ship swarm mission planning and control algorithm optimization training. Ship swarms need to meet the mission planning and control functions in complex dynamic scenarios, that is, the swarm needs to be able to effectively avoid obstacles such as reefs and unrelated ships during the journey, and at the same time, it needs to cruise efficiently under a given trajectory and reach the destination quickly. This requirement brings great challenges to machine learning algorithms: (1) There are many types of obstacles, and avoiding static and moving obstacles requires the model to learn complex and accurate swarm planning strategies; (2) Under the preset route, the ship needs to be controlled to learn to reach the destination in the shortest possible path; (3) Obstacle avoidance and the shortest path are inherently contradictory. For example, in order to avoid fishing boats, the ship may need to learn a waiting or detour strategy, but this strategy itself will undermine the learning of the shortest path.

[0055] like Figure 1As shown, in one embodiment, the multi-task oriented ship cluster planning and control method disclosed in this application includes the following steps:

[0056] The simulation data of the sea area is trained based on the reinforcement learning optimization algorithm, and multiple processes are used for multi-task parallel training during the training process;

[0057] Each ship calculates the difference between its action-value function and state-value function to obtain an advantage value. This advantage value is then combined with directional entropy to encourage ships to choose advantage values ​​in directions that minimize entropy. The advantage value calculation utilizes the Advantage mechanism of the Dueling DQN algorithm, ultimately yielding a value reference based on the behavior itself. This value reflects whether the current behavior can significantly alter the state of the scene. Ultimately, directional entropy is incorporated into the policy network update, encouraging ships to choose advantage values ​​in directions that minimize entropy. This effectively ensures that the final scheduling strategy will not fall into a local optimum, freeing ships from being constrained by the observed local situation and increasing the likelihood that the neural network will find the global optimal solution.

[0058] Calculate the error between the advantage value estimated by the command center and the advantage value calculated for each ship based on directional entropy, and calculate the Q function based on the error;

[0059] Adopting the "actor-critic" method in reinforcement learning to make decisions, the Q function is optimized by gradient descent method, and the optimal parameters in reinforcement learning are obtained through training;

[0060] The actual state of the sea area is input into the trained reinforcement learning model, and the action strategy of each ship is output. This framework can be extended to multiple different scenarios. Real interaction data is stored in the experience knowledge base and trained offline with the knowledge graph to effectively optimize the neural network parameters. Ultimately, the "actor-critic" method of reinforcement learning is used for decision-making. That is, the current state of the sea area is input, the probability of each ship's possible action is output, and the current action of each ship is sampled.

[0061] In one embodiment, a swarm of ships must jointly intercept an opposing ship and determine whether interception is necessary. Since our fleet is smaller than the enemy's, simple strategies like one-on-one defense cannot meet the mission's requirements. Therefore, effective cooperation among these ships is essential to maximize the mission's success rate. This application addresses the characteristics of swarms and designs an algorithm to optimize their operational strategies.

[0062] In one embodiment, the advantage value is calculated as follows:

[0063] A π (s, a) = Q π (s, a)-V π(s)

[0064] π is the strategy, Q π (s, a) is the expected reward of taking action a in state s, V π (s) is the expected return of following the current policy in state s;

[0065] The advantage value is divided into n direction levels, and each direction level has a certain probability p in all direction levels. i , the advantage value based on directional entropy is calculated as follows:

[0066]

[0067] It represents the advantage value estimate of ship x for the behavior based on current knowledge.

[0068] In one embodiment, the error between the advantage value calculated by the ship based on directional entropy and the advantage value estimated by the command center is calculated as follows:

[0069]

[0070] in and They represent the advantage value based on directional entropy calculated by the ship and the advantage value estimated by the command center, i is the scene number, M is the scene set, and τ is a hyperparameter.

[0071] In one embodiment, the Q function is as follows:

[0072]

[0073] in, is the Q function value, is the action at time t+1, is the value at time t+1, is the total reward value of scene i at time t, and π(i) is the strategy under scene i.

[0074] In one embodiment, the reward function between ship x and ship y is:

[0075] R i =λ1r c0 +λ2r en

[0076] Among them, r co Reward for value improvement, r en is the reward for situation improvement, λ1 and λ2 are adjustment factors;

[0077] The value improvement reward is calculated as follows:

[0078]

[0079] is the state action value of ship x when ship y cooperates with ship x, t is the current moment, t-1 is the previous moment, is the state action value of ship x when ship y did not cooperate with ship x at the previous moment;

[0080] The situation improvement bonus is a bonus given for changes in the number of cooperating ships. The situation improvement bonus is calculated as follows:

[0081]

[0082] For the knowledge of the situation before the change, is the knowledge in the changed situation, θ is the knowledge parameter, are the state-value pairs of all ships in the situation.

[0083] In one embodiment, the total reward value of scene i at time t is calculated as follows:

[0084]

[0085] L is the total number of ships in scenario i, r l is the reward value of the l-th ship, α l is the weight of the reward value of the l-th ship.

[0086] The beneficial effects of this application are as follows:

[0087] This application aims to optimize the training needs of ship obstacle avoidance and cruising scenarios, and is based on building a ship cluster scheduling method that is more suitable for high-precision sea area control needs. By calculating the advantage value based on directional entropy for each ship and optimizing the deviation between the advantage value of each ship and the command center's estimate, the scheduling decision in each task can be further adjusted, and finally a ship cluster planning and control effect for multiple tasks (ship obstacle avoidance, cruising scenarios, etc.) can be achieved.

[0088] As used herein, the word "preferred" is intended to serve as an example, instance, or illustration. Any aspect or design described herein as "preferred" is not necessarily to be construed as advantageous over other aspects or designs. Rather, the use of the word "preferred" is intended to present concepts in a concrete manner. As used in this application, the term "or" is intended to mean an inclusive "or" rather than an exclusive "or." That is, unless otherwise specified or clear from the context, "X employs A or B" is intended to mean any of the naturally inclusive permutations. That is, if X employs A; X employs B; or X employs both A and B, then "X employs A or B" is satisfied in any of the foregoing examples.

[0089] Moreover, although the present disclosure has been shown and described with respect to one or implementation, those skilled in the art will think of equivalent variations and modifications based on reading and understanding of this specification and the accompanying drawings. The present disclosure includes all such modifications and variations and is limited only by the scope of the appended claims. In particular, with respect to the various functions performed by the above-mentioned components (such as elements, etc.), the terms used to describe such components are intended to correspond to any component (unless otherwise indicated) that performs the specified function of the component (such as it is functionally equivalent), even if structurally different from the disclosed structure that performs the function in the exemplary implementation of the present disclosure shown herein. In addition, although the specific features of the present disclosure have been disclosed with respect to only one of several implementations, such features can be combined with one or other features of other implementations that can be desired and advantageous for a given or specific application. Moreover, insofar as the terms "including", "having", "containing" or their variations are used in specific embodiments or claims, such terms are intended to be included in a manner similar to the term "comprising".

[0090] The functional units in the embodiments of the present invention may be integrated into a single processing module, or each unit may exist physically separately, or multiple or more units may be integrated into a single module. The aforementioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium. The aforementioned storage medium may be a read-only memory, a magnetic disk, or an optical disk, etc. The aforementioned devices or systems may execute the storage method in the corresponding method embodiment.

[0091] In summary, the above embodiment is one implementation method of the present invention, but the implementation method of the present invention is not limited to the described embodiment. Any other changes, modifications, substitutions, combinations, and simplifications that deviate from the spirit and principles of the present invention should be equivalent replacement methods and are included in the scope of protection of the present invention.

Claims

1. A multi-task oriented ship cluster planning and control method, characterized in that: The following steps are involved: The simulation data of the sea area is trained based on the reinforcement learning optimization algorithm, and multiple processes are used for multi-task parallel training during the training process; Each ship calculates the difference between its action value function and state value function to obtain the advantage value, and combines the advantage value with the directional entropy to encourage the ship to choose the advantage value in the direction of entropy reduction as much as possible; Calculate the error between the advantage value estimated by the command center and the advantage value calculated for each ship based on directional entropy, and calculate the Q function based on the error; Adopting the "actor-critic" approach in reinforcement learning to make decisions, the Q function is optimized through gradient descent, and the optimal parameters in reinforcement learning are obtained through training. The actual state of the sea area is input into the trained reinforcement learning model, and the action strategy of each ship is output.

2. The multi-task oriented ship cluster planning and control method according to claim 1 is characterized in that: The advantage value is calculated as follows: A π (s,a)=Q π (s,a)―V π (s) π is the strategy, Q π (s,a) is the expected reward of taking action a in state s, V π (s) is the expected return of following the current policy in state s; The advantage value is divided into n direction levels, and each direction level has a certain probability p in all direction levels. i , the advantage value based on directional entropy is calculated as follows: It represents the advantage value estimate of ship x for the behavior based on current knowledge.

3. The multi-task oriented ship cluster planning and control method according to claim 2 is characterized in that: Calculate the error between the ship's calculated advantage value based on directional entropy and the advantage value estimated by the command center as follows: in and They represent the advantage value based on directional entropy calculated by the ship and the advantage value estimated by the command center, i is the scene number, M is the scene set, and τ is a hyperparameter.

4. The multi-task oriented ship cluster planning and control method according to claim 3 is characterized in that: The Q function is shown below: in, is the Q function value, is the action at time t+1, is the value at time t+1, is the total reward value of scene i at time t, π (i) is the strategy for scenario i.

5. The multi-task oriented ship cluster planning and control method according to claim 4 is characterized in that: The reward function between ship x and ship y is: R i =λ1r co +λ2r en Among them, r co Reward for value improvement, r en is the reward for situation improvement, λ1 and λ2 are adjustment factors; The value improvement reward is calculated as follows: is the state action value of ship x when ship y cooperates with ship x, t is the current moment, t-1 is the previous moment, is the state action value of ship x when ship y did not cooperate with ship x at the previous moment; The situation improvement bonus is a bonus given for changes in the number of cooperating ships. The situation improvement bonus is calculated as follows: For the knowledge of the situation before the change, is the knowledge in the changed situation, θ is the knowledge parameter, are the state-value pairs of all ships in the situation.

6. The multi-task oriented ship cluster planning and control method according to claim 5 is characterized in that: The total reward value of scene i at time t is calculated as follows: L is the total number of ships in scenario i, r l is the reward value of the lth ship, a l is the weight of the reward value of the l-th ship.

Images