Intelligent ship autonomous obstacle avoidance and target tracking method based on meta learning and reinforcement learning
By introducing meta-learning and reinforcement learning in the intelligent ship's autonomous obstacle avoidance and target tracking tasks, the task is decomposed into obstacle avoidance and tracking, and the internal and external meta-parameter update rules are adopted to solve the inefficiency of traditional deep reinforcement learning algorithms in complex environments and Poor adaptability problem achieves faster convergence and higher adaptability.
Patent Information
- Application Number
- CN202411065210.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-05
- Publication Date
- 2024-11-15
AI Technical Summary
Traditional deep reinforcement learning algorithms have poor convergence characteristics and weak environmental adaptability in intelligent ship autonomous obstacle avoidance and target tracking tasks, and cannot effectively deal with complex dynamic tasks, especially when the environment changes or obstacles change, resulting in inefficiency. .
Using a method based on meta-learning and reinforcement learning, the Markov decision-making process model is constructed, the state space, action space and reward functions are designed, and the meta-reinforcement learning algorithm is introduced to decompose the task into independent obstacle avoidance and tracking tasks, and the internal External meta-parameter update rules, pre-training the basic meta-task set to improve the convergence speed and generalization ability of the model.
It significantly improves the convergence characteristics and environmental adaptability of the autonomous obstacle avoidance and target tracking tasks of smart ships in complex environments, reduces training time, improves engineering efficiency, and improves the universality of the model.
Smart Images

Figure CN118963362A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of ship intelligent control, and in particular to an autonomous obstacle avoidance and target tracking method for an intelligent ship based on meta-learning and reinforcement learning. Background Art
[0002] With the increase in ship density and the interweaving of routes, maritime traffic has become increasingly complex. This complexity makes maritime traffic face more potential dangerous encounter scenarios, highlighting the importance of ship collision avoidance decisions in ensuring maritime traffic safety. Especially in scenarios where dynamic and static obstacle environments are intertwined, ship collision avoidance decisions are particularly critical. Traditional obstacle avoidance algorithms include ant colony algorithms, shortest path guidance vectors, and Bayesian reasoning. These algorithms all convert obstacle avoidance problems into optimization problems. However, these methods are not applicable to obstacle avoidance problems in variable or unknown environments due to their shortcomings such as long iteration time, weak generalization ability, and low intelligence level.
[0003] With the development of artificial intelligence technology, deep reinforcement learning has gradually been used to solve the problems of autonomous obstacle avoidance and target tracking for intelligent ships. However, traditional deep reinforcement learning algorithms have slow training speeds and can only handle tasks in a single environment. When the obstacle environment or target motion trajectory changes, the agent often needs to re-explore and train a lot. Therefore, how to improve the convergence speed and adaptability of deep reinforcement learning algorithms when facing complex dynamic tasks has become a hot topic and difficulty in the field of reinforcement learning. Summary of the invention
[0004] In order to solve the problems of poor convergence characteristics and weak environmental adaptability of traditional deep reinforcement learning in solving the tasks of autonomous obstacle avoidance and target tracking of intelligent ships, this paper proposes an intelligent ship autonomous obstacle avoidance and target tracking method based on meta-learning and reinforcement learning. Both the convergence characteristics of intelligent ship task training and the environmental adaptability for different tasks have been significantly improved.
[0005] The technical means adopted by the present invention are as follows:
[0006] An intelligent ship autonomous obstacle avoidance and target tracking method based on meta-learning and reinforcement learning includes the following steps:
[0007] S1. Based on the Markov decision process, a task model of autonomous obstacle avoidance and target point tracking of intelligent ships in complex environments is constructed, including designing a state space, an action space and a reward function that conform to navigation practice. The state space is a collection of the state of the intelligent ship itself and environmental information. The action space is a set of actions that can be executed by the intelligent ship. The reward function includes a tracking reward, an obstacle avoidance reward and a distance reward. The tracking establishment is used to evaluate the tracking effect, the obstacle avoidance reward is used to evaluate the obstacle avoidance effect, and the distance reward is used to evaluate the distance between the intelligent ship and the target point.
[0008] S2. Based on an improved reinforcement learning algorithm, the autonomous obstacle avoidance and target point tracking task model of the intelligent ship in a complex environment is trained. The improved reinforcement learning algorithm introduces meta-reinforcement learning into the reinforcement learning algorithm, and divides the joint task of autonomous obstacle avoidance and target point tracking of the model training into independent intelligent ship obstacle avoidance tasks and intelligent ship tracking tasks. The specific steps of model training include:
[0009] S201, pre-training the intelligent ship obstacle avoidance task and the intelligent ship tracking task, first learning the static target tracking strategy of the intelligent ship in an obstacle-free environment, then learning the autonomous obstacle avoidance strategy of the intelligent ship in a simple obstacle environment, and finally obtaining a meta-initial parameter that can adapt to the new tasks of autonomous obstacle avoidance and target tracking,
[0010] S202, performing overall training on the autonomous obstacle avoidance and target tracking task model of the intelligent ship in a complex environment, and obtaining correct action strategies for autonomous obstacle avoidance and target tracking of the intelligent ship in different test environments;
[0011] S3. Guide the actions of intelligent ships based on the trained model of autonomous obstacle avoidance and target point tracking in complex environments.
[0012] Further, the state space is expressed as:
[0013]
[0014] Among them, (x t ,y t ) represents the location coordinates of the smart ship, Indicates the target location coordinates, d max represents the maximum Euclidean distance between the smart ship and the target, v max Indicates the maximum sailing speed of the smart ship, v t Indicates the navigation speed of the smart ship at time t;
[0015] The action space is expressed as:
[0016] A=[-1,1,0] T
[0017] Among them, "-1" means turn left, "1" means turn right, and "0" means execute;
[0018] The reward function is expressed as:
[0019] R=r1+r2+r3
[0020]
[0021] r3=-d*0.1
[0022] Among them, r1 is the tracking reward, r2 is the obstacle avoidance reward, the sensing range is preset according to the needs of navigation practice, r3 is the distance reward, and R is the total reward. is the Euclidean distance of obstacles within the sensing range of the smart ship, and d is the Euclidean distance between the smart ship and the target point.
[0023] Furthermore, the intelligent ship obstacle avoidance task and intelligent ship tracking task are pre-trained. First, the static target tracking strategy of the intelligent ship in an obstacle-free environment is learned, and then the autonomous obstacle avoidance strategy of the intelligent ship in a simple obstacle environment is learned. Finally, a meta-initial parameter that can adapt to the new tasks of autonomous obstacle avoidance and target tracking is obtained, including:
[0024] The basic meta-task set T = {T1, T2}, where T j For the jth basic meta-task, create an experience replay pool for each basic meta-task in T are the state, action, reward of the intelligent ship at time t in the jth basic meta-task and the state of the intelligent ship at time t+1, j=1,2;
[0025] During the pre-training process, the intelligent ship trains each basic meta-task in T in turn to obtain a strategy that can adapt to each sub-task, and finally obtains the meta-initial parameters for learning the overall task.
[0026] Furthermore, the intelligent ship trains each basic meta-task in T in turn to obtain a strategy that can adapt to each sub-task, and finally obtains the meta-initial parameters of the overall learning task, including:
[0027] Design internal and external meta-parameter update rules so that internal network training and external meta-parameter update are performed alternately at a certain frequency. The internal network sequentially learns each basic meta-task to obtain different internal parameters. External meta-parameter update obtains meta-initial parameters by optimizing different parameters. The update rules are as follows:
[0028]
[0029] Among them, θ meta It is the external meta-parameter of the actor-target neural network in the DDPG algorithm; is the external meta-parameter of the critic target neural network; n is the number of basic meta-tasks to complete training; τ is a constant that controls the speed of meta-parameter update; Meta-task T j Actor-target neural network parameters during training; Meta-task T j Parameters of the critic target neural network during training, j = 1, 2, after pre-training, θ meta and That is the initial parameter.
[0030] Compared with the prior art, the present invention has the following advantages:
[0031] The present invention constructs a task model of autonomous obstacle avoidance and target point tracking for intelligent ships in complex environments based on the Markov decision process, and designs a state space, action space and reward function that conforms to navigation practice. Model-independent meta-learning is integrated into the deep reinforcement learning algorithm, and an internal and external meta-parameter update rule is designed to improve the convergence speed and generalization ability of the model. In addition, a basic meta-task set is constructed in the model pre-training part to improve the pre-training efficiency in actual engineering. The present invention improves both the convergence characteristics of the intelligent ship task training and the environmental adaptability facing different tasks. At the same time, pre-training using the basic meta-task set is more efficient than the traditional meta-task set. And the meta-learning method and the basic meta-task set have good versatility for deterministic strategy reinforcement learning algorithms. BRIEF DESCRIPTION OF THE DRAWINGS
[0032] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.
[0033] Figure 1 This is a flow chart of the method for autonomous obstacle avoidance and target tracking of an intelligent ship based on meta-learning and reinforcement learning of the present invention.
[0034] Figure 2 A flow chart for constructing a meta-task set in an embodiment of the present invention.
[0035] Figure 3 This is a diagram of the architecture of the task model of autonomous obstacle avoidance and target point tracking of an intelligent ship in a complex environment in an embodiment of the present invention. DETAILED DESCRIPTION
[0036] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.
[0037] like Figure 1 As shown, the present invention provides an intelligent ship autonomous obstacle avoidance and target tracking method based on meta-learning and reinforcement learning, which mainly includes:
[0038] S1. Based on the Markov decision process, a task model for autonomous obstacle avoidance and target point tracking of intelligent ships in complex environments is constructed, including designing a state space, action space and reward function that conform to navigation practice. The state space is a collection of the intelligent ship's own state and environmental information, the action space is a collection of actions that can be executed by the intelligent ship, and the reward function includes tracking reward, obstacle avoidance reward and distance reward. The tracking establishment is used to evaluate the tracking effect, the obstacle avoidance reward is used to evaluate the obstacle avoidance effect, and the distance reward is used to evaluate the distance between the intelligent ship and the target point.
[0039] Specifically, based on the Markov decision process, the present invention constructs a task model for autonomous obstacle avoidance and target tracking of intelligent ships in complex environments, and designs a state space, action space, and reward function that conform to navigation practice. First, the present invention regards the intelligent ship as a motion model in a two-dimensional space, and defines (x t ,y t ) is the position coordinate of the smart ship at time t, is the position coordinate of the target at time t obtained by the smart ship, d t is the Euclidean distance between the smart ship and the target point at time t, v t is the navigation speed of the smart ship at time t, is the straight-line distance between the smart ship and the obstacle at time t, h t is the heading of the smart ship at time t.
[0040] In order to better describe the task of autonomous obstacle avoidance and target tracking of smart ships, it is defined as a Markov decision process (hereinafter referred to as MDP). First, MDP consists of a state space S, an action space A, and a reward function R. Here, the state space of the present invention is a collection of the state of the smart ship itself and the environmental information. In this task, the state space S is composed of the position coordinates (x t ,y t ), target location coordinates Euclidean distance d between the smart ship and the target t, the straight-line distance between the smart ship and the obstacle Smart ship navigation speed v t and the heading h of the smart ship at time t t The data is normalized as shown in the following formula.
[0041]
[0042] Among them, x t ′,y t ′, d t ′, v t ′ is the final normalized result; d max is the maximum Euclidean distance between the smart ship and the target; v max is the maximum navigation speed of the smart ship. The final state space S is expressed as shown in Formula 2.
[0043]
[0044] The action space A is the action that can be executed by the intelligent ship. The actions are set to turn left, turn right and go straight. In order to unify the normalization, the present invention defines left turn, right turn and go straight as -1, 1, 0. The final action space is recorded as shown in Formula 3.
[0045] A=[-1,1,0] T (3)
[0046] The setting of the reward function is crucial to the algorithm training results. Different reward functions have different effects on the model convergence characteristics. In the tasks of autonomous obstacle avoidance and target tracking of intelligent ships, if sparse rewards are used, that is, rewards are fed back only after the intelligent ship tracking succeeds or fails, it will cause the problem of slow convergence speed. Therefore, the present invention sets a continuous reward function as shown in Formula 7.
[0047]
[0048] r3=-d*0.1 (6)
[0049] R=r1+r2+r3 (7)
[0050] Among them, r1 is the tracking reward; r2 is the obstacle avoidance reward; the sensing range is based on the needs of navigation practice. In this invention, the sensing range is set to a circular radius of 4 nautical miles; r3 is the distance reward; R is the total reward; is the Euclidean distance of obstacles within the sensing range of the smart ship; d is the Euclidean distance between the smart ship and the target point.
[0051] S2. Based on an improved reinforcement learning algorithm, the autonomous obstacle avoidance and target point tracking task model of the intelligent ship in a complex environment is trained. The improved reinforcement learning algorithm introduces meta-reinforcement learning into the reinforcement learning algorithm, and divides the joint task of autonomous obstacle avoidance and target point tracking of the model training into independent intelligent ship obstacle avoidance tasks and intelligent ship tracking tasks. The specific steps of model training include:
[0052] S201. Pre-training the intelligent ship obstacle avoidance task and the intelligent ship tracking task is performed. First, the static target tracking strategy of the intelligent ship in an obstacle-free environment is learned, and then the autonomous obstacle avoidance strategy of the intelligent ship in a simple obstacle environment is learned. Finally, a meta-initial parameter that can adapt to the new tasks of autonomous obstacle avoidance and target tracking is obtained.
[0053] S202, performing overall training on the autonomous obstacle avoidance and target tracking task model of the intelligent ship in a complex environment, and obtaining correct action strategies for autonomous obstacle avoidance and target tracking of the intelligent ship in different test environments.
[0054] Specifically, in the deep reinforcement learning training process, the uncertainty of the neural network often leads to poor algorithm convergence characteristics, and the result of the training is a strategy that is only suitable for the current task and environment. To address this problem, the present invention introduces meta-learning based on the DDPG algorithm, referred to as meta-reinforcement learning. The idea is to design an internal and external meta-parameter update rule to obtain a set of meta-initial parameters, thereby improving the convergence speed and environmental adaptability of the model in the face of different tasks.
[0055] Here, if a variety of different complex test environments are used as the meta-task set for solving the intelligent ship autonomous obstacle avoidance and target tracking, it will take a lot of time and reduce the engineering efficiency. Therefore, the present invention decomposes a complex and changeable overall task into multiple basic meta-tasks that only complete a single sub-task goal, and they constitute the basic meta-task set T = {T1, T2, ..., T j}, where T j For the jth basic meta-task, create an experience replay pool for each basic meta-task in T They are the state, action, reward of the smart ship at time t in the jth basic meta-task, and the state of the smart ship at time t+1. In the meta-reinforcement learning pre-training process, the smart ship trains each basic meta-task in T in turn to obtain a strategy that can adapt to each sub-task, and finally obtains the meta-initial parameters of the overall learning task.
[0056] In the intelligent ship autonomous obstacle avoidance and target tracking tasks, the basic task set includes two basic meta-tasks: intelligent ship tracking and intelligent ship obstacle avoidance, such as Figure 2As shown in the figure. In pre-training, we first learn the static target tracking strategy of the intelligent ship in an obstacle-free environment, and then learn the autonomous obstacle avoidance strategy of the intelligent ship in a simple obstacle environment, and finally obtain a meta-initial parameter that can adapt to the new tasks of autonomous obstacle avoidance and target tracking. Since the basic meta-tasks are relatively simple, only a small number of training rounds are needed to obtain the prior knowledge, which improves the efficiency of pre-training.
[0057] In general, the algorithm is divided into two parts: pre-training and overall task training. In the training stage, an internal and external meta-parameter update rule is designed, and the internal network training and external meta-parameter update are performed alternately at a certain frequency. The internal network learns each basic meta-task in turn to obtain different internal parameters, and the external meta-parameter update obtains a meta-initial parameter with strong environmental adaptability by optimizing different parameters. In the overall task training, for the autonomous obstacle avoidance and target tracking of intelligent ships under different test environments, the algorithm can quickly converge with only a small number of training episodes to obtain the correct action strategy. The update rule is shown in Formula 8.
[0058]
[0059] Among them, θ meta It is the external meta-parameter of the actor-target neural network in the DDPG algorithm; is the external meta-parameter of the critic target neural network; n is the number of basic meta-tasks to complete training; τ is a constant that controls the speed of meta-parameter update; Meta-task T j Actor-target neural network parameters during training; Meta-task T j The critic target neural network parameters during training. After pre-training, θ meta and is the initial parameter. Therefore, the overall framework of the algorithm is as follows Figure 3 shown.
[0060] In the embodiment of the present invention, a basic meta-task set T={T1, T2} is formed, where T j For the jth basic meta-task, create an experience replay pool for each basic meta-task in T are the state, action, reward of the intelligent ship at time t in the jth basic meta-task and the state of the intelligent ship at time t+1, j=1,2;
[0061] During the pre-training process, the intelligent ship trains each basic meta-task in T in turn to obtain a strategy that can adapt to each sub-task, and finally obtains the meta-initial parameters of the overall task, including: designing internal and external meta-parameter update rules so that internal network training and external meta-parameter update are performed alternately at a certain frequency, the internal network learns each basic meta-task in turn to obtain different internal parameters, and the external meta-parameter update obtains the meta-initial parameters by optimizing different parameters. The update rules are as follows:
[0062]
[0063] S3. Guide the actions of intelligent ships based on the trained model of autonomous obstacle avoidance and target point tracking in complex environments.
[0064] In view of the problems of low training efficiency and poor environmental adaptability of traditional deep reinforcement learning when solving UAV autonomous obstacle avoidance and target tracking tasks, this paper proposes an intelligent ship autonomous obstacle avoidance method based on meta-learning and reinforcement learning, which can bring new research ideas to the study of intelligent ship autonomous obstacle avoidance problems in the navigation field.
[0065] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.
Claims
1. A method for autonomous obstacle avoidance and target tracking of intelligent ships based on meta-learning and reinforcement learning, characterized in that: The following steps are involved: S1. Based on the Markov decision process, a task model of autonomous obstacle avoidance and target point tracking of intelligent ships in complex environments is constructed, including designing a state space, an action space and a reward function that conform to navigation practice. The state space is a collection of the state of the intelligent ship itself and environmental information. The action space is a set of actions that can be executed by the intelligent ship. The reward function includes a tracking reward, an obstacle avoidance reward and a distance reward. The tracking establishment is used to evaluate the tracking effect, the obstacle avoidance reward is used to evaluate the obstacle avoidance effect, and the distance reward is used to evaluate the distance between the intelligent ship and the target point. S2. Training the autonomous obstacle avoidance and target point tracking task model of the intelligent ship in a complex environment based on an improved reinforcement learning algorithm, wherein the improved reinforcement learning algorithm introduces meta-reinforcement learning into the reinforcement learning algorithm, and divides the joint task of autonomous obstacle avoidance and target point tracking of the model training into independent intelligent ship obstacle avoidance tasks and intelligent ship tracking tasks; The specific steps of model training include: S201, pre-training the intelligent ship obstacle avoidance task and the intelligent ship tracking task, first learning the static target tracking strategy of the intelligent ship in an obstacle-free environment, then learning the autonomous obstacle avoidance strategy of the intelligent ship in a simple obstacle environment, and finally obtaining a meta-initial parameter that can adapt to the new tasks of autonomous obstacle avoidance and target tracking, S202, performing overall training on the autonomous obstacle avoidance and target tracking task model of the intelligent ship in a complex environment, and obtaining correct action strategies for autonomous obstacle avoidance and target tracking of the intelligent ship in different test environments; S3. Guide the actions of intelligent ships based on the trained model of autonomous obstacle avoidance and target point tracking in complex environments.
2. According to claim 1, the method for autonomous obstacle avoidance of intelligent ships based on meta-learning and reinforcement learning is characterized in that: The state space is represented as: Among them, (x t ,y t ) represents the location coordinates of the smart ship, Indicates the target location coordinates, d max represents the maximum Euclidean distance between the smart ship and the target, v max Indicates the maximum sailing speed of the smart ship, v t Indicates the navigation speed of the smart ship at time t; The action space is expressed as: A=[-1,1,0] T Among them, "-1" means turn left, "1" means turn right, and "0" means execute; The reward function is expressed as: R=r1+r2+r3 r3=-d*0.1 Among them, r1 is the tracking reward, r2 is the obstacle avoidance reward, the sensing range is preset according to the needs of navigation practice, r3 is the distance reward, and R is the total reward. is the Euclidean distance of obstacles within the sensing range of the smart ship, and d is the Euclidean distance between the smart ship and the target point.
3. According to claim 1, the method for autonomous obstacle avoidance of intelligent ships based on meta-learning and reinforcement learning is characterized in that: Pre-training is performed on the intelligent ship obstacle avoidance task and the intelligent ship tracking task. First, the static target tracking strategy of the intelligent ship in an obstacle-free environment is learned, and then the autonomous obstacle avoidance strategy of the intelligent ship in a simple obstacle environment is learned. Finally, a meta-initial parameter that can adapt to the new tasks of autonomous obstacle avoidance and target tracking is obtained, including: The basic meta-task set T = {T1, T2}, where T j For the jth basic meta-task, create an experience replay pool for each basic meta-task in T are the state, action, reward of the intelligent ship at time t in the jth basic meta-task and the state of the intelligent ship at time t+1, j=1,2; During the pre-training process, the intelligent ship trains each basic meta-task in T in turn to obtain a strategy that can adapt to each sub-task, and finally obtains the meta-initial parameters for learning the overall task.
4. According to claim 3, the method for autonomous obstacle avoidance of intelligent ships based on meta-learning and reinforcement learning is characterized in that: The intelligent ship trains each basic meta-task in T in turn to obtain a strategy that can adapt to each sub-task, and finally obtains the meta-initial parameters of the overall learning task, including: Design internal and external meta-parameter update rules so that internal network training and external meta-parameter update are performed alternately at a certain frequency. The internal network sequentially learns each basic meta-task to obtain different internal parameters. External meta-parameter update obtains meta-initial parameters by optimizing different parameters. The update rules are as follows: Among them, θ meta It is the external meta-parameter of the actor-target neural network in the DDPG algorithm; is the external meta-parameter of the critic target neural network; n is the number of basic meta-tasks to complete training; τ is a constant that controls the speed of meta-parameter update; Meta-task T j Actor-target neural network parameters during training; Meta-task T j Parameters of the critic target neural network during training, j = 1, 2, after pre-training, θ meta and That is the initial parameter.
Citation Information
Cited By
Underactuated ship motion adaptive control method and device based on meta learning
CN120010251A
Unmanned aerial vehicle intelligent scheduling method and system based on knowledge-driven meta learning device
CN120031361A
