Underwater robot path planning method for unknown environment
By introducing an improved reward feedback mechanism and a layered experience pool extraction mechanism in the underwater robot path planning, combined with the reward mechanism of current direction and shortest path principle, the convergence speed of underwater robot path planning technology in complex unknown marine environments is solved. The problem of slow and low success rate achieves higher path planning success rate and faster convergence speed.
Patent Information
- Application Number
- CN202411493430.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-24
- Publication Date
- 2025-02-14
AI Technical Summary
Existing underwater robot path planning technology in complex unknown marine environments has problems of slow convergence speed and low success rate, and existing improvement methods have failed to effectively improve their adaptability in dynamic and complex environments.
The improved reward feedback mechanism and the hierarchical experience pool extraction mechanism are adopted to balance exploration and utilization through the MOA mechanism, and the reward mechanism combining the current direction and the shortest path principle is formed to form a positive feedback loop to improve the success rate and convergence speed of path planning.
It effectively improves the path planning success rate of underwater robots in complex unknown marine environments, accelerates the convergence speed, reduces the adverse effects of imbalance in exploration and utilization capabilities, and improves environmental adaptability.
Smart Images
Figure CN119440072A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of autonomous underwater vehicle path planning, and in particular to a method for underwater robot path planning for an unknown environment. Background Art
[0002] Due to its flexible and autonomous characteristics, underwater robots are widely used in marine environment monitoring, resource exploration, environmental protection and other fields. As the complexity of underwater robot tasks increases, path planning, as one of its core technologies, directly affects the efficiency and safety of task completion. Traditional path planning algorithms mainly rely on prior information of known environments. However, in real marine environments, underwater robots often face dynamic and unknown environments, including complex terrain, unforeseen obstacles, ocean currents and other influencing factors, making it unsatisfactory to plan the path of underwater robots based on prior information.
[0003] With the development of deep learning technology, deep reinforcement learning (DRL) has gradually been applied to the field of path planning. Deep Q Network (DQN) enables underwater robots to autonomously plan paths and cope with complex environmental changes through interactive learning between underwater robots and the environment. However, due to the lack of prior knowledge of underwater robots in the marine environment, the DQN algorithm has a large randomness in the initial exploration stage, resulting in a slow convergence speed. The exploration and utilization mechanism of DQN is difficult to balance. Excessive exploration will increase the time cost of path planning, while excessive utilization may cause the underwater robot to be unable to find the global optimal path. In addition, the complexity of the underwater environment, such as external factors such as ocean currents, has not been fully considered in the existing DQN algorithm, resulting in poor environmental adaptability. In response to the above problems, the prior art optimizes the sample utilization of DQN by introducing a priority experience replay mechanism, or reduces the Q value estimation bias by using Double DQN. However, these improvements are mainly focused on parameter adjustment and strategy optimization within the algorithm, and have limited improvement in the actual adaptability of underwater robots in dynamic and complex underwater environments. In addition, existing improvement methods often ignore the importance of incentive and feedback mechanisms in their designs and fail to effectively promote a virtuous cycle in the learning process of underwater robots.
[0004] Therefore, a method is needed based on the cooperation of improved reward feedback mechanism and hierarchical experience pool extraction mechanism to form a positive feedback loop, so as to improve the success rate of path planning of underwater robots in complex and unknown marine environments. Summary of the invention
[0005] In view of this, the present invention provides a path planning method for an underwater robot in an unknown environment, which solves the problems of slow convergence speed and low success rate of the existing path planning technology of underwater robots in complex and unknown marine environments.
[0006] To this end, the present invention provides the following technical solutions:
[0007] A path planning method for an underwater robot in an unknown environment, comprising:
[0008] Construct an M-DQN underwater robot path planning model;
[0009] Training the underwater robot path planning model, and using the trained model to perform underwater path planning in an unknown environment;
[0010] The model training includes:
[0011] The underwater robot uses the MOA mechanism to select the next action to interact with the underwater environment;
[0012] Reward or punish the current action through the reward mechanism based on the interactive feedback, and store the action experience in the hierarchical experience pool;
[0013] When the action experience accumulates to a preset amount, a batch of action experience is extracted from the hierarchical experience pool using a hierarchical extraction mechanism for learning;
[0014] The above process is repeated until the object touches an obstacle or reaches the target point, and this round of learning ends.
[0015] Each round of learning is evaluated, and rewards are given based on the evaluation results. The action experience that receives the reward is added to the layered experience pool and participates in the next round of training.
[0016] Furthermore, the MOA mechanism includes:
[0017] Determine whether the underwater robot is performing an exploitation action or an exploration action;
[0018] Dynamically adjust the balance between exploration and exploitation of underwater robots:
[0019] MOA (t) =max(MOA min , (MOA max ×p t ))
[0020] Among them, t represents the current episode number, p represents the rate of change of MOA, and MOA max and MOA min Indicates the maximum and minimum MOA values.
[0021] Furthermore, the interactive feedback includes:
[0022] The Euclidean distance between the underwater robot’s current position and the target point and the direction of the ocean current.
[0023] Furthermore, the reward mechanism includes:
[0024] When the underwater robot shortens the Euclidean distance to the target point, a positive reward is given;
[0025] When the underwater robot increases the Euclidean distance from the target point, a penalty is imposed;
[0026] When the underwater robot moves along the direction of the ocean current, it is given a positive reward;
[0027] When the underwater robot moves against the direction of the current, a penalty will be imposed.
[0028] Further, the layered experience pool includes: a primary learning layer, a secondary learning layer, a penalty experience layer, and a target point reward layer;
[0029] For a round of learning process with a good evaluation result, the action experience of the last step to reach the target point is stored in the target point reward layer; all action experiences except the action experience of the last step to reach the target point in this round of learning are stored in the main learning layer;
[0030] The secondary learning layer stores the underwater robot's experience of not touching obstacles and receiving rewards;
[0031] The penalty experience layer stores the experience of actions that are penalized.
[0032] Furthermore, the hierarchical extraction mechanism:
[0033]
[0034] Among them, layer1 represents the main learning layer, layer2 represents the secondary learning layer, layer3 represents the penalty experience layer, layer4 represents the target point reward layer, t represents the current number of training rounds, and Tmax represents the total number of training rounds.
[0035] Furthermore, the evaluation of each round of learning and reward feedback based on the evaluation results include:
[0036] After a round of learning is completed, the learning process of this round is evaluated, and the evaluation result is good or average;
[0037] If the evaluation result is good, the action experience of this round will be rewarded after the round ends;
[0038] The good result means that the underwater robot reaches the target point and obtains a reward for each action;
[0039] The general description means everything except good.
[0040] Advantages and positive effects of the present invention:
[0041] The present invention promotes the virtuous cycle of deep network training by introducing a reward feedback mechanism and an experience pool stratification mechanism, effectively improves the learning efficiency of underwater robots, and avoids the problem of insufficient path planning accuracy caused by insufficient training in traditional methods. Among them, the reward feedback mechanism and experience pool stratification mechanism designed based on marketing psychology can effectively improve the path planning success rate of underwater robots in complex and unknown marine environments, accelerate the convergence speed, and reduce the adverse effects caused by the imbalance of exploration and utilization capabilities in existing deep Q learning methods. And combined with the reward mechanism of ocean currents and the shortest path principle, it can significantly improve the environmental adaptability of underwater robots, allowing them to navigate stably under changing marine conditions. BRIEF DESCRIPTION OF THE DRAWINGS
[0042] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.
[0043] Figure 1 A flow chart of a method in an embodiment of the present invention;
[0044] Figure 2 The motion state and ocean current direction of the underwater robot in the embodiment of the present invention;
[0045] Figure 3 This is a flow chart of the reward feedback mechanism in a single training process in an embodiment of the present invention;
[0046] Figure 4 The following are the experimental results of unknown environment path planning in the embodiment of the present invention. DETAILED DESCRIPTION
[0047] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.
[0048] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.
[0049] The present invention provides a path planning method for underwater robots in unknown environments. First, a mathematical optimization acceleration function is designed to balance the exploration and utilization capabilities in deep Q learning. Secondly, in order to improve the environmental adaptability of underwater robots and accelerate the convergence of deep networks, a reward mechanism that takes into account the influence of ocean currents and the shortest path principle is proposed. Finally, a reward feedback and experience stratification mechanism is designed based on the principles of marketing psychology, which promotes a virtuous cycle between the deep network training process and the underwater robot learning process.
[0050] Combination Figure 1 As shown, the method steps of the present invention include:
[0051] S1. The underwater robot is completely unknown to the environmental information. The amount of experience of the underwater robot is initialized to 0, and the weights of the neural network that generates the experience value are randomly generated.
[0052] S2, Mathematical optimizer acceleration function mechanism: When the underwater robot arrives at the initial point, it first uses the MOA mechanism to select the next action to be performed. MOA will guide the underwater robot to follow the experience neural network for development actions or blind exploration actions at different stages of underwater robot learning.
[0053] Among them, the parameters of MOA are gradually adjusted during the training process. In the early stage of training, the MOA value is low, which enhances the exploration ability. As the training progresses, the MOA value gradually increases, making more use of existing experience, thereby improving the convergence accuracy and speed of path planning. The MOA mechanism is used to dynamically adjust the balance between exploration and development of underwater robots.
[0054] The change of MOA with the number of training rounds is shown in the formula:
[0055] MOA (t) =max(MOA min , (MOA max ×p t ))
[0056] Among them, t represents the current episode number, p represents the rate of change of MOA, and MOA max and MOA min represents the maximum and minimum values of MOA, which are set to 0.01 and 0.2 respectively in this embodiment according to the basic DQN.
[0057] S3. Dynamic reward mechanism: The underwater robot receives corresponding rewards or penalties based on its direction of movement relative to the ocean current and the difference in the Euclidean distance from the target point. When the underwater robot moves in the direction of the ocean current and shortens the Euclidean distance from the target point, it receives positive rewards; if it moves against the ocean current or increases the Euclidean distance from the target point, it receives penalties.
[0058] Specifically, the reward mechanism calculates the difference in the Euclidean distance between the current position of the underwater robot and the target point for feedback. When the underwater robot approaches the target point, a positive reward is given; when the underwater robot moves away from the target point, a penalty is given. This reward mechanism takes into account the direction and strength of the ocean current. It is set that when the underwater robot moves along the direction of the ocean current, a positive reward is given; if the underwater robot moves against the direction of the ocean current, it will be punished, thereby guiding the underwater robot to make full use of the thrust of the ocean current to save energy and accelerate to the target point. The reward and penalty method is shown in the formula:
[0059]
[0060] Among them, S represents the current state of the autonomous underwater vehicle; S=1 means that the underwater robot has touched an obstacle, and S=done means that the underwater robot has reached the target point. d2 represents the current Euclidean distance from the underwater robot to the target point, and d1 represents the previous Euclidean distance to the target point. Current=false and Current=done represent the upstream and downstream behaviors of the underwater robot, respectively.
[0061] S4. After completing each step, if the underwater robot's experience pool has accumulated enough experience, the underwater robot will extract batch experience from the experience pool for learning according to the hierarchical mechanism in S7.
[0062] S5. The underwater robot interacts with the environment according to the above rules until it touches an obstacle or reaches the target point, ending this round of learning.
[0063] S6. Reward and feedback mechanism based on marketing psychology: The reward and feedback mechanism is based on the "reward psychology" in marketing psychology. After the underwater robot finishes a round of learning, it will evaluate the learning process of this round. If the evaluation results are divided into "good" and "average". For a round of learning process that is evaluated as good, the system will provide additional feedback rewards for the experience learned after the round. This reward and feedback mechanism simulates the feedback strategies provided by operators in the market to obtain consumers' reward psychology, such as the "year-end feedback" behavior. It aims to increase the "purchasing power" of these "consumers".
[0064] S7. Experience pool stratification mechanism:
[0065] All interaction experiences with the environment are stored in the experience pool; experiences are classified and stored in different layers according to their contribution to the underwater robot path planning. The experience pool is divided into four layers: the main learning layer, the secondary learning layer, the penalty experience layer, and the target point reward layer.
[0066] The main learning layer stores experiences with high rewards, which represent successful path planning behaviors. The underwater robot will reuse these experiences first, which include more experiences with rewards.
[0067] The secondary learning layer stores experiences with medium rewards, which do not yield significant returns but are valuable for path planning.
[0068] The penalty experience layer stores the behavioral experience that caused the underwater robot to be punished, and is used to warn the system to avoid similar mistakes in the future.
[0069] The target point reward layer stores the experience of successfully reaching the target point, and these experiences will gradually be used more in the underwater robot's decision-making process as the training progresses.
[0070] The extraction ratio of each layer is designed as follows:
[0071]
[0072] Among them, the experience pool has four layers, t represents the current number of training rounds, and Tmax represents the total number of training rounds.
[0073] The experience pool stratification strategy mainly utilizes the "affinity psychology" in marketing psychology, that is, the experience of gaining feedback will increase loyalty to the brand, so there will be a greater probability of participating in the next round of learning process. After the experience is classified and stored in different levels, it returns to S2 to start a new round of learning process.
[0074] S8. Positive feedback loop based on reward feedback and experience stratification:
[0075] The rewards and experience pool work together to form a positive feedback loop. Every time the underwater robot successfully plans an effective path, the system will update the current reward value and reinforce the successful behavior through the reward feedback mechanism.
[0076] In each round of training, the underwater robot continuously extracts valuable experience from the experience pool for review and learning. As the number of training rounds increases, the underwater robot's ability to adapt to the environment gradually increases, and the path planning accuracy and convergence speed also increase accordingly.
[0077] By introducing the "reward psychology" and "affinity psychology" in marketing psychology, we build a reward feedback mechanism and experience pool stratification strategy, including:
[0078] Reward feedback mechanism: Combining the concept of "reward psychology", for a round of training process that shows excellent path planning behavior, the system provides additional feedback rewards for all its experience after the end of this round. This reward feedback mechanism simulates the satisfaction of consumers in the market after receiving feedback rewards, increases the underwater robot's enthusiasm for learning effective experience, and thus improves the efficiency of experience utilization.
[0079] Experience pool stratification strategy: Based on "affinity psychology", the experience in the experience pool is stratified according to its learning value. The experience pool is divided into four layers, namely:
[0080] Layer 1: The main learning layer stores all experiences that have received high rewards, which are learned by the underwater robot first;
[0081] Layer 2: Secondary learning layer, storing experiences where the underwater robot did not touch obstacles but did not receive significant feedback;
[0082] Layer 3: Penalty experience layer, which stores experiences that lead to penalties and is used to warn underwater robots to avoid similar mistakes;
[0083] Layer 4: Target point reward layer, which stores the experience of all underwater robots successfully reaching the target point. These experiences will be gradually utilized more as the number of training rounds increases.
[0084] During the learning process, the underwater robot extracts batch experience from each layer for learning according to the following formula:
[0085]
[0086] Among them, the experience pool has four layers, t represents the current number of training rounds, and Tmax represents the total number of training rounds.
[0087] Through this hierarchical strategy, different types of experience in the experience pool are selectively used at different learning stages to accelerate convergence and improve the adaptability of underwater robots in different environments. The reward feedback mechanism and the experience pool hierarchical strategy form a virtuous cycle in the learning process of underwater robots. Experience that shows excellent path planning behavior will receive reward feedback, and the feedback experience will be used intensively.
[0088] During the training process, each layer of the experience pool will be dynamically adjusted according to the progress of the algorithm. As the number of training rounds increases, the frequency of experience usage in the main learning layer and the target point reward layer will gradually increase, while the frequency of usage in the penalty experience layer will gradually decrease, thereby improving the learning efficiency and path planning ability of the underwater robot.
[0089] Based on the above method, the effect of the method of the present invention is further illustrated in combination with specific tests:
[0090] S1. The underwater robot is completely unknown to the environmental information. The amount of experience of the underwater robot is initialized to 0, and the weights of the neural network that generates the experience value are randomly generated.
[0091] S2, Mathematical optimizer acceleration function mechanism: Place the underwater robot at the initial point, and first use the MOA mechanism to select the next action to be performed. MOA will guide the underwater robot to follow the experience neural network for development actions or blind exploration actions at different stages of underwater robot learning. The MOA mechanism is used to dynamically adjust the balance between exploration and utilization of the underwater robot.
[0092] S3. Dynamic reward mechanism: The underwater robot receives corresponding rewards or penalties based on its direction of movement relative to the ocean current and the difference in the Euclidean distance from the target point. Specifically, when the underwater robot moves in the direction of the ocean current and shortens the Euclidean distance from the target point, it receives positive rewards; if it moves against the ocean current or increases the Euclidean distance from the target point, it receives penalties. Figure 2 shown.
[0093] S4. After completing each step, if the experience in the experience pool accumulates enough, the underwater robot will extract batch experience from the experience pool according to the hierarchical mechanism for learning.
[0094] S5. The underwater robot interacts with the environment according to the above rules until it touches an obstacle or reaches the target point, ending this round of learning.
[0095] S6. Reward and feedback mechanism based on marketing psychology: The reward and feedback mechanism is based on the "reward psychology" in marketing psychology. Figure 3As shown in the figure. After the underwater robot finishes a round of learning, the learning process of this round is evaluated. If the evaluation results of this round are divided into "good" and "average". If a round of learning process is evaluated as a good one, the system will provide additional feedback rewards for the experience learned after the end of this round. This reward feedback mechanism simulates the feedback strategy provided by operators in the market to obtain consumer reward psychology, such as the "year-end feedback" behavior. It aims to increase the "purchasing power" of these "consumers".
[0096] S7. Experience pool stratification mechanism:
[0097] All interaction experiences with the environment are stored in the experience pool, which is classified and stored in different layers according to its contribution to the underwater robot path planning. The experience pool is divided into four layers: the main learning layer, the secondary learning layer, the penalty experience layer, and the target point reward layer.
[0098] The main learning layer stores experiences with high rewards, which represent successful path planning behaviors. The underwater robot will reuse these experiences first, which include more experiences with rewards.
[0099] The secondary learning layer stores experiences with medium rewards, which do not yield significant returns but are valuable for path planning.
[0100] The penalty experience layer stores the behavioral experience that caused the underwater robot to be punished, and is used to warn the system to avoid similar mistakes in the future.
[0101] The target point reward layer stores the experience of successfully reaching the target point, and these experiences will gradually be used more in the underwater robot's decision-making process as the training progresses.
[0102] The extraction ratio of each layer is designed as follows:
[0103]
[0104] The experience pool stratification strategy mainly utilizes the "affinity psychology" in marketing psychology, that is, the experience of gaining feedback will increase loyalty to the brand, so there will be a greater probability of participating in the next round of learning process. After the experience is classified and stored in different levels, return to step 2 to start a new round of learning process.
[0105] S8. Positive feedback loop based on reward feedback and experience stratification:
[0106] The rewards and experience pool work together to form a positive feedback loop. Every time the underwater robot successfully plans an effective path, the system will update the current reward value and reinforce the successful behavior through the reward feedback mechanism.
[0107] In each round of training, the underwater robot continuously extracts valuable experience from the experience pool for review and learning. As the number of training rounds increases, the underwater robot's ability to adapt to the environment gradually increases, and the path planning accuracy and convergence speed also increase accordingly.
[0108] The optimal model saved in the above steps is used to test the path planning of the underwater robot in an unknown ocean environment. The test results are as follows: Figure 4 And as shown in Table 1.
[0109] Table 1
[0110]
[0111] The above test results show that the MDQN method has excellent path planning capabilities in unknown ocean environments, effectively solving the problems of slow convergence speed and low success rate in existing methods.
[0112] As an important branch of consumer behavior research, marketing psychology focuses on guiding individual decision-making through incentives, feedback, and psychological expectations. This invention applies the incentive mechanism and feedback mechanism in marketing psychology to deep Q learning, which can effectively improve the convergence speed of the algorithm and the ability to balance exploration and utilization. By optimizing the reward mechanism and combining the influence of marine environmental characteristics such as ocean currents, the path planning ability of underwater robots in complex and unknown environments can be further improved, their environmental adaptability can be enhanced, and the overall mission success rate can be improved.
[0113] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.
Claims
1. A method for underwater robot path planning in an unknown environment, characterized in that: include: Construct an M-DQN underwater robot path planning model; Training the underwater robot path planning model, and using the trained model to perform underwater path planning in an unknown environment; The model training includes: The underwater robot uses the MOA mechanism to select the next action to interact with the underwater environment; Reward or punish the current action through the reward mechanism based on the interactive feedback, and store the action experience in the hierarchical experience pool; When the action experience accumulates to a preset amount, a batch of action experience is extracted from the hierarchical experience pool using a hierarchical extraction mechanism for learning; The above process is repeated until the object touches an obstacle or reaches the target point, and this round of learning ends. Each round of learning is evaluated, and rewards are given based on the evaluation results. The action experience that receives the reward is added to the layered experience pool and participates in the next round of training.
2. According to claim 1, a method for underwater robot path planning for an unknown environment is characterized in that: The MOA mechanism includes: Determine whether the underwater robot is performing an exploitation action or an exploration action; Dynamically adjust the balance between exploration and exploitation of underwater robots: MOA (t) =max(MOA min ,(MOA max ×p t )) Among them, t represents the current episode number, p represents the rate of change of MOA, and MOA max and MOA min Indicates the maximum and minimum MOA values.
3. The path planning method for an underwater robot in an unknown environment according to claim 1 is characterized in that: The interactive feedback includes: The Euclidean distance between the underwater robot’s current position and the target point and the direction of the ocean current.
4. The method for underwater robot path planning for an unknown environment according to claim 1 is characterized in that: The reward mechanism includes: When the underwater robot shortens the Euclidean distance to the target point, a positive reward is given; When the underwater robot increases the Euclidean distance from the target point, a penalty is imposed; When the underwater robot moves along the direction of the ocean current, it is given a positive reward; When the underwater robot moves against the direction of the current, a penalty will be imposed.
5. The method for underwater robot path planning for an unknown environment according to claim 1 is characterized in that: The layered experience pool includes: a primary learning layer, a secondary learning layer, a penalty experience layer, and a target point reward layer; For a round of learning process with a good evaluation result, the action experience of the last step to reach the target point is stored in the target point reward layer; all action experiences except the action experience of the last step to reach the target point in this round of learning are stored in the main learning layer; The secondary learning layer stores the underwater robot's experience of not touching obstacles and receiving rewards; The penalty experience layer stores the experience of actions that are penalized.
6. The method for underwater robot path planning for an unknown environment according to claim 1 is characterized in that: The hierarchical extraction mechanism: Among them, layer1 represents the main learning layer, layer2 represents the secondary learning layer, layer3 represents the penalty experience layer, layer4 represents the target point reward layer, t represents the current number of training rounds, and Tmax represents the total number of training rounds.
7. The method for underwater robot path planning for an unknown environment according to claim 1 is characterized in that: Each round of learning is evaluated and rewards are given based on the evaluation results, including: After a round of learning is completed, the learning process of this round is evaluated, and the evaluation result is good or average; If the evaluation result is good, the action experience of this round will be rewarded after the round ends; The good result means that the underwater robot reaches the target point and each action is rewarded; The general description means all cases except good.