A method, electronic device, and medium for underwater robot path planning in unknown dynamic environments

Abstract

The present invention discloses a path planning method, electronic device, and medium for an underwater robot in an unknown dynamic environment, comprising: an intelligent agent selecting an action according to the output of a current Q network, interacting with the environment, and obtaining a state; considering safety constraints in path planning, processing the state to obtain first-category guidance information and second-category guidance information, thereby updating the state; calculating a reward according to the updated state; storing the action, state, and reward in an experience replay pool; repeating the above process until the amount of data in the experience replay pool reaches a threshold; extracting a batch of experience data from the experience replay pool, inputting them into a Q network and a target Q network, respectively, and calculating a Q value and a target Q value; minimizing a loss function using a gradient descent method to update the parameters of the Q network; copying the parameters of the Q network to the target Q network; obtaining a trained target Q network until the loss function converges; configuring a test environment, and using the trained target Q network to output a planned path.

Description

Technical Field

[0001] The present invention relates to the technical field of underwater robot path planning, and in particular to an underwater robot path planning method, electronic equipment, and medium for an unknown dynamic environment. Background Art

[0002] With the deepening of oceanographic research, the demand for autonomous navigation and path planning for underwater robots in complex marine environments is increasing. These environments are characterized by a high degree of uncertainty, including variations in the speed and direction of ocean currents, and the number, location, and motion characteristics of obstacles. These factors significantly increase the robots' reliance on environmental information and complicate information acquisition and processing. Therefore, developing efficient and stable path planning algorithms is crucial for underwater robots to autonomously avoid obstacles and find optimal paths.

[0003] Traditional path planning algorithms (such as those based on graph search or shortest path optimization) perform well in static environments but have significant limitations in dynamic ocean environments. These algorithms typically have long computation times and limited flexibility, making them difficult to adapt to real-time environmental changes. Furthermore, they lack robustness to interference, making them prone to decision errors and path deviations in highly uncertain environments, impacting the robot's safety and mission performance.

[0004] Compared to traditional methods, deep learning-based methods demonstrate a powerful ability to learn complex features and adapt to dynamic environments, improving the accuracy and efficiency of path planning. Currently, in the field of underwater robot path planning, there are methods based on reinforcement learning, methods based on deep neural networks, and deep reinforcement learning methods that combine the two. These methods are widely used for path planning tasks in complex dynamic environments because they can simultaneously process high-dimensional state spaces and learn long-term strategies. However, the direct application of deep reinforcement learning to underwater robot path planning still faces the following difficulties:

[0005] (1) Most current deep reinforcement learning-based methods require learning and decision-making in a high-dimensional state space. However, in real complex environments, the dimensionality of the state space is often very high. Therefore, as the dimensionality of the environment increases, the state space will explode, resulting in reduced sample efficiency and a sharp increase in computational complexity. This makes it extremely difficult to directly apply deep reinforcement learning methods in high-dimensional environments. It is necessary to consider how to effectively reduce and represent the high-dimensional state space to reduce the computational burden and improve learning efficiency.

[0006] (2) In many real-world task environments, reward signals are often very sparse, meaning that the reward is zero in most states, and positive rewards are only obtained when a specific target state is reached. This sparse reward characteristic makes it difficult for the agent to effectively learn valuable behavioral strategies, because most exploratory behaviors do not receive clear feedback, resulting in a slow and inefficient learning process. Therefore, when applying deep reinforcement learning methods, it is necessary to consider how to overcome sparse rewards and design more effective reward mechanisms to promote learning.

[0007] (3) Most current deep reinforcement learning methods can achieve good performance in the training environment, but their generalization ability is often poor when faced with new, unseen environments. This is because deep reinforcement learning models tend to memorize the characteristics of a specific environment rather than learning truly universal strategies. In practical applications, the environment may change or completely new situations may appear, and current methods have difficulty adapting to such changes, and are prone to a sharp decline in performance. Summary of the Invention

[0008] In view of the shortcomings of the existing technology, the present invention provides a path planning method, electronic equipment and medium for an underwater robot in an unknown dynamic environment.

[0009] In a first aspect, an embodiment of the present invention provides a method for underwater robot path planning in an unknown dynamic environment, the method comprising the following steps:

[0010] The agent selects an action based on the output of the current Q network, interacts with the environment, and obtains a state. Taking into account the safety constraints in path planning, the agent processes the state to obtain first- and second-category guidance information, thereby updating the state. Based on the updated state, the agent calculates the reward. The action, state, and reward are stored in the experience replay pool. This process is repeated until the amount of data in the experience replay pool reaches a threshold.

[0011] A batch of experience data is extracted from the experience replay pool and input into the Q network and the target Q network respectively to calculate the Q value and the target Q value; the gradient descent method is used to minimize the loss function to update the Q network parameters; the loss function is used to calculate the difference between the Q value and the target Q value; after each unit training cycle, the parameters of the Q network are copied to the target Q network; until the loss function converges, a trained target Q network is obtained;

[0012] Configure the test environment and use the trained target Q network output to plan the path.

[0013] In a second aspect, an embodiment of the present invention provides an electronic device comprising a memory and a processor, wherein the memory is coupled to the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the above-mentioned underwater robot path planning method for unknown dynamic environments.

[0014] In a third aspect, an embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-mentioned underwater robot path planning method for an unknown dynamic environment.

[0015] In a fourth aspect, an embodiment of the present invention provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the above-mentioned underwater robot path planning method for an unknown dynamic environment.

[0016] Compared with the prior art, the present invention has the following beneficial effects:

[0017] The present invention provides a path planning method for underwater robots in unknown dynamic environments. It uses a dual deep Q-network (DDQN) to process the state and obtain a heuristic guidance mechanism and reward function for the first type of guidance information and the second type of guidance information, which significantly improves the efficiency and safety of path planning. In addition, the method of the present invention can adapt to complex and changing environmental conditions. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0019] Figure 1 This is an overall schematic diagram of an underwater robot path planning method, electronic equipment, and medium for an unknown dynamic environment provided by an embodiment of the present invention;

[0020] Figure 2 is a schematic diagram of a deep reinforcement learning network model provided by an embodiment of the present invention;

[0021] Figure 3 This is a graph showing the experimental results of the method provided by the present invention for multi-task planning in a medium-scale environment;

[0022] Figure 4 This is a graph showing the experimental results of the method provided by the present invention for multi-task planning in a large-scale environment;

[0023] Figure 5is a schematic diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0024] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0025] It should be noted that, unless there is any conflict, the features in the following embodiments and implementations may be combined with each other.

[0026] like Figure 1 and Figure 2 As shown, an embodiment of the present invention provides a path planning method for an underwater robot in an unknown dynamic environment, the method comprising:

[0027] In step S1, the agent selects an action based on the output of the current Q network, interacts with the environment, and obtains a state; considers the safety constraints in path planning, processes the state, obtains the first type of guidance information and the second type of guidance information, and thus updates the state; calculates the reward based on the updated state; stores the action, state, and reward in the experience replay pool; and repeats the above process until the amount of data in the experience replay pool reaches a threshold.

[0028] The process of configuring the environment specifically includes: the environment includes the target position and initial position of the planned path, ocean current information, and obstacle information; in this example, the Canvas control in Tkinter is used to perform visual simulation of the underwater environment, the ocean current information includes the direction and speed of the current, and the obstacle information includes the number, size, position, movement direction and speed of obstacles.

[0029] Furthermore, in this example, taking into account the motion characteristics and capabilities of the underwater robot, its heading is selected at a fixed time and fixed speed, with the current heading being 0° and an interval of 45° left and right as one action. The action set is {-90°, -45°, 0°, 45°, 90°}, a total of five actions.

[0030] The process of processing the state to obtain the first type of guidance information and the second type of guidance information, thereby updating the state, taking into account the safety constraints in path planning, includes:

[0031] Obtain current basic status information, including the current speed of the underwater robot, ocean current speed, and the relative distance and angle between the underwater robot and the target. Also included is obstacle distance information for eight directions within the observation range (with the current heading as 0° and intervals of 45°).

[0032] During the discretization process, considering the safety constraints in path planning, obstacle distance information is divided into three categories based on safety: less than the collision distance, less than the observation distance, and greater than the observation distance. These three types of distance information are represented by the values ​​-1, 0, and 1, respectively. This discretizes the obstacle distance information in the eight directions and obtains the first type of guidance information.

[0033] Based on the first type of guidance information, by predicting the next action and comparing the positions after executing different actions, the action closest to the target position is selected, and the direction corresponding to the action in the first type of guidance information is marked as 1, and other directions are marked as 0 to obtain the second type of guidance information;

[0034] The first type of guidance information, the second type of guidance information and the basic status information are merged to update the status.

[0035] For example, at time t, it is assumed that the obstacle distance information in the basic state information is {D, D, D, D, D, D, D}, where the observation distance is d de , D is a value greater than d de The value of indicates that no obstacles are observed within the observation distance in this direction. is a value less than the detection distance d de But it is greater than the collision distance d coll The value of =({1, 1, 1, 0, 1, 1, 1}) indicates that an obstacle has been observed within the observation distance in that direction. Therefore, the first type of guidance information is obtained, corresponding to {1, 1, 1, 0, 1, 1, 1}. This means that an obstacle has been observed at the fourth direction, 135°, but no obstacles are observed at the remaining directions. Based on the first type of guidance information, considering the underwater robot's five optional actions, the position reached by taking the action in each feasible direction under the influence of the ocean current is calculated. This position and the position of the target point are then calculated. Assuming that the directions 0° and -90° are closest to the target point, the second type of guidance information is {1, 0, 0, 0, 0, 0, 1}. Finally, the basic state information, the first type of guidance information, and the second type of guidance information together constitute the proposed heuristic guidance mechanism based on discretization of value mappings, which can effectively improve the learning efficiency and convergence of the algorithm for continuous raw observation data in the state space. The extended information of the first and second types of guidance information helps accelerate the convergence of the algorithm and enhance the efficiency of task planning.

[0036] The reward calculation process includes:

[0037]

[0038] in:

[0039] r1=Δd*success_reward / d cur

[0040]

[0041]

[0042] r3=k1 cosθ

[0043] d cur =||P agent_now -P goal ||, d bef =||P agent_before -P goal ||

[0044] Δd=d cur -d bef

[0045] Where r1 is the single-step reward, which is used to measure the distance d between the current target position and the underwater robot. cur Under this condition, the reward brought by the distance change Δd corresponding to a single-step action; d cur and d bef Represent the distance between the underwater robot and the target position at the current moment and the previous moment respectively; P agent_now represents the current position of the underwater robot, P agent_before represents the position of the underwater robot at the previous moment, P goal Indicates the location of the target point; The distance between the underwater robot and the obstacle at different angles and directions Evaluate the collision reward corresponding to the distance, r2 is the smallest The maximum value is 0; is the distance between the underwater robot and the obstacle at the i-th position in the basic state information, d coll Represents the collision distance, d de represents the detection distance; θ is the angle between the heading corresponding to the current action of the underwater robot and the heading of the ocean current; r3 represents the reward for the utilization of the ocean current, and k1 represents the weight parameter.

[0046] Step S2: extract a batch of experience data from the experience replay pool and input them into the Q network and the target Q network respectively to calculate the Q value and the target Q value; use the gradient descent method to minimize the loss function to update the Q network parameters; the loss function is used to calculate the difference between the Q value and the target Q value; every unit training cycle, the parameters of the Q network are copied to the target Q network; until the loss function converges, a trained target Q network is obtained.

[0047] Step S3: Configure the test environment and use the trained target Q network to output the planned path.

[0048] In this example, tests were conducted in different underwater simulation environments, focusing on indicators such as path planning efficiency, success rate, path length, and energy consumption, to evaluate the generalization ability and stability of the network. Multiple test environments with different difficulties and characteristics were designed, including but not limited to:

[0049] Obstacle-free static environment: Verify the basic performance of path planning;

[0050] Dynamic obstacle environment: Verify the path planning effect under the conditions of moving obstacles and water flow interference;

[0051] Complex obstacle environment: Tests the robustness of path planning under multiple dynamic obstacles (speed, direction, and initial position are all random values) and complex ocean currents (ocean current speed and direction are all random values).

[0052] Multi-target environment: Set up sequential access tasks, with the previous task target point as the starting point for the next task.

[0053] With an environment size of 250×300, random ocean current speed + fixed number of obstacles + random dynamic obstacles are used as the training environment, with an initial learning rate of 0.0003, a discount factor of 0.9, an experience replay buffer size of 20,000, a batch size of 512, a target network update frequency of 5 epochs, and an epsilon-greedy policy parameter of 0.1.

[0054] To verify the reliability and generalizability of this method, we constructed a series of test environments with varying characteristics. We systematically adjusted environmental parameters, including the scale of the environment, the speed and direction of ocean currents, the number and size of obstacles, the initial positions of obstacles, and their speed and direction. Monte Carlo tests of path planning with random starting and ending points were conducted in these environments with varying parameter combinations, 100 times for each combination. Sequential multi-task planning with random starting and ending points was also tested. The experimental results are detailed in Table 1.

[0055] Table 1: Planning results of the algorithm in this embodiment under different test environments

[0056]

[0057] To further evaluate the performance advantages of this method, we conducted in-depth testing in two representative complex environments: a medium-scale environment and a large-scale environment (Environment 5 and Environment 7 in Table 1 were both set to random ocean current speeds and random moving obstacles). In these two environments, we focused on key performance indicators such as path planning success rate, planned path length, and single-step calculation time. Specific test data is shown in Table 2.

[0058] Table 2: Planning results of the algorithm in this embodiment under the test environment

[0059] environment Success rate Path length (step) Single-step planning time Environment 5 97％ 438 0.0003s Environment 7 96％ 834 0.0003s

[0060] Sequential multi-task planning is performed under the settings of environment 5 and environment 7 respectively. The number of tasks to be planned is 6. Except for the starting point, the target point of the previous planning is used as the starting point of the subsequent planning. The final planning results are shown in Figure 3 and Figure 4 The complete planning result may show an intersection with the final obstacle location due to the dynamic nature of the environment. That is, due to the constant changes in the speed and direction of the obstacle, the original feasible path is occupied by the obstacle.

[0061] In summary, the underwater robot path planning method based on deep reinforcement learning proposed in the present invention utilizes a dual deep Q network (DDQN) combined with a heuristic guidance mechanism and a new hybrid reward function to significantly improve the efficiency and safety of path planning. Tests conducted in different underwater simulation environments show that the method has good generalization capabilities and can adapt to complex and changing environmental conditions. Specifically, Table 1 verifies the generalization performance of the algorithm and shows its adaptability in multiple types of complex environments, while Table 2 lists in detail the various performance indicators in specific environments, further supporting the effectiveness of the method in intelligent autonomous navigation tasks. These results show that the proposed method not only excels in planning efficiency, but also can effectively avoid obstacles in terms of safety, laying a solid foundation for the application of underwater robots in dynamic environments.

[0062] Accordingly, the present application also provides an electronic device, comprising: one or more processors; a memory for storing one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors implement the above-mentioned underwater robot path planning method for an unknown dynamic environment. Figure 5 As shown in FIG, a hardware structure diagram of any device with data processing capability in the underwater robot path planning method for unknown dynamic environments provided by an embodiment of the present invention, except Figure 5In addition to the processor, memory, and network interface shown, any device with data processing capabilities in which the apparatus in the embodiment is located may also include other hardware, generally based on the actual functions of the device with data processing capabilities, which will not be described in detail.

[0063] Accordingly, the present application also provides a computer-readable storage medium having computer instructions stored thereon, which, when executed by a processor, implement the underwater robot path planning method for an unknown dynamic environment as described above. The computer-readable storage medium can be an internal storage unit of any device with data processing capabilities described in any of the aforementioned embodiments, such as a hard disk or memory. The computer-readable storage medium can also be an external storage device, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), an SD card, a flash card (Flash Card), etc. equipped on the device. Furthermore, the computer-readable storage medium can also include both an internal storage unit and an external storage device of any device with data processing capabilities. The computer-readable storage medium is used to store the computer program and other programs and data required by any device with data processing capabilities, and can also be used to temporarily store data that has been output or is to be output.

[0064] The above embodiments are intended only to illustrate the design concepts and features of the present invention. Their purpose is to enable those skilled in the art to understand the contents of the present invention and implement them accordingly. The scope of protection of the present invention is not limited to the above embodiments. Therefore, any equivalent changes or modifications made based on the principles and design concepts disclosed in the present invention are within the scope of protection of the present invention.

Claims

1. A path planning method for underwater robots in unknown dynamic environments, characterized in that: The method comprises the following steps: The agent selects an action based on the output of the current Q network, interacts with the environment, and obtains a state. Taking into account the safety constraints in path planning, the agent processes the state to obtain first- and second-category guidance information, thereby updating the state. Based on the updated state, the agent calculates the reward. The action, state, and reward are stored in the experience replay pool. This process is repeated until the amount of data in the experience replay pool reaches a threshold. A batch of experience data is extracted from the experience replay pool and input into the Q network and the target Q network respectively to calculate the Q value and the target Q value; the gradient descent method is used to minimize the loss function to update the Q network parameters; the loss function is used to calculate the difference between the Q value and the target Q value; after each unit training cycle, the parameters of the Q network are copied to the target Q network; until the loss function converges, a trained target Q network is obtained; Configure the test environment and use the trained target Q network output to plan the path; The process of processing the state to obtain the first type of guidance information and the second type of guidance information, thereby updating the state, taking into account the safety constraints in path planning, includes: Obtaining current basic status information, including: current speed of the underwater robot, ocean current speed, relative distance and angle information between the underwater robot and the target, and distance information of obstacles within the observation range; Considering the safety constraints in path planning, the distance information of obstacles is divided into three categories according to safety: less than the collision distance, less than the observation distance, and greater than the observation distance. These are represented by the label values ​​-1, 0, and 1, respectively. The obstacle distance information in the eight directions is discretized to obtain the first type of guidance information. Based on the first type of guidance information, by predicting the next action and comparing the positions after executing different actions, the action closest to the target position is selected, and the direction corresponding to the action in the first type of guidance information is marked as 1, and other directions are marked as 0 to obtain the second type of guidance information; The first type of guidance information, the second type of guidance information and the basic status information are merged to update the status.

2. The underwater robot path planning method for an unknown dynamic environment according to claim 1, characterized in that: The environment includes the target position and initial position of the planned path, ocean current information, and obstacle information.

3. The underwater robot path planning method for an unknown dynamic environment according to claim 2, characterized in that: The ocean current information includes the direction and speed of the ocean current, and the obstacle information includes the number, size, location, movement direction and speed of obstacles.

4. The underwater robot path planning method for an unknown dynamic environment according to claim 1, characterized in that: The obstacle distance information within the observation range is: Obstacle distance information at eight directions within the observation range; the eight directions are obtained by setting an observation direction at every 45° interval with the current heading as 0°.

5. The underwater robot path planning method for an unknown dynamic environment according to claim 1, characterized in that: The action is based on the current heading being 0°, and the left and right intervals being 45° as one action, and the action set is set to {-90°, -45°, 0°, 45°, 90°}.

6. The underwater robot path planning method for an unknown dynamic environment according to claim 1, characterized in that: The reward calculation process includes: in: r1=Δd*success_reward / d cur r3=k1cosθ d cur =||P agent_now -P goal ||,d bef =||P agent_before -P goal || Δd=d cur -d bef Where r1 is the single-step reward, which is used to measure the distance d between the current target position and the underwater robot. cur Under this condition, the reward brought by the distance change Δd corresponding to a single-step action; d cur and d bef Represent the distance between the underwater robot and the target position at the current moment and the previous moment respectively; P agent_now represents the current position of the underwater robot, P agent_before represents the position of the underwater robot at the previous moment, P goal Indicates the location of the target point; The distance between the underwater robot and the obstacle at different angles and directions Evaluate the collision reward corresponding to the distance, r2 is the smallest The maximum value is 0; is the distance between the underwater robot and the obstacle at the i-th position in the basic state information, d coll Represents the collision distance, d de represents the detection distance; θ is the angle between the heading corresponding to the current action of the underwater robot and the heading of the ocean current; r3 represents the reward for the utilization of the ocean current, and k1 represents the weight parameter.

7. An electronic device comprising a memory and a processor, characterized in that: The memory is coupled to the processor; wherein, the memory is used to store program data, and the processor is used to execute the program data to implement the underwater robot path planning method for an unknown dynamic environment as described in any one of claims 1-6 above.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the underwater robot path planning method for an unknown dynamic environment as described in any one of claims 1 to 6 is implemented.

9. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instruction is executed by a processor, the underwater robot path planning method for an unknown dynamic environment described in any one of claims 1 to 6 is implemented.

Images