Underwater robot path planning method for unknown dynamic environment, electronic equipment and medium

Abstract

The invention discloses an underwater robot path planning method for an unknown dynamic environment, electronic equipment and a medium, and the method comprises the steps: an intelligent agent selects an action according to the output of a current Q network, interacts with the environment, and obtains a state; security constraints in path planning are considered, the state is processed, first-class guidance information and second-class guidance information are obtained, and therefore the state is updated; calculating a reward according to the updated state; storing the action, the state and the reward into an experience playback pool; repeating the process until the data volume in the experience playback pool reaches a threshold value; extracting a batch of empirical data from the empirical playback pool, respectively inputting the empirical data into the Q network and the target Q network, and calculating a Q value and a target Q value; minimizing the loss function by adopting a gradient descent method so as to update Q network parameters; copying the parameters of the Q network to a target Q network; obtaining a trained target Q network until the loss function converges; and configuring a test environment, and outputting a planned path by using the trained target Q network.

Description

Technical Field

[0001] The present invention relates to the technical field of underwater robot path planning, and in particular to an underwater robot path planning method, electronic equipment and medium for an unknown dynamic environment. Background Art

[0002] With the deepening of research in the marine field, the demand for autonomous navigation and path planning of underwater robots in complex marine environments is increasing. These environments are highly uncertain, including changes in the speed and direction of ocean currents, the number, location and motion characteristics of obstacles, etc. These factors significantly increase the robot's dependence on environmental information and increase the difficulty of information acquisition and processing. Therefore, the development of efficient and stable path planning algorithms is crucial for autonomous obstacle avoidance and optimal path finding of underwater robots.

[0003] Traditional path planning algorithms (such as those based on graph search or shortest path optimization) perform well in static environments, but have obvious limitations when facing dynamic ocean environments. These algorithms usually take a long time to calculate, have poor flexibility, and have difficulty responding to environmental changes in real time. In addition, they have insufficient anti-interference capabilities and are prone to decision-making errors or path deviations in highly uncertain environments, affecting the safety of the robot and the effectiveness of task execution.

[0004] Compared with traditional methods, deep learning-based methods demonstrate a powerful ability to learn complex features and adapt to dynamic environments, improving the accuracy and efficiency of path planning. At present, in the field of underwater robot path planning, there are methods based on reinforcement learning, methods based on deep neural networks, and deep reinforcement learning methods that combine the two. Because they can simultaneously process high-dimensional state spaces and learn long-term strategies, they are widely used in path planning tasks in complex dynamic environments. However, there are still the following difficulties when directly applying deep reinforcement learning to underwater robot path planning:

[0005] (1) Most current deep reinforcement learning-based methods require learning and decision-making in a high-dimensional state space. In actual complex environments, the dimensionality of the state space is often very high. Therefore, as the dimension of the environment increases, the state space will grow explosively, resulting in reduced sample efficiency and a sharp increase in computational complexity. This makes it extremely difficult to directly apply deep reinforcement learning methods in high-dimensional environments. It is necessary to consider how to effectively reduce and represent the high-dimensional state space in order to reduce the computational burden and improve learning efficiency.

[0006] (2) In many task environments in real situations, reward signals are often very sparse, that is, the reward is zero in most states, and positive rewards are obtained only when a specific target state is reached. This sparse reward feature makes it difficult for the agent to effectively learn valuable behavior strategies, because most exploratory behaviors do not receive clear feedback, resulting in a slow and inefficient learning process. Therefore, when applying deep reinforcement learning methods, it is necessary to consider how to overcome sparse rewards and design more effective reward mechanisms to promote learning.

[0007] (3) Most current deep reinforcement learning methods can achieve good performance in the training environment, but their generalization ability is often poor when faced with new, unseen environments. This is because deep reinforcement learning models tend to memorize the characteristics of a specific environment rather than learning a truly universal strategy. In actual applications, the environment may change or completely new situations may appear, and current methods are difficult to adapt to such changes, and are prone to a sharp drop in performance. Summary of the invention

[0008] In view of the deficiencies in the prior art, the present invention provides a path planning method, electronic equipment, and medium for an underwater robot in an unknown dynamic environment.

[0009] In a first aspect, an embodiment of the present invention provides a method for underwater robot path planning in an unknown dynamic environment, the method comprising the following steps:

[0010] The agent selects actions based on the output of the current Q network, interacts with the environment, and obtains the state; considers the safety constraints in path planning, processes the state, obtains the first type of guidance information and the second type of guidance information, and updates the state; calculates the reward based on the updated state; stores the action, state, and reward in the experience replay pool; repeats the above process until the amount of data in the experience replay pool reaches the threshold;

[0011] A batch of experience data is extracted from the experience replay pool, and input into the Q network and the target Q network respectively, and the Q value and the target Q value are calculated; the loss function is minimized by the gradient descent method to update the Q network parameters; the loss function is used to calculate the difference between the Q value and the target Q value; every unit training cycle, the parameters of the Q network are copied to the target Q network; until the loss function converges, a trained target Q network is obtained;

[0012] Configure the test environment and use the trained target Q network output to plan the path.

[0013] In a second aspect, an embodiment of the present invention provides an electronic device, comprising a memory and a processor, wherein the memory is coupled to the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the above-mentioned underwater robot path planning method for an unknown dynamic environment.

[0014] In a third aspect, an embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-mentioned underwater robot path planning method for an unknown dynamic environment.

[0015] In a fourth aspect, an embodiment of the present invention provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the above-mentioned underwater robot path planning method for an unknown dynamic environment.

[0016] Compared with the prior art, the present invention has the following beneficial effects:

[0017] The present invention provides an underwater robot path planning method for an unknown dynamic environment. The dual deep Q network (DDQN) is used to process the state to obtain a heuristic guidance mechanism and a reward function for the first type of guidance information and the second type of guidance information, which significantly improves the efficiency and safety of path planning. The method of the present invention can adapt to complex and changeable environmental conditions. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0019] Figure 1 It is an overall schematic diagram of an underwater robot path planning method for an unknown dynamic environment, an electronic device, and a medium provided by an embodiment of the present invention;

[0020] Figure 2 is a schematic diagram of a deep reinforcement learning network model provided by an embodiment of the present invention;

[0021] Figure 3 It is a graph of the test results of multi-task planning in a medium-scale environment using the method provided by the present invention;

[0022] Figure 4 This is a graph showing the test results of the method provided by the present invention for multi-task planning in a large-scale environment;

[0023] Figure 5It is a schematic diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0024] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0025] It should be noted that, in the absence of conflict, the features in the following embodiments and implementations may be combined with each other.

[0026] like Figure 1 and Figure 2 As shown, an embodiment of the present invention provides a path planning method for an underwater robot in an unknown dynamic environment, the method comprising:

[0027] Step S1, the agent selects an action according to the output of the current Q network, interacts with the environment, and obtains the state; considers the safety constraints in the path planning, processes the state, obtains the first type of guidance information and the second type of guidance information, and updates the state; calculates the reward according to the updated state; stores the action, state, and reward in the experience replay pool; repeats the above process until the amount of data in the experience replay pool reaches the threshold.

[0028] Among them, the process of configuring the environment specifically includes: the environment includes the target position and initial position of the planned path, ocean current information, and obstacle information; in this example, the Canvas control in Tkinter is used to perform visual simulation of the underwater environment, the ocean current information includes the direction and speed of the current, and the obstacle information includes the number, size, position, movement direction and speed of obstacles.

[0029] Furthermore, in this example, taking into account the motion characteristics and capabilities of the underwater robot, its heading is selected at a fixed time and fixed speed, with the current heading being 0° and an interval of 45° to the left and right as one action. The action set is {-90°, -45°, 0°, 45°, 90°}, for a total of five actions.

[0030] The process of processing the state by considering the safety constraints in the path planning, obtaining the first type of guidance information and the second type of guidance information, and thus updating the state includes:

[0031] The current basic status information is obtained, which includes: the current speed of the underwater robot, the speed of the ocean current, and the relative distance and angle between the underwater robot and the target. At the same time, it includes the obstacle distance information in eight directions within the observation range (with the current heading as 0° and intervals of 45°).

[0032] In the discretization process, considering the safety constraints in path planning, the distance information of obstacles is divided into three categories according to safety: less than the collision distance, less than the observation distance, and greater than the observation distance. The three types of distance information are represented by the label values ​​-1, 0, and 1, respectively, so as to discretize the obstacle distance information in eight directions and obtain the first type of guidance information;

[0033] Based on the first type of guidance information, by predicting the next action and comparing the positions after executing different actions, the action closest to the target position is selected, and the direction corresponding to the action in the first type of guidance information is marked as 1, and other directions are marked as 0, thereby obtaining the second type of guidance information;

[0034] The first type of guidance information, the second type of guidance information and the basic status information are merged to update the status.

[0035] For example, at time t, assuming that the obstacle distance information in the basic state information is {D, D, D, D, D, D, D}, where the observation distance is d de , D is a value greater than d de The value of indicates that no obstacles are observed within the observation distance in this direction. is a value less than the detection distance d de But it is greater than the collision distance d coll The value of It means that an obstacle is observed within the observation distance of this direction, so the first type of guidance information can be obtained, corresponding to {1, 1, 1, 0, 1, 1, 1, 1}, that is, an obstacle has been observed at the fourth direction 135°, and no obstacles are observed in the other directions. On the basis of the first type of guidance information, considering the five optional actions of the underwater robot, the positions reached by taking actions in the corresponding feasible directions and under the influence of ocean currents are calculated respectively, and the position of this position and the position of the target point are calculated. Assuming that the 0° and -90° directions are closest to the target point, the second guidance information can be obtained as {1, 0, 0, 0, 0, 0, 1}. Finally, the basic state information, the first type of guidance information and the second type of guidance information together constitute the proposed heuristic guidance mechanism based on discretization of value mapping, which can effectively improve the learning efficiency and convergence of the algorithm for continuous original observation data in the state space; the first type of guidance information and the second type of guidance information, this extended information, helps to accelerate the convergence of the algorithm and improve the efficiency of task planning.

[0036] The reward calculation process includes:

[0037]

[0038] in:

[0039] r1=Δd*success_reward / d cur

[0040]

[0041]

[0042] r3=k1 cosθ

[0043] d cur =||P agent_now -P goal ||, d bef =||P agent_before -P goal ||

[0044] Δd=d cur -d bef

[0045] In the formula, r1 is the single-step reward, which is used to measure the distance d between the current target position and the underwater robot. cur Under this condition, the reward brought by the distance change Δd corresponding to a single-step action; d cur and d bef Respectively represent the distance between the underwater robot and the target position at the current moment and the previous moment; P agent_now represents the current position of the underwater robot, P agent_before represents the position of the underwater robot at the previous moment, P goal Indicates the location of the target point; The distance between the underwater robot and the obstacle at different angles and directions Evaluate the collision reward corresponding to the distance, r2 is the smallest The maximum value is 0; is the distance between the underwater robot and the obstacle at the ith position in the basic state information, d coll Denotes the collision distance, d de represents the detection distance; θ is the angle between the heading corresponding to the current action of the underwater robot and the heading of the ocean current; r3 represents the reward for the utilization of the ocean current, and k1 represents the weight parameter.

[0046] Step S2, extract a batch of experience data from the experience replay pool, input them into the Q network and the target Q network respectively, calculate the Q value and the target Q value; use the gradient descent method to minimize the loss function to update the Q network parameters; the loss function is used to calculate the difference between the Q value and the target Q value; every unit training cycle, copy the parameters of the Q network to the target Q network; until the loss function converges, a trained target Q network is obtained.

[0047] Step S3, configure the test environment and use the trained target Q network to output the planned path.

[0048] In this example, tests were conducted in different underwater simulation environments, focusing on indicators such as path planning efficiency, success rate, path length, and energy consumption, to evaluate the generalization ability and stability of the network. Multiple test environments with different difficulties and characteristics were designed, including but not limited to:

[0049] Obstacle-free static environment: verify the basic performance of path planning;

[0050] Dynamic obstacle environment: verify the path planning effect under the conditions of moving obstacles and water flow interference;

[0051] Complex obstacle environment: Test the robustness of path planning under multiple dynamic obstacles (speed, direction, and initial position are all random values) and complex ocean currents (ocean current speed and direction are all random values).

[0052] Multi-target point environment: Set up sequential access tasks, with the previous task target point as the starting point for the next task.

[0053] With an environment size of 250×300, random ocean current speed + fixed number of obstacles + random dynamic obstacles are used as the training environment, with an initial learning rate of 0.0003, a discount factor of 0.9, an experience replay buffer size of 20000, a batch size of 512, a target network update frequency of 5 epochs, and an epsilon-greedy strategy parameter of 0.1.

[0054] During the test, in order to verify the reliability and generalization of this method, we constructed a series of test environments with different characteristics. The environmental parameters were systematically adjusted during the test, including variable factors such as environmental scale, ocean current speed and direction, number and size of obstacles, initial position of obstacles and their movement speed and direction. In these environments with different parameter combinations, Monte Carlo tests of path planning with random starting and ending points were carried out, and each environment combination was conducted 100 times. In addition, sequential multi-task planning with random starting and ending points was tested. The relevant experimental results are detailed in Table 1.

[0055] Table 1: Planning results of the algorithm in this embodiment under different test environments

[0056]

[0057] To further evaluate the performance advantages of this method, we selected two representative complex environments for in-depth testing: a medium-range environment and a large-range environment (both environments 5 and 7 in Table 1 are set to random ocean current speeds and random moving obstacles). In these two environments, key performance indicators such as the success rate of path planning, the length of the planned path, and the single-step calculation time were examined. The specific test data is shown in Table 2.

[0058] Table 2: Planning results of the algorithm in this embodiment under the test environment

[0059] environment Success rate Path length (step) Single-step planning time Environment 5 97％ 438 0.0003s Environment 7 96％ 834 0.0003s

[0060] Sequential multi-task planning is performed under the settings of environment 5 and environment 7 respectively. The number of tasks to be planned is 6. Except for the starting point, the target point of the previous planning is used as the starting point of the subsequent planning. The final planning results are shown in Figure 3 and Figure 4 The complete planning result may show an intersection with the final obstacle location due to the dynamics of the environment, that is, the original feasible path is occupied by obstacles due to the constant changes in the speed and direction of the obstacle.

[0061] In summary, the underwater robot path planning method based on deep reinforcement learning proposed in the present invention utilizes a dual deep Q network (DDQN) combined with a heuristic guidance mechanism and a new hybrid reward function to significantly improve the efficiency and safety of path planning. Tests conducted in different underwater simulation environments show that the method has good generalization capabilities and can adapt to complex and changing environmental conditions. Specifically, Table 1 verifies the generalization performance of the algorithm, showing its adaptability in multiple types of complex environments, while Table 2 lists in detail the various performance indicators in specific environments, further supporting the effectiveness of the method in intelligent autonomous navigation tasks. These results show that the proposed method not only excels in planning efficiency, but also can effectively avoid obstacles in terms of safety, laying a solid foundation for the application of underwater robots in dynamic environments.

[0062] Accordingly, the present application also provides an electronic device, comprising: one or more processors; a memory for storing one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors implement the above-mentioned underwater robot path planning method for an unknown dynamic environment. Figure 5 As shown, a hardware structure diagram of any device with data processing capability for the underwater robot path planning method for unknown dynamic environment provided by an embodiment of the present invention is shown, except Figure 5In addition to the processor, memory and network interface shown, any device with data processing capability in which the apparatus in the embodiment is located may also include other hardware, generally based on the actual functions of the device with data processing capability, which will not be described in detail.

[0063] Accordingly, the present application also provides a computer-readable storage medium on which computer instructions are stored, and when the instructions are executed by a processor, the underwater robot path planning method for an unknown dynamic environment as described above is implemented. The computer-readable storage medium can be an internal storage unit of any device with data processing capabilities described in any of the aforementioned embodiments, such as a hard disk or a memory. The computer-readable storage medium can also be an external storage device, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), an SD card, a flash card (Flash Card), etc. equipped on the device. Furthermore, the computer-readable storage medium can also include both an internal storage unit of any device with data processing capabilities and an external storage device. The computer-readable storage medium is used to store the computer program and other programs and data required by any device with data processing capabilities, and can also be used to temporarily store data that has been output or is to be output.

[0064] The above embodiments are only used to illustrate the design ideas and features of the present invention, and their purpose is to enable those skilled in the art to understand the content of the present invention and implement it accordingly. The protection scope of the present invention is not limited to the above embodiments. Therefore, any equivalent changes or modifications made based on the principles and design ideas disclosed by the present invention are within the protection scope of the present invention.

Claims

1. A method for underwater robot path planning in an unknown dynamic environment, characterized in that: The method comprises the following steps: The agent selects actions based on the output of the current Q network, interacts with the environment, and obtains the state; considers the safety constraints in path planning, processes the state, obtains the first type of guidance information and the second type of guidance information, and updates the state; calculates the reward based on the updated state; stores the action, state, and reward in the experience replay pool; repeats the above process until the amount of data in the experience replay pool reaches the threshold; A batch of experience data is extracted from the experience replay pool, and input into the Q network and the target Q network respectively, and the Q value and the target Q value are calculated; the loss function is minimized by the gradient descent method to update the Q network parameters; the loss function is used to calculate the difference between the Q value and the target Q value; every unit training cycle, the parameters of the Q network are copied to the target Q network; until the loss function converges, a trained target Q network is obtained; Configure the test environment and use the trained target Q network output to plan the path.

2. The method for underwater robot path planning in an unknown dynamic environment according to claim 1, characterized in that: The environment includes the target position and initial position of the planned path, ocean current information, and obstacle information.

3. The method for underwater robot path planning in an unknown dynamic environment according to claim 2, characterized in that: The ocean current information includes the direction and speed of the ocean current, and the obstacle information includes the number, size, location, movement direction and speed of obstacles.

4. The method for underwater robot path planning in an unknown dynamic environment according to claim 1, characterized in that: Considering the safety constraints in path planning, the state is processed to obtain the first type of guidance information and the second type of guidance information, thereby updating the state. The process includes: Acquire current basic status information, the basic status information including: current speed of the underwater robot, ocean current speed, relative distance and angle information between the underwater robot and the target, and distance information of obstacles within the observation range; Considering the safety constraints in path planning, the distance information of obstacles is divided into three categories according to safety: less than the collision distance, less than the observation distance, and greater than the observation distance, which are represented by the label values ​​-1, 0, and 1 respectively, so as to discretize the obstacle distance information in eight directions and obtain the first type of guidance information; Based on the first type of guidance information, by predicting the next action and comparing the positions after executing different actions, the action closest to the target position is selected, and the direction corresponding to the action in the first type of guidance information is marked as 1, and other directions are marked as 0, thereby obtaining the second type of guidance information; The first type of guidance information, the second type of guidance information and the basic status information are merged to update the status.

5. The method for underwater robot path planning in an unknown dynamic environment according to claim 4, characterized in that: The obstacle distance information within the observation range is: Obstacle distance information at eight directions within the observation range; the eight directions are obtained by setting an observation direction at intervals of 45° with the current heading as 0°.

6. The method for underwater robot path planning in an unknown dynamic environment according to claim 1, characterized in that: The action is based on the current heading being 0°, with a left and right interval of 45° being one action, and the action set is set to {-90°, -45°, 0°, 45°, 90°}.

7. The method for underwater robot path planning in an unknown dynamic environment according to claim 1, characterized in that: The reward calculation process includes: in: r1=Δd*success_reward / d cur r3=k1cosθ d cur =||P agent_now -P goal ||,d bef =||P agent_before -P goal || Δd=d cur -d bef In the formula, r1 is the single-step reward, which is used to measure the distance d between the current target position and the underwater robot. cur Under this condition, the reward brought by the distance change Δd corresponding to a single-step action; d cur and d bef Respectively represent the distance between the underwater robot and the target position at the current moment and the previous moment; P agent_now represents the current position of the underwater robot, P agent_before represents the position of the underwater robot at the previous moment, P goal Indicates the location of the target point; The distance between the underwater robot and the obstacle at different angles and directions Evaluate the collision reward corresponding to the distance, r2 is the smallest The maximum value is 0; is the distance between the underwater robot and the obstacle at the ith position in the basic state information, d coll Denotes the collision distance, d de represents the detection distance; θ is the angle between the heading corresponding to the current action of the underwater robot and the heading of the ocean current; r3 represents the reward for the utilization of the ocean current, and k1 represents the weight parameter.

8. An electronic device, comprising a memory and a processor, characterized in that: The memory is coupled to the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the underwater robot path planning method for an unknown dynamic environment as described in any one of claims 1-7 above.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by the processor, the underwater robot path planning method for an unknown dynamic environment as described in any one of claims 1 to 7 is implemented.

10. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instruction is executed by a processor, the underwater robot path planning method for an unknown dynamic environment described in any one of claims 1-7 is implemented.

Images