Behavior-based control method of autonomous underwater robot for comprehensive reinforcement learning
By constructing a dynamic weight parameter and policy network correction method in autonomous underwater vehicles (AUVs), the problem of poor behavior of AUVs in complex environments is solved, and intelligent control and environmental adaptability of autonomous underwater vehicles are realized.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-12
- Publication Date
- 2026-03-13
AI Technical Summary
Existing autonomous underwater vehicles (AUVs) struggle to adapt to environmental changes in complex underwater environments. Fixed weight parameters lead to poor behavioral performance, and model differences between simulated and real environments affect reinforcement learning performance.
A comprehensive reinforcement learning approach is adopted to construct dynamic weight parameters by perceiving the environmental state, combine them with interval planning algorithm for action decision-making, and collect data in the real environment to correct the policy network parameters, thereby achieving adaptive adjustment of behavior weights.
It improves the intelligent behavior of AUVs in unknown and complex environments, adapts to different task scenarios, reduces the performance difference between simulation and real environments, and enhances the flexibility and robustness of the control system.
Smart Images
Figure CN121649974A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of reinforcement learning, behavior-based control, and autonomous underwater robot control, specifically a behavior-based control method for autonomous underwater robots that integrates reinforcement learning. Background Technology
[0002] The underwater environment is highly complex and unpredictable, posing significant challenges and threats to autonomous underwater vehicles (AUVs) performing underwater tasks and ensuring their own safety. To address these challenges, AUVs need to fully understand their external environment through their perception systems and make maneuvering decisions based on the current task and this environmental awareness. Therefore, a decision-making and control system is required to achieve intelligent control of the AUV.
[0003] Behavior-based control architecture was proposed based on the observation and study of insect behavior. "Behavior" is an interpretation of robot responses, and behavior-based robot control is a high-level control method that has demonstrated good performance in robot design. Efficient autonomous robot control relies heavily on the appropriate combination or arbitration of independent behaviors. The interval programming algorithm proposed by MIT is a multi-objective function solution algorithm. This algorithm requires each behavior to provide its own objective function, forming a multi-objective function. The fusion of multiple behaviors is achieved by solving for the action that scores highest on the multi-objective function. Because the objective function provides a wealth of useful information about the behavior, the action requirements of each behavior can be comprehensively considered when fusing multiple behaviors. The optimal action solved using the interval programming algorithm typically exhibits good performance.
[0004] However, in a multi-objective function consisting of multiple behaviors, the weight parameters of each behavior's objective function are fixed, and these weight parameters need to be determined through trial and error. When there are too many behaviors, determining the weight parameters becomes extremely difficult. On the other hand, when the task scenario changes, the original weight parameters become inapplicable. This is because different scenarios require AUVs to focus on different functional performances, and the corresponding behaviors need to be given more decision-making power to improve their performance. However, because the weight parameters are fixed, the behaviors are not allocated enough decision-making power, leading to a deterioration in the AUV's performance. Summary of the Invention
[0005] To address the shortcomings of the aforementioned technologies, the present invention aims to provide a behavior-based control method for autonomous underwater vehicles (AUVs) that integrates reinforcement learning. This method provides dynamic weight parameters for behavior based on environmental perception, enabling the fused actions to better adapt to environmental changes and thus improving the AUV's performance. Furthermore, reinforcement learning participates in the AUV's action decisions indirectly. Even if the policy network trained in a simulation environment deteriorates in the real environment, the AUV's performance will not become significantly worse. Moreover, data can be collected in the real environment to correct the policy network parameters, improving the performance of reinforcement learning implemented in simulation in the real environment.
[0006] The technical solution adopted by this invention to solve its technical problem is: a behavior-based control method for autonomous underwater robots based on integrated reinforcement learning, comprising the following steps:
[0007] The AUV's execution tasks are broken down into multiple independent behaviors;
[0008] A joint multi-objective function is constructed from the objective functions of multiple behaviors, and the optimal action decision is obtained by solving the interval programming method.
[0009] Acquire the state data of the AUV, construct the state S, reward function r, and action A for reinforcement learning, and generate a dataset;
[0010] Construct a policy network and train it using the DQN algorithm based on the dataset to provide weight parameters for each behavior;
[0011] Based on real-time AUV status data, dynamic weight parameters are obtained through a policy network, and the optimal action decision is obtained by combining multiple objective functions.
[0012] The process of decomposing the AUV execution task into multiple independent behaviors includes the following steps: constructing a joint multi-objective function from the objective functions of multiple behaviors, as detailed below:
[0013]
[0014] in, Let ω be the objective function for the i-th action; i The weight parameter for the i-th action is used to characterize the magnitude of the decision weight assigned to the current action. This indicates the AUV's movements, including speed, heading, and depth. This represents the optimal action decision.
[0015] The process of acquiring AUV state data, constructing the state S, reward function r, and action A in reinforcement learning, and generating a dataset includes the following steps:
[0016] The state of the AUV is S = [d1, d2, d3, ..., d...]. n ], where d n This represents the information of the nth action; the weight parameters of each action are represented as action A = [ω1, ω2, ω3, ..., ω...]. n ], where ω n The weight parameter represents the nth action;
[0017] Collect sensor data to generate state S, use a random strategy to generate action A as the weight parameter for each action, calculate reward R according to reward function r, generate SARS′ sequence, S′ is the state at the next moment after executing action A, and store the SARS′ sequence collected at each moment into the dataset to form experience pool Γ.
[0018] Set the range for the weight parameters for each behavior: in, These represent the minimum and maximum values, respectively.
[0019] The construction of the policy network, which involves training the network using the DQN algorithm based on the dataset to provide weight parameters for each behavior, includes the following steps:
[0020] (1) Multiple SARS′ sequences were obtained by uniform sampling in the experience pool Γ;
[0021] (2) Use the sampled SARS' sequences to generate training data for the main network: First, traverse all sequences. During each traversal, combine the variables in state S with all action values in the action space as input to the target network. The target network is then forward-propagated to obtain the output y. out Find the element y with the largest value. max Calculate the action value q = γ·y max +R; γ is the discount factor, R is the calculated reward, and S, A, and q are normalized and saved to the training set. middle;
[0022] (3) Using the training set The main network parameters were trained using the Adam algorithm.
[0023] (4) After the main network parameters have been trained a set number of times, update the main network parameters to the target network.
[0024] The dynamic weight parameters are obtained through a policy network based on the real-time collected AUV status data.
[0025] The optimal action decision is obtained by combining multiple objective functions, including the following steps:
[0026] Based on the real-time collected AUV status data, the status S = [d1, d2, d3, ..., d n ], where d n Information representing the nth action;
[0027] Based on state S, dynamic weight parameters are obtained through the policy network;
[0028] By substituting the weight parameters into the joint multi-objective function, the optimal action decision is obtained by solving the interval programming algorithm.
[0029] The AUV's status data includes its latitude, longitude, depth, heading, and speed.
[0030] The action decision includes the expected values of AUV depth, heading, and speed.
[0031] A behavior-based control system for an autonomous underwater robot based on reinforcement learning, comprising:
[0032] The task decomposition module is used to decompose the execution tasks of AUV into multiple independent behaviors;
[0033] The joint multi-objective function construction module is used to construct a joint multi-objective function from the objective functions of multiple behaviors, so as to obtain the optimal action decision through interval programming method;
[0034] The dataset construction module is used to acquire the state data of AUV, construct the state S, reward function r, and action A for reinforcement learning, and generate the dataset.
[0035] The policy network building module is used to build the policy network. It trains the network using the DQN algorithm based on the dataset to provide the weight parameters for each behavior.
[0036] The action decision module is used to obtain dynamic weight parameters through a policy network based on real-time collected AUV status data, and to obtain the optimal action decision through a joint multi-objective function.
[0037] A computer-readable storage medium storing a computer program that, when executed by a processor, implements a behavior-based control method for an autonomous underwater robot based on integrated reinforcement learning.
[0038] The present invention has the following beneficial effects and advantages:
[0039] 1. The method of this invention combines the advantages of behavior-based control (high flexibility, high speed, high robustness) with reinforcement learning (strong adaptability and autonomous learning), thereby improving the intelligent behavior performance of AUVs in unknown and complex environments.
[0040] 2. The method of the present invention provides behavior weight parameters that can adapt to the underwater environment for behavior-based control through reinforcement learning, and assists behavior-based control in allocating decision-making power to behaviors according to the current underwater environment.
[0041] 3. The method of this invention eliminates the impact of model differences between the simulation environment and the real environment on reinforcement learning performance through secondary learning.
[0042] 4. This invention is a general method without a fixed task scenario, applicable to AUVs performing various tasks, and has broad application prospects. Attached Figure Description
[0043] Figure 1 This is a flowchart of the reinforcement learning training process in the method of this invention;
[0044] Figure 2 This is a diagram of the strategy neural network structure in the method of this invention;
[0045] Figure 3 This is a flowchart of the target network parameter correction process in the method of this invention.
[0046] Figure 4 This is a schematic diagram illustrating the working principle of the behavior-based control system that integrates reinforcement learning in the method of this invention. Detailed Implementation
[0047] The present invention will be further described in detail below with reference to embodiments. The method steps will be described with reference to the accompanying drawings.
[0048] The basic idea of this invention is to decompose the tasks performed by an AUV into multiple independent behaviors using a behavior-based control method. These behaviors are then combined to form a joint multi-objective function, which is solved using an interval programming algorithm to complete the AUV's action decisions. Addressing the limitations of fixed behavior weight parameters in the joint multi-objective function, this invention uses reinforcement learning to provide dynamic weight parameters for each behavior through environmental perception (including acquiring the AUV's own state and marine environmental information). This allows the weight parameters to adaptively change with the environment, allocating decision-making power to behaviors in an optimal manner. To address the performance degradation of reinforcement learning in real-world environments, this invention designs a target network parameter correction method. Data is collected during AUV operation and sent to a shore-based system. The shore-based system uses the received data to train the target network parameters. The trained parameters are then sent to the AUV for network parameter updates, and the AUV collects data using the new network parameters. Data collection and network training occur simultaneously, and this process is repeated. The corrected network is then suitable for real-world environments. This secondary learning method solves the problem of reinforcement learning being difficult to apply in practice due to model differences between real and simulated environments.
[0049] Intelligent control of AUVs is achieved using behavior-based control methods, including the following steps:
[0050] Step 1: Analyze the task requirements of AUV, decompose the task into a set of independent autonomous behaviors, and write objective functions for the behaviors to implement them.
[0051] Step 2: After the behavior design is completed, write a behavior selection mechanism to select behaviors. This mechanism uses a tree structure to generate tasks. The tasks specify the behaviors to be activated. It generates judgment conditions by perceiving the external environment, performs tree search based on the judgment conditions, obtains the tasks to be executed, and activates the corresponding behaviors.
[0052] Step 3: Activate the corresponding behaviors in the task, and use the interval programming algorithm to arbitrate the competition between behaviors to obtain the optimal action. The interval programming algorithm obtains the current optimal action decision of the AUV by solving a joint multi-objective function composed of the objective functions of multiple behaviors. The joint multi-objective function is:
[0053]
[0054] in Let ω be the objective function for the i-th action; i ω is the weight parameter for the i-th action. i The weight parameter determines the size of the decision-making power assigned to the action. That is, the larger the weight parameter, the higher the score of the action on the objective function of the action obtained by the interval programming algorithm, and the better the performance of the action. This indicates the AUV's movements, including speed, heading, and depth. This represents the optimal action decision. Reinforcement learning is used to enable the behavior weight parameters to adapt adaptively to changes in the environment.
[0055] Step 4: The AUV receives the actions obtained by the interval planning algorithm and sends them to the actuator for execution, thereby realizing behavior-based control of the AUV.
[0056] like Figure 1 As shown, the adaptive adjustment of behavior weights using reinforcement learning methods consists of two steps: data collection and network training, including the following steps:
[0057] Step 1: First, define the state of the AUV as S = [d1, d2, d3, ..., d n ], where d n This represents information related to the nth action;
[0058] Step 2: Define the weight parameters for each action as action A = [ω1, ω2, ω3, ..., ω... n ], where ωn Let ω represent the weight parameters for the nth action, where each ω is a non-negative integer. These weight parameters have the following constraints:
[0059] ω1+ω2+ω3+…+ω n =100
[0060] To ensure the above constraints hold, a random strategy is used during the data collection phase to generate actions A that satisfy these constraints. During the execution phase, the actions input to the policy network also satisfy these constraints. Furthermore, based on experience, a range is set for the weight parameters of each action: in Represents ω n The minimum value that can be obtained. Represents ω n The maximum value that can be obtained, this range is determined empirically, when ω n AUVs will exhibit significantly worse performance when outside this range;
[0061] Step 3: Define a reward function r based on the task to be performed by the AUV, and calculate the reward R using the reward function r;
[0062] Step 4: Run the behavior-based control system in the simulation environment. During system operation, collect sensor data to generate states S, use a stochastic policy to generate actions A as weight parameters for each action, calculate the reward R according to the reward function r, and generate a SARS′ sequence. S′ represents the agent's state at the next moment after executing action A. Store the SARS′ sequences collected at each moment into a dataset to form a sufficiently large experience pool Γ. The reward function r can be customized as needed, and both the reward calculation R and the behavior-based reward calculation are existing technologies.
[0063] Step 5: Train the behavior weight parameters using the DQN algorithm to construct a reinforcement learning policy neural network, consisting of a main network and a target network. These two policy neural networks have the same structure. The input dimension of the network is the sum of the number of variables in state S and action A, and the input can be represented as [S, A]. The network output is the action value q, with a dimension of 1. The network structure is as follows... Figure 2 As shown.
[0064] Step 6: Train the policy neural network. The training process is as follows:
[0065] (1) Uniform sampling is performed in the experience pool Γ to obtain a certain number of SARS′ sequences;
[0066] (2) Use the sampled SARS′ to generate training data for the main network: First, traverse all sequences. During each traversal, combine the variables in state S with all action values in the action space as input to the target network. The target network is then forward-propagated to obtain the output y. out y out Given a one-dimensional array, find the element y with the largest value. max Therefore, the action value q = γ·y can be calculated based on this. max +R; γ is the discount factor, with a value of 0.9. Then, S, A, and q are normalized and saved to the training set. In this context, normalization involves normalizing all input variables according to their range of values. The purpose of this is to enable the neural network parameters to converge quickly.
[0067] (3) Using the training set The main network parameters were trained using the Adam algorithm.
[0068] (4) After training the main network parameters a certain number of times, update the main network parameters to the target network;
[0069] Step 7: Integrate the trained target network into the behavior-based control system as a software module. Encapsulate the target network to form a software module named RLWG (Reinforcement Learning Weight Generation). The input of this module is the state S, and the output is the weight parameters of each behavior.
[0070] Because of the differences between simulation and real-world environments, a target network that performs well in simulation may no longer be applicable in real-world environments. However, since the action weights are empirically constrained within a reasonable range and reinforcement learning does not directly participate in the AUV's action decisions, the AUV's performance may deteriorate. This invention proposes that the AUV collect data while performing a task, and use this data collected in the real-world environment to train the target network. This secondary learning approach improves the performance of reinforcement learning in real-world environments. Figure 3 The specific steps of this training method are described below:
[0071] Step 1: The AUV collects data and generates SARS' sequences during operation, and then transmits the SARS' sequences to the shore base via wireless communication;
[0072] Step 2: The shore-based system stores the received data into its local database;
[0073] Step 3: When the number of SARS' in the database exceeds a certain number, Anji continues to train the target network using the strategy network training method described in Section 5, and corrects the network parameters in the target network.
[0074] Step 4: After the target network has been trained a certain number of times, the network parameters are sent to the AUV via wireless communication;
[0075] Step 5: After receiving the network parameters from the shore-based network, the AUV updates the target network with the received network parameters, uses the updated network to generate behavioral weight parameters, continues to perform the task, and collects SARS' sequences.
[0076] Step 6: AUV data collection and shore-based network training are carried out simultaneously. Over time, the parameters of the target network are continuously corrected to better suit the real environment, enabling the AUV to have good behavioral performance in the actual environment.
[0077] like Figure 4 As shown, the behavior-based control system for an autonomous underwater robot based on integrated reinforcement learning, as described in this invention, comprises the following operational steps:
[0078] Step 1: The AUV perceives its own state and the external environment to generate the judgment conditions for the state S and behavior selection mechanism;
[0079] Step 2: Call the RLWG module to assign weight parameters to all behaviors. The steps of RLWG are as follows:
[0080] ① Define variable Q max Define variable A to store the maximum value of q. * Store the action A that maximizes q;
[0081] ② Traverse the action space. During each traversal, combine the state S with the current action A and normalize the combination as input to the target network.
[0082] ③ The target network performs forward propagation, calculating the action value q after taking action A in state S, and q is compared with Q. max Compare, if q > Q max Then update Q max =q, A * =A;
[0083] ④ After completing the traversal, output A. * Into the control system;
[0084] Step 3: After calculating the weight parameters, read A. *We assign weights to all behaviors and combine them to form a joint multi-objective function;
[0085] Step 4: Use the interval planning algorithm to solve the joint multi-objective function to obtain the optimal action of the AUV;
[0086] Step 5: The AUV sends the optimal action to the actuator for execution;
[0087] In addition to the above embodiments, the present invention may have other implementation methods. All technical solutions formed by equivalent substitution or equivalent transformation fall within the protection scope claimed by the present invention.
Claims
1. A behavior-based control method for an autonomous underwater robot integrating reinforcement learning, characterized in that, Includes the following steps: The AUV's execution tasks are broken down into multiple independent behaviors; A joint multi-objective function is constructed from the objective functions of multiple behaviors, and the optimal action decision is obtained by solving the interval programming method. Acquire the state data of the AUV, construct the state S, reward function r, and action A for reinforcement learning, and generate a dataset; Construct a policy network and train it using the DQN algorithm based on the dataset to provide weight parameters for each behavior; Based on real-time AUV status data, dynamic weight parameters are obtained through a policy network, and the optimal action decision is obtained by combining multiple objective functions.
2. The behavior-based control method for an autonomous underwater robot based on integrated reinforcement learning according to claim 1, characterized in that, The process of decomposing the AUV execution task into multiple independent behaviors includes the following steps: constructing a joint multi-objective function from the objective functions of multiple behaviors, as detailed below: in, Let ω be the objective function for the i-th action; i The weight parameter for the i-th action is used to characterize the magnitude of the decision weight assigned to the current action. This indicates the AUV's movements, including speed, heading, and depth. This represents the optimal action decision.
3. The behavior-based control method for an autonomous underwater robot based on integrated reinforcement learning according to claim 1, characterized in that, The process of acquiring AUV state data, constructing the state S, reward function r, and action A in reinforcement learning, and generating a dataset includes the following steps: The state of the AUV is S = [d1, d2, d3, ..., d...]. n ], where d n This represents the information of the nth action; the weight parameters of each action are represented as action A = [ω1, ω2, ω3, ..., ω...]. n ], where ω n The weight parameter represents the nth action; Collect sensor data to generate state S, use a random strategy to generate action A as the weight parameter for each action, calculate reward R according to reward function r, generate SARS′ sequence, S′ is the state at the next moment after executing action A, and store the SARS′ sequence collected at each moment into the dataset to form experience pool Γ.
4. The behavior-based control method for an autonomous underwater robot based on integrated reinforcement learning according to claim 3, characterized in that, Set the range for the weight parameters for each behavior: in, These represent the minimum and maximum values, respectively.
5. The behavior-based control method for an autonomous underwater robot based on integrated reinforcement learning according to claim 1, characterized in that, The construction of the policy network, which involves training the network using the DQN algorithm based on the dataset to provide weight parameters for each behavior, includes the following steps: (1) Multiple SARS′ sequences were obtained by uniform sampling in the experience pool Γ; (2) Use the sampled SARS' sequences to generate training data for the main network: First, traverse all sequences. During each traversal, combine the variables in state S with all action values in the action space as input to the target network. The target network is then forward-propagated to obtain the output y. out Find the element y with the largest value. max Calculate the action value q = γ·y max +R; γ is the discount factor, R is the calculated reward, and S, A, and q are normalized and saved to the training set. middle; (3) Using the training set The main network parameters were trained using the Adam algorithm. (4) After the main network parameters have been trained a set number of times, update the main network parameters to the target network.
6. The behavior-based control method for an autonomous underwater robot based on integrated reinforcement learning according to claim 1, characterized in that, The process of obtaining dynamic weight parameters through a policy network based on real-time collected AUV status data, and then obtaining the optimal action decision through a joint multi-objective function, includes the following steps: Based on the real-time collected AUV status data, the status S = [d1, d2, d3, ..., d n ], where d n Information representing the nth action; Based on state S, dynamic weight parameters are obtained through the policy network; By substituting the weight parameters into the joint multi-objective function, the optimal action decision is obtained by solving the interval programming algorithm.
7. A behavior-based control method for an autonomous underwater robot based on integrated reinforcement learning as described in claim 1 or 6, characterized in that, The AUV's status data includes its latitude, longitude, depth, heading, and speed.
8. A behavior-based control method for an autonomous underwater robot based on integrated reinforcement learning according to claim 1 or 6, characterized in that, The action decision includes the expected values of AUV depth, heading, and speed.
9. A behavior-based control system for an autonomous underwater robot based on integrated reinforcement learning, characterized in that, include: The task decomposition module is used to decompose the execution tasks of AUV into multiple independent behaviors; The joint multi-objective function construction module is used to construct a joint multi-objective function from the objective functions of multiple behaviors, so as to obtain the optimal action decision through interval programming method; The dataset construction module is used to acquire the state data of AUV, construct the state S, reward function r, and action A for reinforcement learning, and generate the dataset. The policy network building module is used to build the policy network. It trains the network using the DQN algorithm based on the dataset to provide the weight parameters for each behavior. The action decision module is used to obtain dynamic weight parameters through a policy network based on real-time collected AUV status data, and to obtain the optimal action decision through a joint multi-objective function.
10. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, implements a behavior-based control method for an autonomous underwater robot based on integrated reinforcement learning as described in any one of claims 1-8.