A cross-domain action sequence decision-making method for amphibious unmanned vehicles based on reinforcement learning

Abstract

The present invention belongs to the technical field of unmanned vehicle action sequence decision-making, and specifically relates to an amphibious unmanned vehicle cross-domain action sequence decision-making method based on reinforcement learning. First, a simulation environment for intelligent agent training and testing is established, and the environment includes environmental changes such as coastal slope, land material, and wind and wave level. Then the environment is measured using a self-attention network. By collecting environmental feedback from the unmanned vehicle, the attention fusion features are calculated in combination with the intelligent agent's own state. Finally, the control of the amphibious unmanned vehicle is achieved in combination with the reinforcement learning framework. The attention fusion features are combined with the unmanned vehicle state as the state information of the reinforcement learning framework, which improves the generalization ability of the algorithm in changing scenarios. The present invention uses the self-attention mechanism to process the feedback data obtained from environmental changes, and combines the reinforcement learning framework to achieve control of the amphibious unmanned vehicle. It can be trained and executed under environmental changes, and can converge to better results quickly.

Description

Technical Field

[0001] The present invention belongs to the technical field of unmanned vehicle action sequence decision making, and in particular relates to an amphibious unmanned vehicle cross-domain action sequence decision making method based on reinforcement learning. Background Art

[0002] The cross-domain scenario of amphibious unmanned vehicles can be described by a partially observable Markov decision process (POMDP). In the process of solving the POMDP problem, two dimensionality disasters are often encountered. The first is the dimensionality disaster in the observation and action space, that is, there is a lot of noise in the information and the possibility of action space is large and large, especially the continuous action space. The second is the dimensionality disaster on the historical information length trajectory, that is, as time goes by, the historical trajectory considered will become longer and longer, which not only increases the computational burden, but also increases the difficulty of extracting useful information from these long sequences. Regarding the dimensionality disaster on the historical information length trajectory, the long short-term memory (LSTM) network and the Transformer architecture are used to alleviate this problem while maintaining the decision quality.

[0003] Zhu et al. combined sequence sample data with LSTM network to enhance the learning ability of sequence sample data, thereby improving the training efficiency and effect of the algorithm. This method is used to deal with the path planning problem of amphibious unmanned vehicles, and the cross-domain action decision-making problem of amphibious unmanned vehicles is also related to morphological control. In order to integrate morphological information into the learning process of the controller, the encoding method proposed by Chen et al. in 2018 combines morphological information and state information to determine the action selection at the next moment. Morphological control based on graph neural network has played an important role in improving the generalization ability of the model. Transformer was proposed by Vaswani et al. in 2017 and is a network structure based entirely on the attention mechanism. This structure completely abandons recursion and convolution, uses the self-attention mechanism to control the path length between remote dependencies, and increases the amount of parallel computation. Hong et al. proposed a transformation strategy model that can effectively encode this information, which is called SWAT (Structure-aWAre Transformer). Gupta et al. believe that the robot morphology is just another mode of controlling the output of Transformer. For this reason, they proposed a robot action space general controller model MetaMorph.

[0004] The cross-domain action sequence decision of an amphibious unmanned vehicle needs to consider the changes in the environment and the changes in the morphology of the unmanned vehicle. Based on this purpose, the present invention proposes a cross-domain action sequence decision method for an amphibious unmanned vehicle based on a reinforcement learning algorithm and a self-attention mechanism, which can be trained and executed under environmental changes and can converge to better results quickly. Summary of the invention

[0005] (I) Purpose of the invention

[0006] The purpose of the present invention is to provide a cross-domain action sequence decision method for an amphibious unmanned vehicle based on reinforcement learning, so as to solve the application problem of the action and timing selection of the amphibious unmanned vehicle in the cross-domain process in changing scenarios.

[0007] (II) Technical solution

[0008] In order to achieve the above objectives and solve the above technical problems, the technical solutions of the present invention are as follows:

[0009] A cross-domain action sequence decision method for an amphibious unmanned vehicle based on reinforcement learning specifically comprises the following steps:

[0010] Step 1: Establish a simulation environment for training and testing the agent, which includes environmental changes such as coastal slope, land material, and wind and wave levels.

[0011] Step 2: Use the self-attention network to measure the environment. The unmanned vehicle collects feedback from the environment and calculates the attention fusion features based on the agent’s own state.

[0012] Step 3: Combine the reinforcement learning framework to realize the control of the amphibious unmanned vehicle. The attention fusion features are combined with the state of the unmanned vehicle as the state information of the reinforcement learning framework, which improves the generalization ability of the algorithm in changing scenarios.

[0013] Effective benefits of the present invention

[0014] 1. This paper discusses the cross-domain action sequence decision problem of amphibious unmanned vehicles for the first time, forms a Markov decision process, and proposes an action sequence decision algorithm based on reinforcement learning.

[0015] 2. The present invention improves the agent network and introduces a self-attention mechanism to process the feedback data obtained from environmental changes, thereby improving the utilization of historical information. The advantage function is calculated using generalized advantage estimation, which reduces the variance while maintaining a certain deviation, thereby more effectively performing policy gradient estimation.

[0016] 3. The present invention sets up an amphibious unmanned vehicle cross-domain scenario with variable environment, including environmental changes such as coastal slope, land material, wind and wave level, which can be used for training and testing of intelligent agents. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] none. DETAILED DESCRIPTION

[0018] The present invention is explained and illustrated in detail below.

[0019] The present invention provides a cross-domain action sequence decision method for an amphibious unmanned vehicle based on reinforcement learning. The method uses a self-attention mechanism to process feedback data of a changing environment and combines a reinforcement learning framework to realize control of an amphibious unmanned vehicle. The specific steps are as follows:

[0020] Step 1: Create a simulation environment for agent training and testing

[0021] In the simulation environment, the cross-domain process is divided into three stages, including the surface stage, the ground stage, and the water-land cross-domain stage. The unmanned vehicle is modeled as a 2D model with the front and rear wheels as the base points, where the unmanned vehicle body is a rectangle, and the front and rear wheels are at equal distances from the center of the unmanned vehicle. In the surface stage, environmental changes include the level of wind and waves on the water surface. The unmanned vehicle is driven by a water surface power source, taking into account the relationship between the buoyancy, resistance, speed, and shape changes of the vehicle body in the water. The resistance of water is calculated according to the fluid resistance formula R = 0.5vC d ·Avρ·v 2 , where C dis the drag coefficient, A is the reference area of ​​the unmanned vehicle, ρ is the density of the fluid, and v is the speed of the unmanned vehicle. The buoyancy of water follows the buoyancy formula. Since the buoyancy is always equal to the gravity in the water surface stage, the reference area only needs to be calculated according to the default draft and tire position. In the ground stage, environmental changes include the material of the ground. The unmanned vehicle is driven by tires or tracks, and the relationship between the friction, gravity, speed and shape of the vehicle body on the ground is considered. The specific approach is similar to the water surface stage. The difference is that the resistance of water is replaced by the friction of the ground, and the buoyancy is replaced by the support force of the ground. The support force is always equal to the gravity, so the friction formula only needs to be calculated according to gravity. In the cross-domain stage, environmental changes include wind and wave levels, ground materials and ground slopes. The unmanned vehicle is driven by both the water surface power source and the ground power source. Consider the relationship between the friction of the vehicle body on the ground, the resistance in the water, gravity, buoyancy, speed and shape. The specific approach is similar to the previous two stages, except that the buoyancy of the water and the support force of the ground are changing. As the water depth decreases, the buoyancy of the water decreases, the support force increases, and the reference area of ​​the unmanned vehicle also decreases. This further leads to a decrease in water resistance and an increase in ground friction. At the same time, the position of the tire will directly affect the water resistance and ground friction. The extension of the tire will increase the reference area of ​​the unmanned vehicle and increase the water resistance. The retraction of the tire will cause the chassis of the unmanned vehicle to touch the land, and the friction will be extremely large. At the same time, in the cross-domain stage, there will be situations where the power direction and slope are different. The collision formula is used to reflect the force of this process. The combined effect of the impact force and other forces on the unmanned vehicle during the collision is received by the sensor as the observation information of the model.

[0022] Step 2: Introduce the self-attention mechanism to process the feedback data obtained from environmental changes

[0023] Because the environment is partially observable, the observed data cannot directly correspond to the state space. At the same time, due to environmental changes, the dimension of the observed data also changes to a certain extent. The self-attention network is used to process the observed information, and the feedback data obtained by the unmanned vehicle through the environment can be processed by the self-attention network to predict the state information of the environment.

[0024] Specifically, the state information of the unmanned vehicle, the observation information of the environment feedback, and the historical information are input into the self-attention network to calculate the feature vector. In this process, the information is first concatenated as a sequence and converted into a vector form X, and then the vector X is input into the Encoder layer of the transformer. The vector X is input into two fully connected layers to obtain the query vector Q and the key vector K, and the state information and observation information of the unmanned vehicle are input into another fully connected layer to obtain the value vector V. These three vectors are input into the self-attention network to calculate the feature vector. The feature is concatenated with vector X in the residual network, and then linearly transformed, RELU activated, and concatenated with the residual network again. The final vector is normalized and input to the Decoder layer. The Decoder layer is roughly the same as the Encoder layer. Finally, a linear network is used to adjust the data dimension, and softmax is used to predict the state information vector. The vector is combined with the current state information as the new state information and input into the reinforcement learning framework.

[0025] Step 3: Combine reinforcement learning framework to realize control of amphibious unmanned vehicle

[0026] Step 3.1: Initialize the neural network PPO model, which consists of two parts: strategy and value function. The strategy is a function that outputs actions based on the state, while the value function is a function that outputs values ​​based on the state.

[0027] Step 3.2: Next, use the agent network to interact with the environment, obtain the state of the environment, and obtain the state vector through the network in step 2. Use linear layer transformation to match the dimension of the state vector with the state space of the environment, select and execute the corresponding action, and obtain the corresponding reward and next state.

[0028] Step 3.3: Calculate the advantage function and target value function through historical information. Calculate the proximal ratio clipping loss and value function loss through the current advantage function and value function, and further calculate the total loss function.

[0029] Step 3.4: Update the strategy, value function, and self-attention network through back propagation. Repeat steps 3.2 to 3.4 until the algorithm converges and a feasible strategy is obtained.

[0030] This paper proposes a cross-domain action sequence decision method for amphibious unmanned vehicles based on reinforcement learning algorithm and self-attention mechanism, aiming to adapt to environmental changes in the cross-domain problem of amphibious unmanned vehicles. This method uses the self-attention mechanism to process the feedback data obtained from environmental changes, and combines the reinforcement learning framework to realize the control of the amphibious unmanned vehicle. It can be trained and executed under environmental changes, and can converge to better results quickly.

[0031] The above contents are further detailed descriptions of the present invention in combination with specific implementation methods, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For ordinary technicians in the technical field to which the present invention belongs, several simple deductions or substitutions can be made without departing from the concept of the present invention, which should be regarded as falling within the protection scope of the present invention.

Claims

1. A cross-domain action sequence decision method for an amphibious unmanned vehicle based on reinforcement learning, which uses a self-attention mechanism to process feedback data of a changing environment and combines a reinforcement learning framework to realize the control of an amphibious unmanned vehicle, characterized in that: The specific steps include: Step 1: Establish a simulation environment for agent training and testing In the simulation environment, the cross-domain process is divided into three stages: the surface stage, the ground stage and the water-land cross-domain stage; The unmanned vehicle is modeled as a 2D model with the front and rear wheels as the base points, where the body of the unmanned vehicle is a rectangle, and the front and rear wheels are at equal distances from the center of the unmanned vehicle; In the surface stage, environmental changes include the wind and wave level on the water surface. The unmanned vehicle is driven by the surface power source, taking into account the relationship between the buoyancy, resistance, speed and shape change of the vehicle body in the water; the water resistance is calculated according to the fluid resistance formula R = 0.5 C d ·A·ρ·v 2 , where C d is the drag coefficient, A is the reference area of ​​the unmanned vehicle, ρ is the density of the fluid, and v is the speed of the unmanned vehicle; the buoyancy of water is calculated according to the buoyancy formula, and the area is calculated according to the default draft and tire position; In the ground stage, environmental changes include the material of the ground. The unmanned vehicle is driven by tires or tracks, and the relationship between the friction, gravity, speed and shape of the vehicle on the ground is considered. The specific approach is similar to the water stage, except that the water resistance is replaced by the friction of the ground, and the buoyancy is replaced by the support force of the ground. The support force is always equal to the gravity, so the friction formula only needs to be calculated according to gravity. In the cross-domain stage, environmental changes include wind and wave levels, ground material and ground slope. The unmanned vehicle is driven by both water surface power source and ground power source. The relationship between the friction of the vehicle body on the ground, the resistance in the water, gravity, buoyancy, speed and shape is considered. The specific approach is similar to the surface stage and the ground stage, except that the buoyancy of the water and the support force of the ground are changing. As the water depth decreases, the buoyancy of the water decreases, the support force increases, and the reference area of ​​the unmanned vehicle also decreases. This further leads to a decrease in water resistance and an increase in ground friction. At the same time, the position of the tires will directly affect the water resistance and ground friction. The extension of the tires will increase the reference area of ​​the unmanned vehicle and increase the water resistance. The retraction of the tires will cause the chassis of the unmanned vehicle to touch the land, which will cause great friction. At the same time, in the cross-domain stage, there will be situations where the power direction and slope are different. The collision formula is used to reflect the force of this process. The combined effect of the impact force and other forces on the unmanned vehicle during the collision is received by the sensor as the observation information of the model. Step 2: Introduce the self-attention mechanism to process the feedback data obtained from environmental changes The observation information is processed using the self-attention network. The feedback data obtained by the unmanned vehicle through the environment is processed by the self-attention network to predict the state information of the environment. The specific approach is as follows: the state information of the unmanned vehicle itself, the observation information and historical information of the environmental feedback are input into the self-attention network to calculate the feature vector; first, the information is spliced ​​as a sequence and converted into a vector form X, and then the vector X is input into the Encoder layer of the transformer; the vector X is input into two fully connected layers to obtain the query vector Q and the key vector K, and the state information and observation information of the unmanned vehicle are input into another fully connected layer to obtain the value vector V; these three vectors are input into the self-attention network to calculate the feature The feature is concatenated with the vector X in the residual network, and then linear transformation, RELU activation and residual network concatenation are performed again. The final vector is normalized and input into the Decoder layer. The Decoder layer is roughly the same as the Encoder layer. Finally, a linear network is used to adjust the data dimension, and softmax is used to predict the state information vector. The vector is combined with the current state information as the new state information and input into the reinforcement learning framework. Step 3: Combine reinforcement learning framework to realize control of amphibious unmanned vehicle Step 3.1, initialize the neural network PPO model, which consists of two parts: strategy and value function; strategy is a function that outputs actions according to the state, and value function is a function that outputs values ​​according to the state; Step 3.2: Use the agent network to interact with the environment, obtain the environment state, and obtain the state vector through the network in step 2; match the state vector with the state space of the environment through linear layer transformation, select and execute the corresponding action, and obtain the corresponding reward and next state; Step 3.3, calculate the advantage function and the target value function through historical information; calculate the proximal ratio clipping loss and the value function loss through the current advantage function and the value function, and further calculate the total loss function; Step 3.4: Update the strategy, value function, and self-attention network through back-propagation; repeat steps 3.2 to 3.4 until the algorithm converges and obtains a feasible strategy.

Images