A training method for automatic mode switching and ground obstacle avoidance of an amphibious unmanned vehicle

Abstract

The application provides a training method for automatic mode switching and ground obstacle avoidance of an amphibious unmanned vehicle, and the method comprises the following steps: step 1) constructing a training scene based on a Gazebo simulator; step 2) inputting perception information obtained by the amphibious unmanned vehicle in the training scene into a decision neural network based on the principle of machine learning and reinforcement learning, planning a moving track and switching a motion mode by adopting an epsilon-greedy method combined with a curiosity-driven strategy, and transmitting an action instruction into the Gazebo simulator to guide the motion of the unmanned vehicle; step 3) storing motion-related data into an experience pool, and optimizing the decision neural network by randomly extracting a plurality of groups of data from the experience pool based on an n-step time difference method; returning to step 2) until the unmanned vehicle triggers a termination condition or meets a preset training step length, and then a training cycle is terminated; and step 4) when the total number of training cycles is not reached, returning to step 1) to start a new training cycle, otherwise a trained decision neural network is obtained.

Description

Technical Field

[0001] This invention belongs to the field of intelligent driving technology and relates to the problem of automatic switching of motion modes in multimodal vehicles, and in particular to a training method for automatic mode switching and ground obstacle avoidance of amphibious unmanned vehicles. Background Technology

[0002] With the promotion of three-dimensional transportation and the inability of existing ground-based vehicles to fundamentally solve traffic congestion due to their ground-based driving characteristics, amphibious vehicles have emerged. An amphibious vehicle is a vehicle that combines flight and ground-based driving modes. It can switch between modes based on factors such as safety, stability, and energy consumption. However, it is constrained by various factors, such as low reliability of mode switching, unreasonable mode switching, and insufficient safety.

[0003] Because reinforcement learning is based on machine learning principles and follows Markov decision processes, it requires a large amount of data to support the training of the neural network in the decision-maker and the optimization of multimodal switching strategies. Traditional data collection methods often require drivers to operate vehicles in real-world environments and collect the necessary training data through sensors. This method is time-consuming and resource-intensive to collect sufficient data to support training. Furthermore, human intervention by the driver also leads to significant limitations in the collected data, resulting in low coverage of extreme situations. Summary of the Invention

[0004] The purpose of this invention is to overcome the shortcomings of existing technologies and propose a training method for automatic mode switching and ground obstacle avoidance of amphibious unmanned vehicles. This application is based on an artificial intelligence-based method for automatically switching the amphibious vehicle's travel modes while simultaneously considering automatic obstacle avoidance on the ground. Since the stability and rationality of the decision-making model are currently the main obstacles to automatic mode switching in amphibious vehicles, reinforcement learning methods based on machine learning provide an effective solution to this problem. In other words, the AI ​​mode switcher based on reinforcement learning can adaptively and automatically select the favorable travel mode according to the current environment. Data acquisition based on a simulation environment can quickly provide sufficient and high-quality data, generating mature mode switching strategies in a short time.

[0005] To achieve the above objectives, this invention proposes a training method for automatic mode switching and ground obstacle avoidance of an amphibious unmanned vehicle, the method comprising:

[0006] Step 1) Construct a training scenario based on the Gazebo simulator, including an amphibious unmanned vehicle, obstacle cones, and walls;

[0007] Step 2) Based on the principles of machine learning and reinforcement learning, the current perception information obtained by the amphibious unmanned vehicle in the training scenario is input into the decision neural network. The strategy of combining the greedy method and the curiosity-driven method is used to plan the trajectory and determine whether it is necessary to switch motion modes. The action command is then transmitted to the Gazebo simulator to command the amphibious unmanned vehicle in the Gazebo simulator to move.

[0008] Step 3) Store the motion-related data of each step into the experience pool, and randomly select several sets of data from the experience pool based on the n-step temporal difference method to optimize the decision neural network; then proceed to step 2) until the amphibious unmanned vehicle triggers the termination condition or meets the preset training step length, then terminate a training cycle.

[0009] Step 4) If the preset total number of training cycles is not met, proceed to Step 1) to start a new training cycle; otherwise, the trained decision neural network is obtained.

[0010] As an improvement to the above method, step 1) includes:

[0011] Several obstacle cones are randomly generated and randomly distributed within a rectangular area of ​​the wall with a set length and width.

[0012] Set a fixed starting point and a fixed ending point for the amphibious unmanned vehicle's movement;

[0013] The amphibious unmanned vehicle is set to move within a rectangular area of ​​the wall. During its movement, it can freely switch between flight and ground driving modes. It will fail if it collides with an obstacle cone or wall.

[0014] As an improvement to the above method, the motion of the amphibious unmanned vehicle includes seven effective actions. In ground driving mode, actions [0,1,2,3,4,5] are effective actions, and in flight mode, actions [5,6] are effective actions.

[0015] Action [0] is to move straight at the set wheel rotation speed;

[0016] The action [1] includes the set linear velocity of the wheel rotation, the angular velocity of the wheel rotation is the first wheel rotation velocity, and the direction is to the right;

[0017] The action [2] includes the set linear velocity of the wheel rotation, the angular velocity of the wheel rotation is the second wheel rotation speed, and the direction is to the right;

[0018] The action [3] includes the set linear velocity of the wheel rotation, the angular velocity of the wheel rotation is the first wheel rotation velocity, and the direction is to the left;

[0019] The action [4] includes the set linear velocity of the wheel rotation, the angular velocity of the wheel rotation is the second wheel rotation speed, and the direction is to the left;

[0020] Action [5] provides the lift required for takeoff or forward flight by rotating the rotor;

[0021] Action [6] is to slow down the rotor speed and land the unmanned vehicle stably.

[0022] As an improvement to the above method, the amphibious unmanned vehicle includes a lidar and an altimeter, used to acquire perception information including two-dimensional images and altitude information in the training scenario.

[0023] As an improvement to the above method, the decision neural network is a DDQN network, with inputs of a two-dimensional image and height information, and outputs an action value function Q-value. The decision neural network includes a three-layer convolutional neural network, two parallel four-layer fully connected layers, and a summation function; the specific processing includes:

[0024] A three-layer convolutional neural network processes the input two-dimensional image. The output of the convolutional neural network is stacked into a one-dimensional vector, which is then concatenated with height information in two separate paths. This concatenation is used as the input to two fully connected layers, passed through four fully connected layers, and finally summed by a summing function to output the corresponding action value function Q: Q(s). t ,a t ;θ t ),in,

[0025] s t Let a be the state value of the amphibious unmanned vehicle at time t. t Let θ be the motion of the amphibious unmanned vehicle at time t. t These are the parameters of the current decision neural network.

[0026] As an improvement to the above method, the greedy method in step 2) includes:

[0027] Uniform sampling is performed on the action value function Q value within the range of (0,1). If the sampled value is less than the set epsilon value, the action is selected by random sampling. If the sampled value is greater than the set epsilon value, the action with the highest probability in the action probability distribution is selected.

[0028] As an improvement to the above method, the curiosity-driven method in step 2) includes:

[0029] Input the current state s t Get the corresponding action a t The current action is input into the environment to obtain the state s for the next moment. t+1 ;

[0030] By using a pre-established feature extraction network, features f are obtained. t+1 , will f t+1and a t Simultaneously, the input is fed into the forward prediction generation network to obtain the predicted feature value f'. t+1 ;

[0031] By analyzing f t+1 and f' t+1 The intrinsic incentive reward function is obtained by doing the difference.

[0032] Construct f using the mean square loss function t+1 and f' t+1 The loss function is calculated, and the forward prediction generator network is continuously optimized through a prediction optimizer to make the predictions more accurate.

[0033] As an improvement to the above method, the method described above involves adjusting f t+1 and f' t+1 The intrinsic incentive reward function is obtained by doing the difference. Specifically, it includes:

[0034] When the ground driving mode moves 1m along the x-axis

[0035] When the ground driving mode moves backward by 1m along the x-axis

[0036] For every 1m deviation from the center line

[0037] With each takeoff,

[0038] With each landing,

[0039] When the training step length in flight mode

[0040] When the training step length is in the ground mode

[0041] When a collision occurs

[0042] When you reach the finish line,

[0043] As an improvement to the above method, the motion-related data for each step in step 3) includes the current state s. t Action a t Intrinsic motivation and pre-set external stimuli generated by the environment

[0044] A training system for automatic mode switching and ground obstacle avoidance of an amphibious unmanned vehicle, comprising: a training scenario construction module, an action decision module, a network optimization module, and a training output module; wherein,

[0045] The training scenario construction module is used to construct training scenarios based on the Gazebo simulator, including amphibious unmanned vehicles, obstacle cones, and walls.

[0046] The action decision module is used to input the current perception information obtained by the amphibious unmanned vehicle in the training scenario into the decision neural network based on the principle of machine learning and reinforcement learning. It uses a strategy that combines the greedy method and the curiosity-driven method to plan the trajectory and determine whether it is necessary to switch motion modes. The action command is then transmitted to the Gazebo simulator to command the amphibious unmanned vehicle in the Gazebo simulator to move.

[0047] The network optimization module is used to store the motion-related data of each step into the experience pool, and randomly extract several sets of data from the experience pool based on the n-step temporal difference method to optimize the decision neural network; then it switches to the action decision module until the amphibious unmanned vehicle triggers the termination condition or meets the preset training step length, then a training cycle is terminated.

[0048] The training output module is used to switch to the training scenario construction module to start a new training cycle if the preset total number of training cycles has not been met; otherwise, a trained decision neural network is obtained.

[0049] Compared with the prior art, the advantages of the present invention are:

[0050] 1. This invention proposes a method for training a modal decision simulator for amphibious unmanned vehicles based on reinforcement learning. The method employs the DDQN network in the reinforcement learning algorithm, effectively eliminating the problem of overestimation of the state-action value Q in reinforcement learning. It also incorporates a curiosity-driven mechanism to encourage the unmanned vehicle to explore, and reduces the estimation error of the state-action value Q through an n-step TD error estimation method.

[0051] 2. This invention proposes a simulator training environment that is compatible with reinforcement learning algorithms. This environment is based on the Gazabo simulator and follows the OpenAI-gym programming format, and is also highly adaptable to other reinforcement learning algorithms. The simulator models a real amphibious unmanned vehicle, reproducing its dynamic model in the real world and realistically simulating the information collection of the amphibious unmanned vehicle's sensors from the surrounding environment.

[0052] 3. This method is a reliable training method in the new field of multimodal autonomous driving, and provides an feasible solution for this field. Attached Figure Description

[0053] Figure 1 This is a schematic diagram of a simulation scene;

[0054] Figure 2 This is a LiDAR perception map;

[0055] Figure 3 This is the perception map after introducing the directional potential field;

[0056] Figure 4 It is a perceptual image after polar coordinate transformation;

[0057] Figure 5 It is a randomly generated training map;

[0058] Figure 6 It is a test map for amphibious unmanned vehicles;

[0059] Figure 7 It is a decision-making neural network structure;

[0060] Figure 8 This is a diagram of a curiosity network;

[0061] Figure 9 This is a flowchart of the training method for automatic mode switching and ground obstacle avoidance of the amphibious unmanned vehicle of the present invention;

[0062] Figure 10 This is the algorithm flowchart of the present invention. Detailed Implementation

[0063] In view of the above problems, several aspects of the present invention are proposed, providing a method and system for rapidly training automatic mode switching of amphibious unmanned vehicles, hereinafter referred to as amphibious unmanned vehicles or unmanned vehicles, and implementing them through the Gazabo simulation environment and a decision-making method based on reinforcement learning.

[0064] The first aspect of this invention is to provide a simulation platform that fully simulates the mode switching of amphibious vehicles, built on the Gazebo environment, enabling the amphibious unmanned vehicle simulation platform to be applicable to all mainstream reinforcement learning algorithms. The platform may include: an unmanned vehicle model in the simulator, fully simulating the perception and movement states of the amphibious vehicle in a real environment; obstacles in the simulator, simulating obstacles and treacherous terrain in the real environment; and a simulator-algorithm terminal interface, connecting the simulator and the decision network to maintain data transmission between them.

[0065] The second aspect of this invention is the design of a fast convergence decision algorithm based on Dueling DQN. This algorithm includes: a network primarily based on Dueling DQN, increasing network stability; the addition of a Prioritized Experience Replay mechanism to accelerate convergence; an n-step temporal difference method for network updates, increasing stability during the update process; and for the n-step temporal difference agent, a curiosity-driven exploration approach is adopted to increase the probability of the autonomous vehicle exploring the unknown state space.

[0066] The main process of this method is as follows:

[0067] First, the environment is set up in Gazabo, creating the necessary conditions for the amphibious unmanned vehicle. This includes obstacles to detect collisions and a field to determine if the vehicle has crossed the boundary. Then, the environment is initialized by placing the amphibious vehicle at the starting point and randomly arranging obstacles in the simulated training area. The training environment is now complete.

[0068] Next, the decision-making algorithm is trained. This algorithm is based on the principles of machine learning and reinforcement learning and is written in a Python script. The algorithm first interacts with the Gazebo environment. The autonomous vehicle in the simulation environment perceives its surroundings and feeds the perceived information into the decision neural network. The decision neural network makes decisions based on the information perceived by the autonomous vehicle and its current state, plans the trajectory, and determines whether to switch motion modes. This instruction is then passed to the simulator, which instructs the autonomous vehicle to move according to the physics engine in the Gazebo simulation environment. The above process is repeated until the autonomous vehicle collides with an obstacle, reaches the destination, or exceeds the site limitations.

[0069] Subsequently, by recording the obtained trajectory, the decision-making algorithm calculates the reward obtained by the autonomous vehicle for this trajectory, calculates the corresponding loss function of the neural network, and performs gradient descent based on the loss function to optimize and update the decision-making neural network. After the decision-making neural network is optimized, the Gazebo simulator rearranges the environment, and the autonomous vehicle moves again according to the instructions of the decision-making neural network, starting a new training cycle.

[0070] Repeat the above process until the decision neural network converges and the unmanned vehicle successfully reaches the destination from the starting point. Furthermore, the unmanned vehicle can find the path multiple times without collisions, indicating that the decision neural network has converged and can help the amphibious unmanned vehicle move.

[0071] The technical solution of the present invention will be described in detail below with reference to the accompanying drawings and embodiments.

[0072] Example 1

[0073] Embodiment 1 of the present invention proposes a training method for automatic mode switching and ground obstacle avoidance of amphibious unmanned vehicles.

[0074] The first aspect of this invention is the Gazebo simulation platform built according to the OpenAI gym format, which exhibits strong adaptability to different algorithms. Its scenario consists of three parts: an amphibious unmanned vehicle (UAV), obstacle cones, and walls restricting its movement range. Specifically, the amphibious UAV starts from a fixed starting point and moves towards a fixed endpoint position pre-set in the simulator. The UAV's movement range cannot exceed the area defined by the walls. During its movement, the UAV can freely switch between flight and ground driving modes; a collision with an obstacle results in failure. The scenario setup diagram is shown below. Figure 1 As shown.

[0075] To better simulate the trajectory of vehicles in reality, an Ackerman chassis was selected to build the kinematic model of the amphibious unmanned vehicle in the simulator's environment, and the motion space in the simulation environment was designed based on the chassis characteristics. The settings are shown in Table 1.

[0076] Table 1. Movement space of amphibious unmanned vehicles (Note: Right is the positive direction for turning, and left is the negative direction for turning).

[0077]

[0078]

[0079] In ground driving mode, actions [0,1,2,3,4,5] are valid actions for the unmanned vehicle; in flight mode, actions [5,6] are valid actions. The seven actions of the unmanned vehicle in the simulator can cover most motion states of unmanned vehicles in the real world, greatly increasing the realism of the simulator.

[0080] For the sensors of the amphibious unmanned vehicle in the simulator, this invention selects two sensors, lidar and altimeter, to collect information for network training.

[0081] LiDAR, serving as the input for obstacle perception, calculates map information within a certain range around the autonomous vehicle using the relative positions of surrounding obstacles in a coordinate system with the current autonomous vehicle as the origin. This local map is then enriched with information using a potential field, such as... Figure 2 As shown. To guide the amphibious unmanned vehicle's movement, another positive potential field is added to the perception map. The direction from the center of the perception map to the center of the potential field is the direction the unmanned vehicle will take towards its destination, as shown. Figure 3 As shown, by transforming the perception map with directional potential field from Euclidean space to polar coordinate space, the perception capability of the autonomous vehicle's decision-making neural network is enhanced, such as... Figure 4 As shown.

[0082] To increase the diversity of collected data, enhance the transferability of reinforcement learning decision networks, and thus improve the adaptability of autonomous vehicles to different environments, this invention randomly generates the positions and numbers of obstacles in the simulator before each training session, thereby generating different maps for the autonomous vehicle to collect data and iterate the decision network. Before each training session, the simulator randomly generates 0-5 obstacles, randomly distributed within a rectangular area of ​​length [5, 15] and width [-3.05, 3.05], as shown below. Figure 5 As shown.

[0083] In addition to the training map for the autonomous vehicle, this invention also designs a test map for the autonomous vehicle. To control for variables arising from map changes, objectively evaluate the training level of the autonomous vehicle's decision network, and test the vehicle's obstacle avoidance capabilities and takeoff / landing timing decisions, the obstacle positions in this map are fixed, such as... Figure 6 As shown, by repeatedly conducting independent tests on the same map, the autonomous vehicle's mode switching and autonomous driving capabilities were accurately evaluated.

[0084] The second aspect of this invention is: an automatic mode switching decision algorithm for autonomous vehicles based on the stable baseline 3 DQN algorithm. This invention improves and incorporates a Double DQN network, a Priority Experience Replay process, an N-step TD iterative update method, and a Curiosity-driven Exploration incentive mechanism, which significantly accelerates the training process and effectively solves the gradient explosion and gradient vanishing problems in neural networks. This allows for rapid and stable iteration of the decision neural network to update the autonomous vehicle's mode switching strategy.

[0085] The decision network in this invention takes as input a combination of two-dimensional images and one-dimensional data. For the input two-dimensional image signal, this invention employs a three-layer convolutional neural network for image information processing. The outputs of the convolutional neural network are stacked into a one-dimensional vector, which is then concatenated with height information as a vector and used as input to subsequent fully connected layers. After four fully connected layers, the corresponding state space Q-values ​​are output. The specific network architecture is as follows... Figure 7 As shown.

[0086] S1:

[0087] In the network described above, given the input observations, the output value corresponding to each action, also known as the state-action value function Q, can be expressed as Q(s). t ,a t ;θ t ), where s t Let a be the observation value at time t.t Let θ be the action at time t. t These are the parameters of the current network.

[0088] The Dueling DQN algorithm is based on the advantage function in reinforcement learning, and can be expressed as:

[0089] A * (s,a)=Q * (s,a)-V * (s)

[0090] Where A * (s,a) is the dominant function for executing a in state s, Q. * Given state s, execute the state value function V for state a in state s. * (s) is the value function in state s. A * (s,a) represents the advantage of performing action a compared to the average action under state s. The larger the value of a, the better the action is.

[0091] The maximum value is derived by taking the maximum value of both sides of Formula 1 with respect to action a. a A * (s,a), obtained through the following formula:

[0092] max a A * (s,a)=max a Q * (s,a)-V * (s)

[0093] The basic formula for Dueling DQN can be derived as follows:

[0094] Q * (s,a)=V * (s)+A * (s,a)-max a A * (s,a)

[0095] S2:

[0096] In the Dueling DQN network, action sampling is performed using the epsilon-greedy method, where the probability of random exploration in each step is ∈. The process is as follows: uniform sampling is performed within the range (0,1). If the sampled value is less than the epsilon value, the algorithm selects an action from the action space through random sampling. If the sampled value is greater than the epsilon value, the algorithm directly selects the action with the highest probability in the action probability distribution.

[0097] S3:

[0098] To enhance the agent's exploratory nature in unknown states, this invention employs both an epsilon-greedy network and a curiosity-incentivized network. The curiosity-incentivized network is constructed as follows: Figure 8 As shown in the image. Its workflow is as follows:

[0099] 1) First, similar to traditional reinforcement learning, the current state s is input. t Get the corresponding action a t `at`, and input the current action into the environment to obtain the action `s` corresponding to the next moment. t+1 .

[0100] 2) Using a fixed, non-updating feature extraction network, the input state at time t+1 is used to obtain the feature f. t+1 The feature f at time t+1 t+1 and the action a at time t t Simultaneously, the input is fed into the forward prediction generation network to obtain the predicted feature value f'. t+1

[0101] 3) Through f t+1 and f' t+1 The method of doing the difference yields an intrinsic incentive reward function.

[0102] 4) Construct f using the mean square loss function t+1 and f' t+1 The loss function is calculated, and the forward prediction generator network is continuously optimized by an optimizer, with the aim of making the prediction generator network more accurate.

[0103] S4:

[0104] To ensure that the parameter set update direction in the decision network conforms to the expected performance, this invention designs a specific reward signal for amphibious unmanned vehicles, as shown in Table 2:

[0105] Table 2 Reward Signal Design

[0106] Serial Number Reward signal 1 In ground driving mode, a reward of 50 is given for every 1 meter traveled along the x-axis. 2 For every 1 meter of backward movement along the x-axis in the ground driving mode, a penalty of 50 is incurred. 3 A penalty of 0.1 is applied for every 1 meter of deviation from the center line. 4 Each takeoff incurs a penalty of 10. 5 A penalty of 5 is incurred for each landing. 6 In flight mode, a penalty of 11 is applied for each training step. 7 The ground walking modality receives a reward of 1 for each training step. 8 A collision results in a 1000-point penalty, ending the training cycle and returning to the starting point. 9 Reaching the finish line earns a 100-point reward and ends the training cycle; return to the starting point.

[0107] S5:

[0108] The reward signal in step S3 provides the reward corresponding to each step of the autonomous vehicle. For the reward signal, this invention uses a Dueling DQN network estimation method to calculate the state-action function Q. Unlike the traditional DQN algorithm, this invention uses two networks to predict the state-action function Q value, reducing the impact of overestimation on the final policy. The calculation of the predicted Q value is as follows:

[0109]

[0110] The DQN algorithm in stable baseline3 only uses the traditional DQN formula. This invention improves the stable baseline3 network by applying Double DQN to estimate the state-action value function Q, reducing overestimation and accelerating the convergence of the algorithm.

[0111] S6:

[0112] Traditional DQN algorithms store collected data in a data pool (buffer) during interaction with the environment. During network updates, data in the data pool is uniformly sampled, and the sampled data is used for network iteration. Unlike traditional DQN, this invention uses a Prioritized ExperienceReplay Buffer method. This method determines the priority of each data set based on its temporal error, ensuring that data with larger temporal errors are trained first, and repeatedly sampling high-priority data in subsequent training. This method ensures that data containing more information is prioritized for network input, globally accelerating network learning. The formula for calculating temporal error is:

[0113]

[0114] The traditional DQN algorithm updates using a one-step time difference algorithm, i.e.

[0115]

[0116] in To estimate the Q value, R t+1 Let γ be the reward at time t+1, and γ be the discount rate. For the parameters of the target network, As the training network Q(s) t ,a t ;θ t The target value is updated in a way that easily leads the neural network into local optima, ignoring the global optimum and ultimately degrading the performance of the decision network. To obtain a more accurate Q-value estimate, this invention uses an n-step temporal difference method, the specific calculation formula of which is as follows:

[0117]

[0118] The n-step temporal difference method can effectively compensate for the overestimation problem caused by single-step temporal difference updates by sampling data in multiple steps.

[0119] Combining the simulator environment and algorithms S1-S5, the specific implementation method of this invention is as follows: Figure 9 As shown,

[0120] 1) Initialize the DQN network Curiosity Network f target (s0,a0), f(s0,a0), initializes the hyperparameters n0, N, ∈, k, n, w, γ, where s0 is the initial state, and a0 is the action collected based on the initial state and the simulator training environment is reset, i.e., a new training environment is randomly generated. The autonomous vehicle uses a random action sampling exploration method to explore for n0 steps, and the collected data is stored in the experience buffer.

[0121] 2) Begin a reinforcement learning training cycle.

[0122] 3) Begin a training step in this reinforcement learning cycle.

[0123] 4) In this step, the surrounding environment is first observed and observations are collected.

[0124] 5) Based on the observations, select actions. The action selection process is the same as step S2 described above. Action sampling is performed using a combination of greedy and curiosity-driven strategies.

[0125] 6) Calculate the report obtained by the autonomous vehicle for this action, and calculate the corresponding reward signal according to Table 2 through process S3.

[0126] 7) Execute the action in the environment to obtain the next observation information and store the previous information in the experience buffer.

[0127] 8) Train the curiosity network using observations and actions, with the loss being:

[0128]

[0129] 9) Extract k sets of data from the buffer, and obtain the estimated target state-action value Q, which is Y, according to the state-action value estimation method of the n-step TD formula in step S4. t .

[0130] 10) Train the Dueling DQN network using gradient descent. The loss calculation formula is as follows:

[0131]

[0132] At the same time, the priority of the corresponding data in the experience buffer is updated according to the loss.

[0133] 11) Every m steps, the Adam optimizer optimizes the neural network based on the calculated loss function and updates the network weights once.

[0134] 12) End a training step.

[0135] 13) Update ∈,w based on the number of training steps.

[0136] 14) End a training cycle.

[0137] 15) Repeat the above training process 2)-14).

[0138] A schematic diagram of the reinforcement learning algorithm can be seen. Figure 10 .

[0139] Example 2

[0140] Embodiment 2 of the present invention proposes a training system for automatic mode switching and ground obstacle avoidance of an amphibious unmanned vehicle, implemented based on the method of Embodiment 1. The system includes: a training scenario construction module, an action decision module, a network optimization module, and a training output module; wherein,

[0141] The training scenario construction module is used to construct training scenarios based on the Gazebo simulator, including amphibious unmanned vehicles, obstacle cones, and walls.

[0142] The action decision module is used to input the current perception information obtained by the amphibious unmanned vehicle in the training scenario into the decision neural network based on the principle of machine learning and reinforcement learning. It uses a strategy that combines the greedy method and the curiosity-driven method to plan the trajectory and determine whether it is necessary to switch motion modes. The action command is then transmitted to the Gazebo simulator to command the amphibious unmanned vehicle in the Gazebo simulator to move.

[0143] The network optimization module is used to store the motion-related data of each step into the experience pool, and randomly extract several sets of data from the experience pool based on the n-step temporal difference method to optimize the decision neural network; then it switches to the action decision module until the amphibious unmanned vehicle triggers the termination condition or meets the preset training step length, then a training cycle is terminated.

[0144] The training output module is used to switch to the training scenario construction module to start a new training cycle if the preset total number of training cycles has not been met; otherwise, a trained decision neural network is obtained.

[0145] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to the embodiments, those skilled in the art should understand that modifications or equivalent substitutions to the technical solutions of the present invention do not depart from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A training method for automatic mode switching and ground obstacle avoidance of an amphibious unmanned vehicle, characterized in that, The method includes: Step 1) Construct a training scenario based on the Gazebo simulator, including an amphibious unmanned vehicle, obstacle cones, and walls; the amphibious unmanned vehicle is based on an Ackerman chassis and a kinematic model is established, defining a variety of effective actions in both ground driving and flight modes; Step 2) Based on the principles of machine learning and reinforcement learning, the current-moment perception information acquired by the amphibious unmanned vehicle in the training scenario is input into the decision neural network, and then... A strategy combining greedy and curiosity-driven methods plans the trajectory and determines whether a motion mode switch is needed. Action commands are then transmitted to the Gazebo simulator to direct the movement of the amphibious unmanned vehicle within the simulator. The decision neural network is a DDQN network, with inputs consisting of a 2D image acquired by a lidar and altitude information obtained by an altimeter, and an output consisting of an action value function. value; Step 3) Store the motion-related data of each step into the experience pool, and randomly select several sets of data from the experience pool based on the n-step temporal difference method to optimize the decision neural network; then go to step 2) until the amphibious unmanned vehicle triggers the termination condition or meets the preset training step length, then terminate a training cycle. Step 4) If the preset total number of training cycles has not been reached, proceed to Step 1) to start a new training cycle; otherwise, the trained decision neural network is obtained.

2. The training method for automatic mode switching and ground obstacle avoidance of an amphibious unmanned vehicle according to claim 1, characterized in that, Step 1) includes: Several obstacle cones are randomly generated and randomly distributed within a rectangular area of ​​the wall with a set length and width. Set a fixed starting point and a fixed ending point for the amphibious unmanned vehicle's movement; The amphibious unmanned vehicle is set to move within a rectangular area of ​​the wall. During its movement, it can freely switch between flight and ground driving modes. It will fail if it collides with an obstacle cone or wall.

3. The training method for automatic mode switching and ground obstacle avoidance of an amphibious unmanned vehicle according to claim 2, characterized in that, The effective actions of the amphibious unmanned vehicle include: in ground driving mode, actions [0,1,2,3,4,5] are effective actions; in flight mode, actions [5,6] are effective actions. Action [0] is to move straight at the set wheel rotation speed; The action [1] includes the set linear velocity of the wheel rotation, the angular velocity of the wheel rotation is the first wheel rotation velocity, and the direction is to the right; The action [2] includes the set linear velocity of the wheel rotation, the angular velocity of the wheel rotation is the second wheel rotation speed, and the direction is to the right; The action [3] includes the set linear velocity of the wheel rotation, the angular velocity of the wheel rotation is the first wheel rotation velocity, and the direction is to the left; The action [4] includes the set linear velocity of the wheel rotation, the angular velocity of the wheel rotation is the second wheel rotation speed, and the direction is to the left; Action [5] provides the lift required for takeoff or forward flight by rotating the rotor; Action [6] is to slow down the rotor speed and land the unmanned vehicle stably.

4. The training method for automatic mode switching and ground obstacle avoidance of an amphibious unmanned vehicle according to claim 1, characterized in that, The decision neural network comprises a three-layer convolutional neural network, two parallel four-layer fully connected layers, and a summation function; the specific processing includes: A three-layer convolutional neural network processes the input two-dimensional image. The outputs of the convolutional neural network are stacked into a one-dimensional vector, which is then concatenated with height information in two separate paths. This concatenation is used as the input to two fully connected layers, passed through four fully connected layers, and finally summed by a summing function to output the corresponding action value function. value: ,in, , for , These are the parameters of the current decision neural network.

5. The training method for automatic mode switching and ground obstacle avoidance of an amphibious unmanned vehicle according to claim 1, characterized in that, Step 2) Greedy methods include: Action value function The value is uniformly sampled within the range (0,1). If the sampled value is less than the set epsilon value, the action is selected by random sampling. If the sampled value is greater than the set epsilon value, the action with the highest probability in the action probability distribution is selected.

6. The training method for automatic mode switching and ground obstacle avoidance of an amphibious unmanned vehicle according to claim 4, characterized in that, The curiosity-driven method in step 2) includes: Enter the current status Get the corresponding action It inputs the current action into the environment to obtain the state at the next moment. ; Features are obtained through a pre-established feature extraction network. ,Will and Simultaneously, the input is fed into the forward prediction generation network to obtain the predicted feature values. ; Through the and The intrinsic incentive reward function is obtained by doing the difference. ; Constructing using the mean square loss function and The loss function is calculated, and the forward prediction generator network is continuously optimized through a prediction optimizer to make the predictions more accurate.

7. The training method for automatic mode switching and ground obstacle avoidance of an amphibious unmanned vehicle according to claim 6, characterized in that, The above through and The intrinsic incentive reward function is obtained by doing the difference. Specifically, it includes: When the ground driving mode moves 1m along the x-axis ; When the ground driving mode moves backward by 1m along the x-axis ; For every 1m deviation from the center line ; With each takeoff ; With each landing, ; When the training step length in flight mode ; When the training step length is in the ground mode ; When a collision occurs ; When you reach the finish line, .

8. The training method for automatic mode switching and ground obstacle avoidance of an amphibious unmanned vehicle according to claim 7, characterized in that, The motion-related data for each step in step 3) includes the current state. ,action Intrinsic motivation and pre-set external stimuli generated by the environment .

9. A training system for automatic mode switching and ground obstacle avoidance of an amphibious unmanned vehicle, characterized in that, The system includes: a training scenario construction module, an action decision module, a network optimization module, and a training output module; wherein, The training scenario construction module is used to construct training scenarios based on the Gazebo simulator, including amphibious unmanned vehicles, obstacle cones, and walls; the amphibious unmanned vehicles are based on the Ackerman chassis to establish a kinematic model and define a variety of effective actions including ground driving mode and flight mode; The action decision module, based on machine learning and reinforcement learning principles, inputs the current-moment perception information acquired by the amphibious unmanned vehicle in the training scenario into a decision neural network, employing... A strategy combining greedy and curiosity-driven methods plans the trajectory and determines whether a motion mode switch is needed. Action commands are then transmitted to the Gazebo simulator, directing the amphibious unmanned vehicle within the simulator to move. The decision neural network is a DDQN network, with inputs consisting of a 2D image acquired by a lidar and altitude information obtained by an altimeter, and an output consisting of an action value function. value; The network optimization module is used to store the motion-related data of each step into the experience pool, and randomly extract several sets of data from the experience pool based on the n-step temporal difference method to optimize the decision neural network; then it switches to the action decision module until the amphibious unmanned vehicle triggers the termination condition or meets the preset training step length, then a training cycle is terminated. The training output module is used to switch to the training scenario construction module to start a new training cycle if the preset total number of training cycles has not been met; otherwise, a trained decision neural network is obtained.

Images