Robot navigation method based on multi-mode inverse reinforcement learning

Abstract

The invention provides a robot navigation method based on multi-mode inverse reinforcement learning, which comprises the following steps: acquiring robot pose data, acquiring laser radar data, and generating a grid map; the laser radar data, the raster map and the robot pose information generate fusion observation representation through a multi-modal fusion network; initializing network parameters, taking the fusion observation representation as expert demonstration track data, and pre-training a generator and a discriminator in a generative adversarial inverse reinforcement learning framework through an expert demonstration track to obtain a pre-training model; standardizing a reward function through expert trajectory data; and constructing a generative adversarial reinforcement learning overall framework, training to obtain an intelligent control model, and deploying the model to realize intelligent control of the robot. According to the method, multi-modal data fusion and generative adversarial reinforcement learning can be realized, and the performance of the robot in the aspect of navigation is effectively improved.

Description

Technical Field

[0001] The present invention belongs to the field of robot intelligent navigation, and in particular relates to a robot navigation method based on multimodal inverse reinforcement learning. Background Art

[0002] Robot navigation technology is one of the research hotspots of mobile robots. Currently, robot intelligent navigation has been widely used in various scenarios. In these scenarios, robots need to autonomously cope with complex environments and complete a series of instructions to achieve mission objectives. To achieve this goal, robots need to make multiple navigation decisions, that is, complete path planning while performing dynamic obstacle avoidance.

[0003] However, current robot navigation technology still faces many challenges. Traditional robot navigation methods usually rely on single sensor information, which leads to limitations. For example, lidar has difficulty identifying transparent objects, and cameras are easily affected by changes in lighting, which makes it difficult for robots to fully and accurately perceive the surrounding environment, thus affecting navigation performance. In addition, the real world environment is complex and changeable, and robots need to make reasonable path planning and obstacle avoidance decisions based on real-time perceived environmental information. In addition, navigation methods based on deep reinforcement learning are difficult to design appropriate reward functions in complex environments, resulting in poor performance of robots.

[0004] CN117369458A discloses a local path planning method for outdoor complex environments based on inverse reinforcement learning, which uses GPS, cameras and laser radar on the robot to collect environmental information, and integrates the data of different sensors through data fusion algorithms. However, this technology relies on accurate sensor data and complex optimization algorithms, has high requirements on sensor accuracy and matching degree, and has low tolerance for sensor noise and environmental changes. CN117723064A discloses a navigation method and system for intelligent inspection robots with multi-sensor data fusion, which improves navigation function through hardware design and visual modules, but mainly relies on pre-set rules and simple visual recognition, has limited perception ability for complex environments, and lacks global path planning ability. CN118289110A discloses a multimodal intelligent perception quadruped bionic detection robot and its control method, which uses inverse reinforcement learning and particle swarm algorithm to optimize local paths, collect images in real time and calculate the optimal path, and adapt to outdoor complex environments, but this technology does not have a global vision, is prone to fall into local optimal solutions, and is greatly affected by the quality of expert demonstrations, and is prone to unstable training processes due to differences in preferences of expert examples.

[0005] In recent years, some new robot navigation technologies have emerged, such as multimodal information fusion and inverse reinforcement learning, which provide new ideas for solving robot navigation problems. However, effectively fusing sensor information of different modalities is still a challenge. At the same time, high-quality expert demonstrations are expensive and difficult to obtain, and different preferences of expert demonstrations may lead to differences in performance. Summary of the invention

[0006] In order to overcome the above technical problems, the purpose of the present invention is to provide a robot navigation method based on multimodal inverse reinforcement learning. By constructing a multimodal fusion network to fuse lidar, grid map and robot posture, a new observation representation is generated as reinforcement learning observation information to achieve alignment and fusion of different modalities. In addition, generative adversarial inverse reinforcement learning is used to decouple complex tasks into several subtasks, multiple discriminators are used for each subtask, and the reward function of each subtask is weighted and fused. The merged standardized reward function is used to train the strategy, which effectively reduces the errors caused by different preferences of expert demonstrations;

[0007] In order to achieve the above object, the present invention adopts the following technical solutions:

[0008] A robot navigation method based on multimodal inverse reinforcement learning comprises the following steps:

[0009] S1: Obtain robot posture data, collect lidar data, and generate a grid map;

[0010] S2: LiDAR data, grid map and robot pose information generate fused observation representation through a multimodal fusion network;

[0011] S3: Initialize network parameters, use the fused observation representation as expert demonstration trajectory data, and pre-train the generator and discriminator in the generative adversarial reinforcement learning framework through the expert demonstration trajectory to obtain the pre-trained model;

[0012] S4: Normalize the reward function using expert trajectory data;

[0013] S5: Build a generative adversarial reinforcement learning framework, obtain an intelligent control model after training, and deploy the model to realize intelligent control of the robot.

[0014] Preferably, step S1 obtains robot posture data, collects laser radar data, and generates a grid map, specifically:

[0015] Step S1.1: The wheel odometer uses the speed and angular velocity obtained by observing the left and right wheel speeds to substitute into the kinematic model of the mobile robot to deduce the current posture information P of the robot;

[0016] Step S1.2: The collected odometer data and lidar data R are corrected for each laser beam using a coordinate system conversion method;

[0017] Step S1.3: Generate a grid map M based on the corrected lidar data.

[0018] Preferably, step S2 fuses the lidar data, grid map and robot posture information into a new observation representation through a neural network model, specifically:

[0019] Step S2.1: The grid map M is passed through the convolutional layer to obtain the grid map representation F M ,There are three convolutional layers, namely CL1, CL2 and CL3, and each convolutional layer is followed by the activation function ReLU;

[0020] Step S2.2: The lidar data R is passed through the convolutional layer to obtain the radar representation F R ,There are two convolutional layers, CL4 and CL5, and each convolutional layer is followed by the activation function ReLU;

[0021] Step S2.3: Grid map representation F M and radar characterization F R After fusion at the fully connected layer, the fusion representation F is obtained MR , where there are two fully connected layers, namely FCL1 and FCL2, and each fully connected layer is followed by the activation function ReLU;

[0022] Step S2.4: The pose information P is passed through the fully connected layer to obtain the pose representation F P Among them, there is one fully connected layer, which is FCL3, and the activation function ReLU is used after the fully connected layer;

[0023] Step S2.5: Pose representation F P and characterization F MR After fusion at the fully connected layer, the fusion representation F is obtained MRP , among which, there is one fully connected layer, which is FCL4.

[0024] Preferably, step S3 initializes the network parameters, uses the fused observation representation as the expert demonstration trajectory data, and pre-trains the generator and discriminator in the generative adversarial reinforcement learning framework through the expert demonstration trajectory to obtain a pre-trained model, specifically:

[0025] Step S3.1: Set a total of m subtasks, and the expert demonstration trajectory set S consists of n expert trajectories, which are expressed as follows:

[0026] S=[τ1,τ2,…,τ n ]

[0027] Among them, the i-th expert trajectory τ i It is expressed as follows:

[0028]

[0029] Among them, (x, a) represents a state-action pair, j represents a moment in the task cycle, and n i represents the cycle length of a trajectory;

[0030] Step S3.2: Initialize the generator G(·|θ G ) and m discriminators

[0031] Step S3.3: For each expert trajectory Generator G(·|θ G ) will first follow the state sequence Generate action sequence

[0032] The j-th discriminator according to To determine the action from the generated action Or expert action a i ;

[0033] Step S3.4: Set the joint optimization objective of the generator and the j-th discriminator as follows:

[0034]

[0035] Pre-train the generator and m discriminators according to a joint optimization objective.

[0036] Preferably, step S4 normalizes the reward function using expert trajectory data, specifically:

[0037] Step S4.1: Calculate the mean μ of the reward function i and standard deviation σ i , the calculation method is as follows:

[0038]

[0039] Among them, n i represents the length of the expert trajectory;

[0040] Step S4.2: According to the mean μ i and standard deviation σ i , set the standardized reward function As shown below:

[0041]

[0042] Preferably, step S5 constructs a generative adversarial reinforcement learning overall framework, obtains an intelligent control model after training, and deploys the model to realize intelligent control of the robot, specifically:

[0043] Step S5.1: Policy network π(x t |θ G ) According to the state x t Generate Action Generate state-action pairs The sequence is regarded as the generated trajectory

[0044] Step S5.2: Objective function D of the discriminator of the basic generative adversarial network * (·) means the following:

[0045]

[0046] Among them, p(·) represents the true distribution and q(·) represents the generator distribution;

[0047] Introducing the Boltzmann distribution c θ (·) replaces p(·), then the objective function is expressed as:

[0048]

[0049] Among them, 1 / Z represents the distribution adjustment coefficient;

[0050] The objective function can be further simplified as:

[0051]

[0052] Among them, f θ (s,a) represents the reward function;

[0053] According to the objective function D θ (·), setting the objective function d of the i-th subtask in generative adversarial reinforcement learning i (x,a), as shown below:

[0054]

[0055] Among them, r i By the discriminator D(·|θ D ) is given as follows:

[0056] r 1,t ,...,r m,t =D(x t ,a t |θ D )

[0057] Step S5.3: Set the value network objective function L c ;

[0058] Value network objective function L c It consists of two parts, namely the weight network optimization objective And the state value function optimization goal

[0059] Set the weight network optimization target As shown below:

[0060]

[0061] in, represents the jth reward at time t, w i (·) represents the jth weight network, γ represents the discount factor, and θ C Represents the value network parameters, including θ G and the parameters of the weight network;

[0062] State value function optimization goal as follows:

[0063]

[0064] Among them, g(·) represents the weight parameter of w(·) after being processed by the softmax function. The process is as follows:

[0065]

[0066] Step S5.4: Calculate the advantage function. The advantage function calculation method is as follows:

[0067]

[0068] Step S5.5: Set the objective function L of the policy network a The optimization objectives are as follows:

[0069]

[0070] Among them, clip represents the clipping function, ∈ represents the boundary parameter;

[0071] The gradient calculation method is as follows:

[0072]

[0073] Among them, θ' G represents the historical parameters of the generator, and p(a|x) represents the policy distribution.

[0074] Compared with the prior art, the present invention has the following beneficial effects:

[0075] (1) The present invention provides the robot with more comprehensive and accurate environmental perception information through a multimodal fusion network, and reduces the behavioral differences caused by expert demonstrations with different preferences through a generative inverse reinforcement learning method, thereby improving the accuracy of the reward function and the generalization ability of the strategy.

[0076] (2) The present invention can be effectively applied to robot dynamic obstacle avoidance and path planning scenarios in complex environments, which can effectively improve the robot's navigation capability while accelerating the training convergence speed. BRIEF DESCRIPTION OF THE DRAWINGS

[0077] Figure 1 It is a flow chart of the present invention;

[0078] Figure 2 Schematic diagram of the multimodal fusion network structure in the present invention;

[0079] Figure 3 A schematic diagram of a generative adversarial reinforcement learning method in the present invention;

[0080] Figure 4 Schematic diagram of the generator training process in the present invention. DETAILED DESCRIPTION

[0081] Unless otherwise specified, any feature disclosed throughout this specification may be replaced by an alternative feature that is equivalent or has a similar purpose. That is, each feature is only an example in a series of equivalent or similar features.

[0082] In order to make the purpose, technical solution and advantages of the present invention clearer, the present invention will be described in more detail below in conjunction with the drawings of the specification. It should be noted that the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0083] Example:

[0084] Based on previous work, this paper innovatively proposes a robot navigation method based on multimodal inverse reinforcement learning. Multimodal fusion technology is used to construct expert demonstrations containing rich environmental information, and a generative adversarial inverse reinforcement learning method is used to learn the reward function from the expert demonstrations.

[0085] Reference Figure 1 , a robot navigation method based on multimodal inverse reinforcement learning provided by the present invention includes:

[0086] Step S1 obtains robot posture data, collects lidar data, and generates a grid map:

[0087] Step S1.1: The wheel odometer uses the speed and angular velocity obtained by observing the rotation speed of the left and right wheels, substitutes them into the kinematic model of the mobile robot, and deduce the current posture information P of the robot.

[0088] Step S1.2: The collected odometer data and lidar data R are corrected for each laser beam using the coordinate system conversion method.

[0089] Step S1.3: Generate a grid map M based on the corrected LiDAR data. The grid map model divides the environment into independent grids of the same size, and each grid represents the area contained in the environment.

[0090] Reference Figure 2 , step S2 fuses the lidar data, grid map and robot pose information into a new observation representation through a neural network model:

[0091] Step S2.1: The grid map M is passed through the convolutional layer to obtain the grid map representation F M , where there are three convolution layers, namely CL1, CL2 and CL3, and each convolution layer is followed by the activation function ReLU. In the embodiment of the present invention, the sizes of the convolution kernels of CL1, CL2 and CL3 are 8*8, 4*4 and 4*4 respectively, the step sizes are 4*4, 2*2 and 2*2 respectively, and the number of convolution kernels is 32.

[0092] Step S2.2: The lidar data R is passed through the convolutional layer to obtain the radar representation F R There are two convolution layers, CL4 and CL5, and each convolution layer is followed by the activation function ReLU. In the embodiment of the present invention, the sizes of the convolution kernels of CL4 and CL5 are 8*8 and 4*4, respectively, the step sizes are 4*4 and 2*2, respectively, and the number of convolution kernels is 32.

[0093] Step S2.3: Grid map representation F M and radar characterization F R After fusion at the fully connected layer, the fusion representation F is obtained MR , wherein there are two fully connected layers, namely FCL1 and FCL2, and each fully connected layer is followed by an activation function ReLU. In the embodiment of the present invention, the number of neurons in FCL1 and FCL2 is 128.

[0094] Step S2.4: The pose information P is passed through the fully connected layer to obtain the pose representation F P , wherein there is one fully connected layer, namely FCL3, after which activation function ReLU is applied. In the embodiment of the present invention, FCL3 has 128 neurons.

[0095] Step S2.5: Pose representation F P and characterization F MR After fusion at the fully connected layer, the fusion representation F is obtained MRP , wherein there is one fully connected layer, namely FCL4. In the embodiment of the present invention, the number of neurons of FCL4 is 128.

[0096] Step S3 initializes the network parameters, takes the fused observation representation as the expert demonstration trajectory data, and pre-trains the generator and discriminator in the generative adversarial reinforcement learning framework through the expert demonstration trajectory to obtain the pre-trained model:

[0097] Reference Figure 3 The generative adversarial inverse reinforcement learning framework separates strategy execution from value assessment and uses a generative adversarial network to learn the reward function through expert demonstration. The generator plays the role of the policy network, while the value network consists of a discriminator and a weight network. In addition, the representation learning module is used as an alternative: if the dimensionality transformation of the fused observation representation is required, it can be adjusted through the representation learning module; if there is no such requirement, this module can be omitted.

[0098] Step S3.1: Set a total of m subtasks, and the expert demonstration trajectory set S consists of n expert trajectories, which are expressed as follows:

[0099] S=[τ1,τ2,…,τ n ]

[0100] Among them, the i-th expert trajectory τ i It is expressed as follows:

[0101]

[0102] Among them, (x, a) represents a state-action pair, j represents a moment in the task cycle, and n i Represents the cycle length of a trajectory.

[0103] Step S3.2: Initialize the generator G(·|θ G ) and m discriminators

[0104] Step S3.3: For each expert trajectory Generator G(·|θ G ) will first follow the state sequence Generate action sequence The j-th discriminator according to To determine the action from the generated action Or expert action a i .

[0105] Step S3.4: Set the joint optimization objective of the generator and the j-th discriminator as follows:

[0106]

[0107] Pre-train the generator and m discriminators according to a joint optimization objective.

[0108] Step S4 normalizes the reward function by the expert trajectory data:

[0109] In order to stabilize the training, the reward function used in each iteration needs to be standardized. This is because the reward function predicted by the discriminator is used to train the generator policy during the generator training process. A fixed reward function can often ensure the smooth convergence of the policy network. However, the reward function will be updated during each iteration. Directly using the reward function provided by the discriminator will increase the volatility of the policy network training. Therefore, after each round of discriminator training, the reward function predicted by the discriminator is standardized using expert demonstrations.

[0110] Step S4.1: Calculate the mean μ of the reward function i and standard deviation σ i , the calculation method is as follows:

[0111]

[0112] Among them, n i Represents the length of the expert trajectory.

[0113] Step S4.2: According to the mean μ i and standard deviation σ i , set the standardized reward function As shown below:

[0114]

[0115] Step S5 builds the overall framework of generative adversarial reinforcement learning, obtains the intelligent control model after training, and deploys the model to realize the intelligent control of the robot:

[0116] Step S5.1: Policy network π(x t |θ G ) According to the state x t Generate Action Generate state-action pairs The sequence is regarded as the generated trajectory

[0117] Step S5.2: Objective function D of the discriminator of the basic generative adversarial network * (·) means the following:

[0118]

[0119] Among them, p(·) represents the true distribution and q(·) represents the generator distribution. Introducing the Boltzmann distribution c θ (·) replaces p(·), then the objective function is expressed as:

[0120]

[0121] Among them, 1 / Z represents the distribution adjustment coefficient. The objective function can be further simplified as:

[0122]

[0123] Among them, f θ (s,a) represents the reward function. According to the objective function D θ (·), setting the objective function d of the i-th subtask in generative adversarial reinforcement learning i (x,a), as shown below:

[0124]

[0125] Among them, r i By the discriminator D(·|θ D ) is given as follows:

[0126] r 1,t ,…,r m,t =D(x t ,a t |θ D )

[0127] Step S5.3: Set the value network objective function L c . Reference Figure 4 , the policy network consists of two parts: the subtask network and the weight network. As described in step S3, the subtask network is Figure 3 The discriminator in is pre-trained with expert demonstrations. The output packet of the value network represents m weights and the state value function. During the training process, the policy network (also Figure 3 The generator in ) makes decisions based on the state-value function.

[0128] Value network objective function L c It consists of two parts, namely the weight network optimization objective And the state value function optimization goal Set the weight network optimization target As shown below:

[0129]

[0130] in, represents the jth reward at time t, w i (·) represents the jth weight network, γ represents the discount factor, and θ C Represents the value network parameters, including θ G and the parameters of the weight network. The state value function optimizes the objective as follows:

[0131]

[0132] Among them, g(·) represents the weight parameter of w(·) after being processed by the softmax function. The process is as follows:

[0133]

[0134] Step S5.4: Calculate the advantage function. The advantage function calculation method is as follows:

[0135]

[0136] Step S5.5: Set the objective function L of the policy network a The optimization objectives are as follows:

[0137]

[0138] Among them, clip represents the clipping function, ∈ represents the boundary parameter. The gradient calculation method is as follows:

[0139]

[0140] Among them, θ' G represents the historical parameters of the generator, and p(a|x) represents the policy distribution.

[0141] The above embodiments are only preferred embodiments of the present invention, and are not intended to limit the scope of the present invention. For those skilled in the art, without departing from the spirit of the present invention, the present invention may be subjected to several deformations, improvements and equivalent substitutions, which are deemed to fall within the scope of protection of the claims of the present invention.

Claims

1. A robot navigation method based on multimodal inverse reinforcement learning, It is characterized in that include: Step S1: Obtain robot posture data, collect lidar data, and generate a grid map; Step S2: The lidar data, grid map and robot pose information are used to generate fused observation representation through a multimodal fusion network; Step S3: Initialize the network parameters, use the fused observation representation as the expert demonstration trajectory data, and pre-train the generator and discriminator in the generative adversarial reinforcement learning framework through the expert demonstration trajectory to obtain the pre-trained model; Step S4: normalize the reward function by expert trajectory data; Step S5: Construct the overall framework of generative adversarial reinforcement learning, obtain the intelligent control model after training, and deploy the model to realize the intelligent control of the robot.

2. The robot navigation method based on multimodal inverse reinforcement learning according to claim 1, It is characterized in that Step S1 includes: Step S1.1: The wheel odometer uses the speed and angular velocity obtained by observing the left and right wheel speeds to substitute into the kinematic model of the mobile robot to deduce the current posture information P of the robot; Step S1.2: The collected odometer data and lidar data R are corrected for each laser beam using a coordinate system conversion method; Step S1.3: Generate a grid map M based on the corrected lidar data.

3. The robot navigation method based on multimodal inverse reinforcement learning according to claim 1, It is characterized in that Step S2 includes: Step S2.1: The grid map M is passed through the convolutional layer to obtain the grid map representation F M , where there are three convolutional layers, namely CL 1 , CL 2 and CL 3 , each convolutional layer is followed by the activation function ReLU; Step S2.2: The lidar data R is passed through the convolutional layer to obtain the radar representation F R , where there are two convolutional layers, namely CL 4 and CL 5 , each convolutional layer is followed by the activation function ReLU; Step S2.3: Grid map representation F M and radar characterization F R After fusion at the fully connected layer, the fusion representation F is obtained MR , where the fully connected layer has two layers, namely FCL 1 and FCL 2 , each fully connected layer is followed by the activation function ReLU; Step S2.4: The pose information P is passed through the fully connected layer to obtain the pose representation F P , where there is one fully connected layer, FCL 3 , after the fully connected layer, the activation function ReLU is used; Step S2.5: Pose representation F P and characterization F MR After fusion at the fully connected layer, the fusion representation F is obtained MRP , where there is one fully connected layer, FCL 4 .

4. The robot navigation method based on multimodal inverse reinforcement learning according to claim 1, It is characterized in that Step S3 includes: Step S3.1: Set a total of m subtasks, and the expert demonstration trajectory set S consists of n expert trajectories, which are expressed as follows: S=[τ 1 ,t 2 ,…,t n ] Among them, the i-th expert trajectory τ i It is expressed as follows: Among them, (x, a) represents a state-action pair, j represents a moment in the task cycle, and n i represents the cycle length of a trajectory; Step S3.2: Initialize the generator G(·|θ G ) and m discriminators Step S3.3: For each expert trajectory Generator G(·|θ G ) will first follow the state sequence Generate action sequence The j-th discriminator according to To determine the action from the generated action Or expert action a i ; Step S3.4: Set the joint optimization objective of the generator and the j-th discriminator as follows: Pre-train the generator and m discriminators according to a joint optimization objective.

5. The robot navigation method based on multimodal inverse reinforcement learning according to claim 1, It is characterized in that Step S4 includes: Step S4.1: Calculate the mean μ of the reward function i and standard deviation σ i , the calculation method is as follows: Among them, n i represents the length of the expert trajectory; Step S4.2: According to the mean μ i and standard deviation σ i , set the standardized reward function As shown below:

6. The robot navigation method based on multimodal inverse reinforcement learning according to claim 1, It is characterized in that Step S5 includes: Step S5.1: Policy network π(x t |θ G ) According to the state x t Generate Action Generate state-action pairs The sequence is regarded as the generated trajectory Step S5.2: Objective function D of the discriminator of the basic generative adversarial network * (·) means the following: Among them, p(·) represents the true distribution and q(·) represents the generator distribution; Introducing the Boltzmann distribution c θ (·) replaces p(·), then the objective function is expressed as: Among them, 1 / Z represents the distribution adjustment coefficient; The objective function can be further simplified as: Among them, f θ (s,a) represents the reward function; According to the objective function D θ (·), setting the objective function d of the i-th subtask in generative adversarial reinforcement learning i (x,a), as shown below: Among them, r i By the discriminator D(·|θ D ) is given as follows: r 1,t ,…,r m,t =D(x t ,a t |θ D ) Step S5.3: Set the value network objective function L c ; Value network objective function L c It consists of two parts, namely the weight network optimization objective And the state value function optimization goal Set the weight network optimization target As shown below: in, represents the jth reward at time t, w i (·) represents the jth weight network, γ represents the discount factor, and θ C Represents the value network parameters, including θ G and the parameters of the weight network; State value function optimization goal as follows: Among them, g(·) represents the weight parameter of w(·) after being processed by the softmax function. The process is as follows: Step S5.4: Calculate the advantage function. The advantage function calculation method is as follows: Step S5.5: Set the objective function L of the policy network a The optimization objectives are as follows: Among them, clip represents the clipping function, ∈ represents the boundary parameter; The gradient calculation method is as follows: Among them, θ' G represents the historical parameters of the generator, and p(a|x) represents the policy distribution.

Images