An indoor navigation method based on vision and radar information fusion and reinforcement learning

Abstract

The application discloses an indoor navigation method based on visual and radar information fusion and reinforcement learning. In order to overcome the problem that the robot has poor environmental perception ability in the existing robot autonomous navigation technology, resulting in low robot navigation success rate in unknown environment, the application comprises the following steps: installing a laser radar and an RGB-D visual sensor on a trolley, establishing a corresponding simulation trolley model, establishing a perception module and a decision module of a mobile robot, establishing a multi-stage multi-modal information fusion framework and a corresponding neural network model, and adopting a near-end strategy optimization algorithm to calculate an equivalent advantage function according to a reward. The application can improve the environmental perception ability of the mobile robot, so that the environmental information can be better acquired and utilized, the long-term advantages of the deep reinforcement learning decision ability are combined, the autonomous learning ability of the robot is helped, and the robot can be navigated without a map.

Description

Technical Field

[0001] This invention relates to the field of intelligent robot navigation, and in particular to an indoor navigation method based on the fusion of visual and radar information and reinforcement learning. Background Technology

[0002] For mobile robots across various fields, autonomous navigation is a fundamental and crucial capability. The basic goal of navigation is to plan a path while autonomously avoiding obstacles, based on collected environmental information and under certain constraints, such as finding the shortest path, minimizing mechanical movement, and avoiding collisions. In some existing scenarios, especially in simple environments with only static obstacles, traditional navigation algorithms have achieved good results. However, due to their high dependence on the accuracy of the environmental model, their navigation performance is less effective in complex, dynamically changing, or even unknown environments.

[0003] Traditional laser SLAM navigation performs poorly in indoor environments with numerous irregular obstacles (such as office chairs), failing to successfully avoid them. Vision-based end-to-end navigation algorithms suffer from slow convergence due to the high dimensionality of image information, resulting in poor performance. LiDAR point clouds can actively perceive objects over a large horizontal range and are less affected by lighting conditions, but have a limited vertical field of view. Depth images, on the other hand, have a larger vertical field of view but a smaller horizontal field of view, and are two-dimensional, resulting in higher dimensionality than LiDAR data. RGB images provide texture and other visual information that can be used for higher-precision visual tasks, but are significantly affected by lighting conditions. Furthermore, since the framework for fusing multi-sensor information is not yet established, directly introducing visual images increases the dimensionality of perceived information, leading to poorer environmental awareness.

[0004] For example, a Chinese patent document, "An Indoor Mapping and Navigation System for a Warehouse Robot Based on LiDAR," with publication number CN113093761A, discloses a warehouse robot that automatically scans the surrounding environment using LiDAR to obtain a set of laser data points. When constructing a two-dimensional grid map based on the laser data point set using an approximation simultaneous localization and mapping algorithm, it uses least squares or singular value decomposition to estimate the pose of the laser data point set. Based on the pose estimation results, it constructs a two-dimensional grid map and updates it to the warehouse system in real time. It also uses the Dijkstra algorithm to determine the optimal path based on the two-dimensional grid map and performs positioning and navigation according to the optimal path. The warehouse system receives and stores the two-dimensional grid map and sends command information to the warehouse robot. A mobile control terminal sends command information to the warehouse robot. Since the two-dimensional grid map uses the same precision to represent objects at different distances during positioning, it is necessary to compensate for the difference in precision. However, this solution does not perform corresponding compensation operations, which easily leads to navigation or positioning errors. Therefore, this solution has certain shortcomings. Summary of the Invention

[0005] This invention primarily addresses the problem of poor environmental perception in existing robot autonomous navigation technologies, leading to low navigation success rates in unknown environments. It provides an indoor navigation method based on the fusion of visual and radar information and reinforcement learning, which can improve the mobile robot's environmental perception, thereby better acquiring and utilizing environmental information. By combining the advantages of deep reinforcement learning decision-making capabilities, it helps the robot acquire autonomous learning capabilities, enabling navigation even without a map.

[0006] The above-mentioned technical problems of the present invention are mainly solved by the following technical solutions:

[0007] This invention includes: installing a LiDAR and an RGB-D vision sensor on a small vehicle, both mounted at the front of the vehicle, and establishing a corresponding simulated vehicle model using URDF files; establishing a perception module and a decision module for the mobile robot, respectively, establishing a multi-stage multimodal information fusion framework and a corresponding neural network model, employing a proximal policy optimization algorithm to calculate an equivalent advantage function based on rewards, thereby adjusting the probability of choosing a behavior; equipping the trained perception module and the decision module to be trained, and placing the simulated vehicle in different simulation environments for training; applying navigation commands to the robot in a real scene and obtaining target point vectors, the robot acquiring basic navigation information and generating environmental feature vectors, combining the target point vectors and environmental feature vectors with the input to the decision module, and the decision module controlling the robot to complete obstacle avoidance navigation.

[0008] Preferably, the neural network model includes six sub-networks: an RGB image encoding sub-network, an image semantic segmentation sub-network, a LiDAR data encoding sub-network, a prior knowledge utilization module, a target point information processing module, and a decoding sub-network containing two fully connected layers and a long short-term memory network. The neural network model is trained using an object set collected by a simulated vehicle model to obtain a trained perception module. In this step, the RGB visual information is binarized and semantically segmented to extract more abstract intermediate features, thereby reducing the impact of the difference between the virtual and real environments on the navigation process.

[0009] As a preferred approach, for environments with static obstacle distributions, an Actor-Critic network architecture is employed. The output actions utilize a continuous action space, where the Critic network predicts the state value function, and the Actor network outputs action probability distribution parameters. During training, actions are obtained by sampling historical action distributions, and during testing, the mean of the distribution is directly used as the action output. For environments with dynamic obstacle distributions, the navigation process is modeled as a partially observable Markov decision problem, and a Long Short-Term Memory (LSTM) network is introduced for modeling and solving. To solve the partially observable Markov decision problem and avoid dimensionality explosion, this step uses an LTM network to model the agent's historical observation sequence data, fitting the agent's confidence level for solution. To leverage the inference advantages of the LTM network and increase trajectory smoothness, the action sequence for the next 3 seconds is output, and the output vector is added to the state vector of the next planning process.

[0010] Preferably, the multimodal fusion framework adopts a multi-stage fusion method, which fuses the depth image with the registered color image, only considering the depth of infeasible areas as the basis for obtaining pseudo-LiDAR data; the pseudo-LiDAR data is fused with LiDAR data, and the LiDAR data within the field of view of the depth camera is corrected, finally obtaining the corrected LiDAR data for subsequent navigation.

[0011] As a preferred approach, the vehicle is treated as an intelligent agent. The agent's position and navigation target point are randomly initialized. The agent performs corresponding actions based on the acquired information. The environment, based on a reward function and sensor information, completes one interaction, and the interaction data is stored in an experience replay pool. When the data in the experience replay pool reaches a set threshold, the data is retrieved, and the state value and advantage function are calculated. The loss function of the Actor network is calculated with the aim of maximizing the advantage of the Actor network's output actions. The network parameters of the agent are updated using gradient ascent, and the new network parameters are used to output actions, continuing to interact with the environment until training ends, resulting in a trained decision module and a complete end-to-end navigation system for the mobile robot. In this step, the training process progresses from shallow to deep, gradually increasing the environmental complexity by adding dynamic obstacles and increasing road complexity. This avoids slow convergence in the early stages of training and enhances the generalization ability of the navigation algorithm.

[0012] Preferably, the robot is placed in a real-world scenario, and navigation commands to target points are applied to it. The robot's target processing module processes these commands to obtain a target point vector. Based on the basic information acquired at each moment, the robot inputs the above-mentioned information and navigation information obtained by the sensors into a trained perception module. A multimodal information fusion framework is used to perform information fusion to obtain an environmental feature vector. The basic information includes RGB image information, depth image information, and LiDAR point cloud information. The environmental features obtained in the above steps are input into a trained decision module. The PPO algorithm outputs the robot's action command for that moment based on the input feature vector, until the robot reaches the destination and completes the navigation task.

[0013] Preferably, the reward function includes a safety performance index, an efficiency index, and a trajectory optimization performance index; the safety performance index includes negative rewards for colliding with obstacles and negative rewards for the distance to obstacles, the efficiency index includes positive rewards for reaching the endpoint and positive rewards for the difference in distance between adjacent time points and the endpoint, and the trajectory optimization performance index includes negative rewards for the magnitude of acceleration and negative rewards for the difference in angular velocity direction between adjacent time points; this can effectively avoid the problem of sparse rewards and improve the training effect.

[0014] The beneficial effects of this invention are:

[0015] 1. The present invention provides an indoor navigation method based on visual and radar information fusion and reinforcement learning. By fusing RGB image data and depth image data and correcting LiDAR data within the visible range of a depth camera, a multi-stage multimodal fusion framework is obtained. This reduces the difference between the virtual and real environments in the reinforcement learning algorithm training process and the actual process, and improves the multimodal perception effect in indoor navigation scenarios with irregular obstacles.

[0016] 2. This invention provides an indoor navigation method based on visual and radar information fusion and reinforcement learning. It utilizes deep reinforcement learning to enhance decision-making capabilities, employs the PPO reinforcement learning algorithm suitable for continuous spaces, and introduces a long short-term memory network to model the sequential data of the navigation process, thereby enabling it to possess a certain inference ability and improving the robot's autonomous learning capability. 3. This invention also provides an indoor navigation method based on visual and radar information fusion and reinforcement learning. It extracts abstract features, uses semantic features to improve the environmental feature extraction effect, and gradually increases the navigation difficulty by sequentially modifying the training map, thereby enhancing the robot's generalization ability, improving the robot's navigation effect, and increasing the robot's navigation success rate in dynamic and unknown environments.

[0017] 4. The present invention provides an indoor navigation method based on the fusion of visual and radar information and reinforcement learning. It uses a long short-term memory network to process time-series data. By modeling the navigation process in an environment that may contain dynamic obstacles as a partially observable Markov decision process, it uses historical data from the same navigation process to output actions at multiple future moments and adds them to the state space during the next planning phase. This enables the robot to have a certain reasoning ability and improves trajectory smoothness. Attached Figure Description

[0018] Figure 1 This is a flowchart of an indoor navigation method based on visual and radar information fusion and reinforcement learning according to the present invention;

[0019] Figure 2 This is a framework diagram of a car navigation algorithm based on the fusion of visual and radar information and reinforcement learning, which is an indoor navigation method of the present invention. Detailed Implementation

[0020] The technical solution of the present invention will be further described in detail below through embodiments and in conjunction with the accompanying drawings.

[0021] Example:

[0022] This embodiment presents an indoor navigation method based on the fusion of visual and radar information and reinforcement learning, such as... Figure 1 As shown, it includes the following steps:

[0023] Step 1: Mount the LiDAR and RGB-D vision sensor on the automated guided vehicle (AGV) (this invention uses the Turtlebot 2 vehicle as the experimental vehicle). Both the LiDAR and RGB-D vision sensor are mounted at the front of the vehicle, with sufficient space reserved for the LiDAR's scanning (360° horizontal range) to avoid obstruction. The vision sensor is pointed in the direction of travel. This invention uses a Kinect V2 as the vision sensor. Model the vehicle and sensors using URDF files to create a robot simulation model, which will be used for subsequent training of the network model in the simulation environment. Figure 2 This is the overall navigation algorithm framework for the car.

[0024] Step 2: Complete as follows Figure 2 The perception module of the automated guided vehicle (AGV) shown is established, along with a multi-stage, multi-modal information fusion framework and a corresponding neural network model. This neural network model includes six sub-networks: an RGB image encoding sub-network, an image semantic segmentation sub-network, a LiDAR data encoding sub-network, a LiDAR data correction module, a target point information processing module, and a decoding sub-network containing three fully connected layers and a long short-term memory (LSTM) network unit. Since there are significant differences in appearance and lighting between the virtual and real environments, RGB images are greatly affected. Therefore, RGB images cannot be directly used for path planning. To mitigate the impact of these differences, a different approach is needed.

[0025] On the one hand, data augmentation is needed to facilitate subsequent feature extraction; on the other hand, it needs to be replaced with a more abstract visual intermediate feature representation—semantic segmentation image, and then fused with depth image to better filter out interference factors in depth image that may affect navigation performance, while retaining as many important features as possible for planning and control, thereby supporting generalization for different scenarios as much as possible.

[0026] Point cloud images acquired by lidar sensors can provide depth information, but the computational cost is high. Therefore, projection processing is required. In common forward-view projection and bird's-eye view projection, epipolar raster mapping based on the forward view can preserve points in the point cloud image. Therefore, the epipolar raster mapping method is used to process the point cloud image data, obtaining an omnidirectional distance array centered on the vehicle. For feature fusion of multimodal information, the binarized semantic segmentation results of the RGB image are fused with the registered depth image. Only the depth data of impassable areas is retained in the depth image, and the minimum value is taken column-wise to achieve dimensionality reduction of the depth image data.

[0027] After obtaining the processed pseudo-LiDAR data, it is fused with the LiDAR data within the spectral range of the depth image data. The minimum depth value at the same location is used for subsequent navigation, thus correcting the LiDAR data. During training, various scenarios are used, changing the lighting, obstacle shape and position, and the shape and width of feasible roads. Fine-tuning is performed on the pre-trained network by acquiring images of the current scene, completing the feature extraction of sensor information. The specific steps are described below.

[0028] A semantic segmentation network was trained on an indoor object dataset using a simulated car model. The pre-trained network was fine-tuned using images from both the virtual and real-world environments to improve segmentation accuracy in experimental settings. The indoor object dataset used was the classic MIT open-source semantic segmentation dataset ADE20k. To enable the network to extract more abstract intermediate features and mitigate the impact of environmental differences when transitioning from the virtual to the real environment, the label files in the dataset were modified to include both "passable" and "impassable" labels. To further improve semantic segmentation accuracy in relevant experimental scenarios, the pre-trained network was fine-tuned using images from both virtual and real environments. Training was conducted using supervised learning, and the semantic segmentation results show each pixel in the image with a corresponding label: 0 for impassable and 1 for passable.

[0029] The RGB image and the depth image are registered. Since the field of view of the depth camera is smaller than that of the RGB image, the registration process involves attaching the corresponding RGB data to the pixels of the depth image to obtain the registered image. Then, only the distance values ​​of the impassable areas are considered. In order to reduce the impact of depth camera measurement errors and individual erroneous pixels on the results, the minimum value is selected when the continuous length of the impassable area is greater than the minimum threshold, thereby reducing measurement errors. Then, only one distance value is retained in each column, thus realizing the dimensionality reduction of the two-dimensional depth image and obtaining pseudo-LiDAR data. The pseudo-LiDAR data is then updated with the minimum value at the corresponding position of the pseudo-LiDAR data and the corresponding lateral angle range data of the pseudo-LiDAR data, thereby correcting the pseudo-LiDAR data and obtaining the corrected LiDAR data.

[0030] Step 3: Complete as follows Figure 2 The decision-making module of the mobile robot shown is established by using deep reinforcement learning algorithm to build a corresponding neural network model. The proximal policy optimization algorithm (PPO) is adopted, which is an on-policy policy gradient algorithm suitable for high-dimensional continuous action space. The equivalent advantage function is calculated based on the reward, so as to directly adjust the probability of the selected behavior. The advantage function calculation formula is shown in Equation (1):

[0031]

[0032] PPO employs an Actor-Critic architecture, where the Critic network is responsible for estimating the state-value function to ensure greater accuracy. In this method, the output actions utilize a continuous action space, so the Actor network is responsible for outputting action distribution parameters. The goal is to maximize the advantage function value of the output actions. During training, actions are sampled from this distribution to enhance exploration. During testing, the action values ​​are directly output based on the mean of the output action distribution parameters. To improve the utilization of sampled data, the PPO algorithm uses importance sampling, adjusting the updated loss function based on the gap between the current optimized policy and the policy in the exploration environment. Preset parameters constrain this adjustment, preventing the updated policy from deviating excessively from the old policy.

[0033] Furthermore, this invention makes the following improvements to the PPO algorithm for indoor scenes containing dynamic obstacles:

[0034] Reward function reconstruction: The reward function is an important signal driving the learning strategy of an agent in reinforcement learning. It includes safety performance indicators (negative reward for colliding with obstacles, positive reward for reaching the destination, and negative reward for the distance from obstacles) and trajectory optimization performance indicators (negative reward for the magnitude of linear velocity acceleration and negative reward for the magnitude of angular velocity change).

[0035] Adding an LSTM network layer: The PPO actor and critic networks are themselves multilayer perceptron (MLP) structures, using three fully connected layers. They do not have the ability to model the long and short term of sequence data separately. This invention modifies the multilayer perceptron by replacing the middle fully connected layer with an LSTM layer. The cell state of the LSTM network is used to fit the hidden state in a partially observable Markov decision process, thereby achieving the solution.

[0036] Output multiple action distribution parameters: Traditional navigation algorithms output the action at the next moment, suitable for situations where the environment does not change much and there is no need to predict the future state. This invention outputs multiple action distribution parameters, representing the action sequence within the next 3 seconds, reflecting the prediction of the surrounding environment state within the next 3-second time window, and using the action sequence as part of the state input for the next planning moment, improving prediction performance and trajectory smoothness.

[0037] The state space consists of three parts: the corrected lidar information, the relative position of the target point and the vehicle, and the vehicle's motion. The corrected lidar information and target information are generated by the perception module, and the vehicle's motion represents the linear velocity and angular velocity sequence output by the vehicle at the previous moment.

[0038] The action space adopts a continuous action space. Although discrete actions are easier to converge, the corresponding solution space is smaller, which may lead to getting trapped in local optima and cause the vehicle trajectory to be less smooth. Considering that the reinforcement learning algorithm PPO can be applied to a continuous state space, this invention adopts a continuous action space, with the linear velocity v ranging from [0, 1.8] m / s and the angular velocity ω ranging from [-π / 3, π / 3] rad / s.

[0039] Step 4: Equip the trained perception module and the decision module to be trained, and place the simulated car in different simulation environments for training. Training navigation strategies based on reinforcement learning requires a large amount of interaction data between the agent and the environment. Considering factors such as the difficulty of environment construction, the ease of information acquisition, and the difficulty of deploying the model on a real vehicle, this invention uses Gazebo to build the simulation environment, which can directly load the URDF file obtained from modeling the car in Step 1. To avoid problems such as slow network convergence speed and low exploration success rate leading to a slow training process due to an overly complex environment in the early stages of training, a design approach of gradually increasing environmental complexity is adopted. Starting with static obstacles in the simplest 20*20 environment, the environment area and complexity are gradually increased. Dynamic obstacles and complex scenarios such as long straight corridors, which are difficult for traditional algorithms to handle, are added using a domain random data generation method, gradually completing the training of the reinforcement learning algorithm model for the decision module.

[0040] The simulated car is treated as an intelligent agent. The agent's position and navigation target point are randomly initialized. An initial experience replay pool of 4000 is used to store interaction data for updating reinforcement learning model parameters. The experience replay pool can store information about the interaction process between the agent and the environment, improving sample utilization efficiency and avoiding excessive hardware performance requirements for online interaction. Furthermore, the replay pool can store and retrieve complete interaction data, thus obtaining complete interaction sequences for training a PPO network containing LSTM layers. The PPO algorithm divides the same batch of samples in the experience pool into multiple mini-batches and reuses them multiple times. Simultaneously, it limits the parameter update magnitude when calculating the policy gradient, thereby avoiding training instability. In this invention, interaction data from the same complete navigation process within a batch of samples are divided into the same mini-batch, thus maintaining the temporal correlation of the corresponding data.

[0041] The specific training process is as follows: First, initialize the Critic network Q. θ and Actor Network The corresponding parameters are θ. Initialize the experience replay pool. Concatenate the environmental feature vector, target point information vector, and the linear and angular velocity vectors planned by the vehicle in the previous moment from the perception module, and use this as the state input s. tAfter passing through the Actor network, the action distribution parameters of the agent are obtained. In this invention, a normal distribution is used as the basis for agent action sampling. Therefore, the Actor network obtains the mean μ and variance σ of the action distribution. Sampling the above distribution yields the action command vector, denoted as a. t In other words, linear velocity and angular velocity. This invention outputs six action distributions, thus obtaining a sequence of actions within the next 3 seconds arranged in time sequence. The agent then performs its first action according to the action command and obtains the corresponding reward r from the environment. t With the new sensor information, the sensing module can obtain the state input s for the next moment. t+1 This completes an interaction with the environment, sending the interaction data and the end marker (s) to the environment. t a t r t s t+1 Once the data in the experience replay pool is reached (or a collision occurs with an obstacle), the interaction is stored in the pool. This process is repeated until the target point is reached or a collision with an obstacle occurs. At this point, the current round of interaction ends, and the user returns to the starting point to begin the next round. After the experience replay pool has accumulated a certain amount of data, the data is retrieved to update the agent's network parameters. All interaction data from eight complete navigation sequences are randomly selected, and then the network is updated.

[0042] The Actor network is updated so that the output action advantage is maximized. After extracting the interaction data, the state value of each time step is first calculated using the Critic network, and then the advantage function of each state is calculated using equation (1). Using equation (2) as the loss function, the gradient ascent method is used to update the multi-step gradient.

[0043]

[0044] The Critic network is updated using the multi-step TD method, employing equation (3). Starting from the current state, the reward returned by the environment at each step is multiplied by the discount factor and then summed to obtain the discounted reward G. t Using G t The current state value v(s) predicted by the Critic network t The mean squared loss function between the two sides is used as the loss function for gradient descent training.

[0045] G t =r t+1 +γr t+2 +…+γ T-t r T+1 +γ T+1-t v(s T+1 (3)

[0046] Where γ represents the reward discount parameter, v(s) T+1 ) represents the state value at time T+1 output by the Critic network.

[0047] The cache pool data is then cleared, and actions are output using new network parameters to continue interacting with the environment until training is complete. Once training is finished, the trained decision-making module is obtained, resulting in a complete end-to-end navigation system for the mobile robot.

[0048] Step 5: Place the robot in a real-world environment, which may differ from the virtual environment, to test the generalization performance of the navigation method. Apply a navigation command to the robot for the target point, i.e., the target position relative to the robot's position vector. The robot's target processing module then processes the command vector to obtain the target point vector (x...). t y t );

[0049] Step 6: At each moment, the robot acquires RGB-D image information and LiDAR point cloud information from the installed vision sensor and LiDAR sensor, and inputs them and navigation target point information into the trained perception module. The multimodal information fusion framework obtained in Step 2 is used to perform information fusion and obtain environmental feature vectors.

[0050] Step 7: After concatenating the environmental feature vector obtained in Step 6 with the current linear velocity and angular velocity vector of the vehicle, input the concatenation into the trained decision module. The PPO algorithm will output the robot's action command at that moment based on the input feature vector until the robot reaches the destination and completes the navigation task.

[0051] It should be understood that the embodiments are for illustrative purposes only and are not intended to limit the scope of the invention. Furthermore, it should be understood that after reading the teachings of this invention, those skilled in the art can make various alterations or modifications to the invention, and these equivalent forms also fall within the scope defined by the appended claims.

Claims

1. An indoor navigation method based on visual and radar information fusion and reinforcement learning, characterized in that, include: S1: Install a LiDAR and an RGB-D vision sensor on the car. Both the LiDAR and the vision sensor are installed at the front of the car. Use URDF files to create a corresponding simulation car model. S2: Establish the perception module and decision-making module of the mobile robot respectively. Develop a multi-stage, multi-modal information fusion framework and a neural network model including an RGB image encoding subnetwork, an image semantic segmentation subnetwork, a LiDAR data encoding subnetwork, a prior knowledge utilization module, a target point information processing module, and a decoding subnetwork containing two fully connected layers and a long short-term memory network. The fusion framework fuses the depth image with the registered color image, considering only the depth of infeasible regions as the basis for obtaining pseudo-LiDAR data. This pseudo-LiDAR data is then fused with the LiDAR data. The LiDAR data within the depth camera's field of view is corrected to obtain corrected LiDAR data. A near-end strategy optimization algorithm is used to calculate an equivalent advantage function based on the reward, thereby adjusting the probability of the chosen action. S3: Equipped with a pre-trained perception module and a decision module to be trained, the simulation car is placed in different simulation environments for training; S4: In a real-world scenario, navigation commands are applied to the robot and the target point vector is obtained. The robot acquires basic navigation information and generates an environmental feature vector. The target point vector and the environmental feature vector are then input into the decision module, which controls the robot to complete obstacle avoidance navigation.

2. The indoor navigation method based on visual and radar information fusion and reinforcement learning according to claim 1, characterized in that, Step S2 also includes S2.1: training the neural network model using the set of items collected by the simulated car model to obtain a trained perception module.

3. The indoor navigation method based on visual and radar information fusion and reinforcement learning according to claim 1, characterized in that, S2.2: For environments with static obstacle distribution, an Actor-Critic network architecture is adopted, and the output action adopts a continuous action space. The Critic network predicts the state value function, and the Actor network outputs the action probability distribution parameters. During training, the action is obtained by sampling the historical action distribution, and during testing, the mean of the distribution is directly used as the action output. For environments with dynamic obstacle distribution, the navigation process is modeled as a partially observable Markov decision problem, and a Long Short-Term Memory network is introduced for modeling and solving.

4. The indoor navigation method based on visual and radar information fusion and reinforcement learning according to claim 1, characterized in that, The pseudo-LiDAR data is fused with LiDAR data, and the LiDAR data within the field of view of the depth camera is corrected. The final corrected LiDAR data is then used for subsequent navigation.

5. An indoor navigation method based on visual and radar information fusion and reinforcement learning according to claim 1, 2, 3, or 4, characterized in that, S3.1: Treat the vehicle as an intelligent agent, randomly initialize the agent's position and navigation target point, and the agent performs corresponding actions based on the acquired information. The environment completes an interaction based on the reward function and sensor information, and the interaction data is stored in the experience replay pool. S3.2: When the data in the experience replay pool reaches a set threshold, the data is retrieved and the state value and advantage function are calculated. The loss function of the Actor network is calculated with the aim of maximizing the advantage of the Actor network's output action. The network parameters of the intelligent agent are updated using the gradient ascent method, and the new network parameters are used to output actions. The interaction with the environment continues until the end of training, resulting in a trained decision module, thus obtaining a complete end-to-end navigation system for the mobile robot.

6. The indoor navigation method based on visual and radar information fusion and reinforcement learning according to claim 5, characterized in that, S4.1: Place the robot in a real-world scene and apply navigation commands to the target point. The robot's target processing module processes the commands to obtain the target point vector. S4.2: Based on the basic information acquired at each moment, the robot inputs the basic information obtained from the sensors and the target point vector into the trained perception module. A multimodal information fusion framework is used to perform information fusion to obtain environmental feature vectors. The basic information includes RGB image information, depth image information, and LiDAR point cloud information. S4.3: Input the environmental features obtained in step S4.2 into the trained decision module. The PPO algorithm outputs the robot's action command at that moment based on the input feature vector until the robot reaches the destination and completes the navigation task.

7. The indoor navigation method based on visual and radar information fusion and reinforcement learning according to claim 5, characterized in that, The reward function includes safety performance indicators, efficiency indicators, and trajectory optimization performance indicators; the safety performance indicators include negative rewards for colliding with obstacles and negative rewards for the distance to obstacles, the efficiency indicators include positive rewards for reaching the destination and positive rewards for the difference in distance between adjacent time points and the destination, and the trajectory optimization performance indicators include negative rewards for the magnitude of acceleration and negative rewards for the difference in angular velocity direction between adjacent time points.

Images