A method, system, device and medium for autonomous navigation of a mobile scanning robot.

Abstract

This invention relates to the field of robot navigation technology, and discloses a method, system, device, and medium for autonomous navigation of a mobile scanning robot. The method is applied to a mobile robot and includes: acquiring the polar coordinate pose of the mobile robot in a target environment, and a LiDAR point cloud obtained by the mobile robot scanning the target environment; extracting features from the polar coordinate pose and the LiDAR point cloud to obtain pose features containing the spatial geometric relationship between the mobile robot and the navigation target in the target environment, and environmental features containing the distribution of obstacles in the target environment; fusing the pose features and environmental features to obtain fused features; and determining the navigation strategy of the mobile robot based on the PPO algorithm using the fused features for autonomous navigation. The pruning range of the PPO algorithm is dynamically adjusted according to the KL divergence between the old and new navigation strategies to control the magnitude of the PPO algorithm's navigation strategy update, thus achieving autonomous navigation in a dynamically unknown environment.

Description

Technical Field

[0001] This invention relates to the field of robot navigation technology, and in particular to a method, system, device and medium for autonomous navigation of a mobile scanning robot. Background Technology

[0002] Autonomous Mobile Robots (AMRs), with their environmental perception, autonomous decision-making, and navigation capabilities, have become an indispensable part of modern society and industrial systems. Among these capabilities, autonomous navigation is the core of mobile scanning robots' task execution, requiring them to independently complete environmental perception, path planning, and motion control without human intervention.

[0003] Current robot navigation methods heavily rely on high-precision pre-built maps and assume that the environment remains static during task execution, making them ill-suited for navigation tasks in complex or unknown environments. However, in practical applications such as warehousing and logistics, post-disaster building structure scanning, or service robots, the environment is often dynamically changing and some information is unknown, resulting in static map-based navigation methods performing poorly in various aspects, such as generalization ability, real-time performance, and safety. Summary of the Invention

[0004] The purpose of this invention is to provide an autonomous navigation method, system, device, and medium for mobile scanning robots, which can solve the problems of insufficient generalization ability, real-time performance, and safety of existing robot navigation methods when facing dynamic and unknown environments.

[0005] To address the aforementioned technical problems, embodiments of the present invention provide an autonomous navigation method for a mobile scanning robot, applicable to a mobile robot, comprising the following steps:

[0006] The polar coordinate pose of the mobile robot in the target environment and the LiDAR point cloud obtained by the mobile robot scanning the target environment are obtained.

[0007] Feature extraction is performed on polar coordinate pose and lidar point cloud to obtain pose features containing the spatial geometric relationship between the mobile robot and the navigation target in the target environment, as well as environmental features containing the distribution of obstacles in the target environment.

[0008] The pose features and environmental features are fused to obtain the fused features;

[0009] Based on the Proximal Policy Optimization (PPO) algorithm, a navigation strategy for a mobile robot is determined by fusing features to enable autonomous navigation. The pruning range of the PPO algorithm is dynamically adjusted according to the KL divergence between the old and new navigation strategies to control the magnitude of the navigation strategy update.

[0010] Optionally, the fusion of pose features and environmental features to obtain fused features includes:

[0011] Using pose features as the multi-head self-attention mechanism as the Query and environmental features as the Key / Value, the feature correlation between pose features and environmental features is obtained to filter out environmental regions in the target environment that are related to the current navigation intent.

[0012] The environmental features of the selected environmental regions are fused and then spliced ​​with the environmental features of the target environment to obtain the fused features.

[0013] Optionally, when the KL divergence of the new and old navigation strategies in the PPO algorithm exceeds 1.5 times the preset threshold, the minimum clipping range is adopted; when it is less than 1.5 times the preset threshold, the maximum clipping range is adopted.

[0014] Optionally, the clipping range of the PPO algorithm is dynamically adjusted based on the KL divergence between the old and new navigation strategies, and then processed using the exponential moving average (EMA) method.

[0015] Optionally, the reward function of the PPO algorithm includes: a first reward function that rewards based on whether the mobile robot reaches the area where the navigation target is located; a second reward function that rewards based on the distance between the mobile robot and the navigation target; a third reward function that rewards based on the change in distance of the mobile robot relative to the next waypoint within consecutive time steps; and a fourth reward function that rewards based on whether the mobile robot collides with an obstacle.

[0016] Optionally, the step of extracting features from the polar coordinate pose and the lidar point cloud to obtain pose features containing the spatial geometric relationship between the mobile robot and the navigation target in the target environment, and environmental features containing the distribution of obstacles in the target environment, includes:

[0017] The 2D polar coordinate pose is mapped to 128 dimensions through the first layer MLP in the two-layer multilayer perceptron MLP. After regularization, it is extended to 256 dimensions through the second layer MLP to obtain pose features that include the spatial geometric relationship between the mobile robot and the navigation target in the target environment.

[0018] The 64-dimensional LiDAR point cloud is embedded into a 128-dimensional space through a projection layer. After modeling the long-range dependency of the LiDAR point cloud through a multi-head self-attention mechanism, the environmental features including the distribution of obstacles in the target environment are obtained by combining the feedforward network processing.

[0019] Optionally, the Proximity Policy Optimization (PPO) algorithm determines the navigation strategy of the mobile robot by fusing features for autonomous navigation, including:

[0020] Based on the PPO algorithm, the angular velocity and linear velocity of the mobile robot are determined by fusing features, and autonomous navigation is performed based on the angular velocity and linear velocity of the mobile robot.

[0021] Embodiments of the present invention also provide an autonomous navigation system for a mobile scanning robot, comprising:

[0022] The data acquisition module is used to acquire the polar coordinate pose of the mobile robot in the target environment, as well as the LiDAR point cloud obtained by the mobile robot scanning the target environment.

[0023] The feature extraction module is used to extract features from the polar coordinate pose and the lidar point cloud respectively, so as to obtain pose features containing the spatial geometric relationship between the mobile robot and the navigation target in the target environment, and environmental features containing the distribution of obstacles in the target environment.

[0024] The feature fusion module is used to fuse pose features and environmental features to obtain fused features;

[0025] The autonomous navigation module is used to determine the navigation strategy of the mobile robot based on the Proximal Policy Optimization (PPO) algorithm by fusing features for autonomous navigation. The pruning range of the PPO algorithm is dynamically adjusted according to the KL divergence between the old and new navigation strategies to control the magnitude of the navigation strategy update of the PPO algorithm.

[0026] Embodiments of the present invention also provide a computer device, including: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the above-described autonomous navigation method for a mobile scanning robot.

[0027] Embodiments of the present invention also provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the above-described autonomous navigation method for a mobile scanning robot.

[0028] The autonomous navigation method for mobile scanning robots provided by this invention has at least the following beneficial effects:

[0029] In this invention, the mobile robot uses the PPO algorithm to learn navigation strategies from raw low-dimensional perception data (including the polar coordinate pose of the mobile robot in the target environment and the lidar point cloud obtained by the mobile robot scanning the target environment). It does not rely on a pre-built accurate environment map, which significantly improves the adaptability of the mobile robot in unknown or dynamic environments.

[0030] The dataset it relies on includes environmental features of obstacle distribution in the target environment and pose features of spatial geometric relationship between the mobile robot and navigation target in the target environment, which enhances environmental perception and provides "high-quality state input" for the PPO algorithm. This enables the policy network of the PPO algorithm to accurately understand the pose information of the mobile robot, the surrounding obstacle information and the target orientation information, and provide a reliable environmental semantic basis for subsequent decision-making.

[0031] Meanwhile, when searching for the optimal navigation strategy, the PPO algorithm dynamically adjusts the pruning range based on the KL divergence between the old and new navigation strategies. This controls the magnitude of the navigation strategy update by the PPO algorithm, solves the problem of strategy update imbalance, and allows the navigation strategy based on accurate environmental perception to both expand the exploration range for rapid adaptation when obstacles appear and tighten constraints for robust convergence when the environment is stable, effectively balancing the stability of strategy exploration and update. Attached Figure Description

[0032] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this invention, illustrate exemplary embodiments of the invention and are used to explain the invention, but do not constitute an undue limitation of the invention. In the drawings:

[0033] Figure 1 A flowchart illustrating an autonomous navigation method for a mobile scanning robot provided by the present invention;

[0034] Figure 2 A schematic diagram of the framework for an autonomous navigation method for a mobile scanning robot provided by the present invention;

[0035] Figure 3 This is a schematic diagram of the structure of a multimodal self-attention feature extraction network provided by the present invention;

[0036] Figure 4 A schematic diagram of a Markov decision process provided by the present invention;

[0037] Figure 5 A schematic diagram of an experimental environment provided for this invention;

[0038] Figure 6 A schematic diagram of static environment test results provided by the present invention;

[0039] Figure 7 This is a schematic diagram of the results of a dynamic environment experiment provided by the present invention. Detailed Implementation

[0040] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this invention, and not all of them. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0041] The technical solutions provided by the various embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0042] One embodiment of the present invention relates to an autonomous navigation method for a mobile scanning robot, applied to a mobile robot. The specific process of the autonomous navigation method for the mobile scanning robot in this embodiment can be as follows: Figure 1 As shown, it includes:

[0043] Step 101: Obtain the polar coordinate pose of the mobile robot in the target environment, and the LiDAR point cloud obtained by the mobile robot scanning the target environment.

[0044] Step 102: Perform feature extraction on the polar coordinate pose and the lidar point cloud to obtain pose features that include the spatial geometric relationship between the mobile robot and the navigation target in the target environment, as well as environmental features that include the distribution of obstacles in the target environment.

[0045] Step 103: Fuse the pose features and environmental features to obtain the fused features.

[0046] Step 104: Based on the Proximal Policy Optimization (PPO) algorithm, the navigation strategy of the mobile robot is determined by fusing features to enable autonomous navigation; wherein, the pruning range of the PPO algorithm is dynamically adjusted according to the KL divergence of the new and old navigation strategies to control the magnitude of the navigation strategy update of the PPO algorithm.

[0047] The following is based on Figure 2 The framework shown is used to specifically explain the implementation details of the autonomous navigation method for the mobile scanning robot in this embodiment. The following implementation details are provided for ease of understanding and are not necessary for implementing this solution.

[0048] In step 101, the mobile robot receives input LiDAR point cloud data (1×64 dimensions, 360° obstacle distance) and polar coordinate pose information (ρ, θ, 2 dimensions). The polar coordinate pose information is used to indicate the relative position of the robot and the target position, where ρ represents the distance between the two and θ represents the azimuth angle of the target position relative to the robot. Then, dynamic Z-score standardization is performed on both types of data. By maintaining the mean and standard deviation of the data in real time, the data is converted into a standard normal distribution with a mean of 0 and a standard deviation of 1 to ensure that the feature scales of the two types of data are consistent in the subsequent processing.

[0049] In steps 102 and 103, firstly, pose features containing the spatial geometric relationship between the mobile robot and navigation targets in the target environment are extracted using a two-layer multilayer perceptron (MLP) and regularization processing. Secondly, environmental features containing obstacle distribution in the target environment are extracted using a projection layer, a multi-head self-attention mechanism, and a feedforward network. Then, using pose features as the query for the multi-head self-attention mechanism and environmental features as the key / value pair, the feature correlation between pose features and environmental features is obtained to filter out environmental regions in the target environment related to the current navigation intent. The environmental features of the filtered environmental regions are then fused and concatenated with the environmental features of the target environment to obtain the fused features.

[0050] In its specific implementation, this embodiment designs a method as follows: Figure 3 The multimodal self-attention feature extraction network shown fuses LiDAR point cloud and polar coordinate pose information to generate robust environmental feature representations, including a polar coordinate branch, a LiDAR branch, and a cross-attention fusion module. A specific improvement to this network is the introduction of a Transformer-based cross-modal fusion mechanism, replacing the simple feature concatenation or single-modal processing methods in traditional reinforcement learning. Traditional methods, when processing multi-source perception data, often neglect the spatial topological relationships of LiDAR point clouds and the geometric correlation with robot pose, leading to insufficient environmental feature modeling, especially in dynamic obstacle scenarios where crucial information is easily lost.

[0051] To address this issue, this embodiment employs a dual-branch + cross-attention fusion structure: the polar coordinate branch captures the relative positional features of the robot and the target through a two-layer MLP, the LiDAR branch models the spatial dependencies of 360° obstacles through a multi-head self-attention mechanism, and finally, cross-attention achieves dynamic alignment of pose and environmental information. The core idea behind this design is to preserve the unique attributes of multimodal data while strengthening cross-modal associations through an attention mechanism, enabling the model to more accurately understand the environmental structure.

[0052] The polar coordinate branch is a feature extraction subnetwork that processes robot pose information. The input is 2D polar coordinates (ρ, θ), which are processed by a two-layer MLP (2→128→256) and regularization (layer normalization, Dropout). The output is pose features containing spatial geometric relationships. Specifically: the first layer maps the 2D pose data to 128 dimensions, followed by layer normalization, Dropout (probability 0.1), and ReLU activation; the second layer expands to 256 dimensions, outputting pose features. This embodiment uses an MLP structure because pose information is low-dimensional structured data, and the multilayer perceptron can efficiently capture its distance and angular relationships with the target point, providing geometric priors for environmental interaction. The LiDAR branch is a feature extraction subnetwork that processes point cloud data. The input is 1×64-dimensional LiDAR distance data (uniformly sampled at 360°). The 64-dimensional point cloud is embedded into a 128-dimensional space (64→128) through a projection layer. A multi-head self-attention mechanism (8 heads) is used to model the long-range dependencies of the point cloud. After processing by a feedforward network (128→256→256), the output is 256-dimensional environmental features (128→256), which includes environmental features related to obstacle distribution. The self-attention mechanism is used because LiDAR point clouds have a ring-shaped spatial distribution. Self-attention can model the long-range dependencies between any two points, effectively capturing the global layout of obstacles and avoiding the limitation of traditional convolution focusing only on local information.

[0053] The cross-attention fusion module is a core component for achieving multimodal feature interaction. Using the polar coordinate branch output as the query and the LiDAR branch output as the key / value, it calculates feature correlation through 8-head attention, concatenates the fusion result with the original features from both branches, and compresses it into a 512-dimensional feature, enhancing the semantic representation of the environment. This module allows the robot's pose information to actively "query" key areas in the environment related to navigation (such as obstacles in the target direction), dynamically assigning feature weights and improving feature discriminativeness in complex scenes. The 512-dimensional fused feature output by this module directly serves as the input state for the subsequent PPO policy network (Actor network) and value network (Critic network). Compared to the raw LiDAR data or simply concatenated pose information used in traditional PPO, this fused feature strengthens the correlation between obstacle distribution and the spatial relationship between the robot and the target through a self-attention mechanism. This upgrades the decision-making basis of the policy network from fragmented data to structured environmental cognition, providing the agent with a more comprehensive and easily understandable environmental state.

[0054] In step 104, this embodiment proposes an improved reinforcement learning navigation framework based on the Proximal Policy Optimization (PPO) algorithm. The classic PPO algorithm, by introducing a pruning objective function, achieves a good balance between training stability and policy performance and has been widely applied to policy optimization problems in complex action spaces. However, the fixed pruning threshold in PPO is difficult to adapt to changes in the policy update rate under dynamic environments: it may limit exploration capabilities in the early stages of policy convergence, while easily causing policy oscillations in the later stages. Therefore, this embodiment proposes a dynamic pruning range adaptive policy based on a Kullback-Leibler (KL) divergence feedback mechanism. This mechanism uses KL divergence as a measure of policy update differences and designs a dual closed-loop feedback control mechanism to adaptively adjust the pruning parameters to achieve fine control over the policy update amplitude. This aims to ensure policy update stability while improving exploration efficiency and training speed, thereby enhancing the overall robustness of policy learning.

[0055] Specifically, the classic PPO algorithm effectively balances the stability and sample efficiency of policy updates by introducing importance sampling and pruning mechanisms. However, its fixed pruning range design has inherent flaws when facing complex dynamic environments: on the one hand, in the early stages of training when the policy parameters have not yet converged, overly strong pruning constraints may inhibit the agent's exploration behavior; on the other hand, when the approximate KL divergence caused by policy updates deviates significantly from the target threshold, the fixed pruning range cannot dynamically adjust the constraint strength, which may lead to policy update instability. To solve this problem, this embodiment designs a Dynamic Clip Range Adaptation (DCRA) mechanism, which achieves adaptive optimization of pruning parameters through a closed-loop feedback control system.

[0056] A dynamic pruning range adaptive mechanism is used to optimize the policy update process of the PPO algorithm, achieving a balance between exploration and robustness. This mechanism includes a KL divergence estimation module, a dual-loop feedback controller, and an Exponential Moving Average (EMA) smoothing module. The improvement lies in replacing the fixed pruning threshold of PPO with a dynamic pruning range: by constructing a KL divergence-driven adaptive adjustment mechanism, using KL divergence to measure the difference between the old and new policies, designing a piecewise linear controller to dynamically adjust the pruning range, and using EMA smoothing to reduce volatility. The core idea of ​​this design is to dynamically correlate the policy update magnitude with the environment, ensuring monotonic policy improvement while flexibly adapting to exploration needs.

[0057] The core of dynamic clipping range adjustment lies in establishing a nonlinear mapping relationship between KL divergence and clipping range. The KL divergence estimates before and after the policy update are defined as follows:

[0058] ;

[0059] In the formula, 𝑁 represents the batch size of the experience playback. and These represent the strategies before and after the update, respectively. 𝑜ld and represent the policy parameters before and after the update, respectively. a i and s i Let represent the action and state in the i-th round, respectively. This estimator reflects the distributional shift in the policy update by calculating the mean of the log probability difference between the old and new policies. To avoid estimation bias, the actual implementation must ensure the independent and identically distributed nature of the sampled data and use a sufficiently large batch size.

[0060] Then, based on the KL divergence estimate, a piecewise linear controller is designed to generate the target clipping range. :

[0061] ;

[0062] In the formula, and These represent the minimum and maximum values ​​of the clipping range, respectively. For the preset target threshold, and Let represent the maximum and minimum values ​​of the KL divergence, respectively. When the KL divergence significantly exceeds the safety threshold, the controller outputs the minimum clipping range to strengthen the constraint and suppress the risk of policy mutation. When the KL divergence is too low, the maximum clipping range is used to expand the policy search space and promote exploration. In the intermediate region, linear interpolation is used to achieve a smooth transition and avoid training oscillations caused by parameter jumps. It is worth noting that the linear interpolation weights make the adjustment rate of the clipping range in the intermediate region positively correlated with the KL divergence deviation, forming an adaptive gradient adjustment characteristic.

[0063] The theoretical basis of this control mechanism originates from the trust region theory of policy optimization. KL divergence essentially measures the difference between the distributions of the old and new policies, while the pruning range corresponds to the physical meaning of the trust region radius. By dynamically adjusting the trust region size, the algorithm can adaptively adjust the step size of policy updates while ensuring monotonic improvement. The KL divergence estimation module is the unit for measuring policy differences; it quantifies the drasticness of policy updates by calculating the mean log probability difference between the old and new policies on the same state-action pairs. KL divergence is used because it effectively characterizes the difference between two probability distributions, providing an interpretable quantitative basis for policy adjustment and avoiding the limitations of subjectively setting thresholds.

[0064] The dual-loop feedback controller is the decision-making unit for the clipping range. It generates the target clipping range based on a preset KL target threshold (i.e., a preset threshold, such as 0.01): when the KL divergence exceeds 1.5 times the threshold, it outputs the minimum clipping range (e.g., 0.1) to suppress aggressive updates; when the KL divergence is less than 0.5 times the threshold, it outputs the maximum clipping range (e.g., 0.3) to encourage exploration; the intermediate intervals are smoothly transitioned through linear interpolation. This piecewise control can finely adjust the constraint strength according to the policy convergence state, balancing stability and exploration efficiency.

[0065] After the clipping range of the PPO algorithm is dynamically adjusted based on the KL divergence between the old and new policies, it is further processed using the EMA method. The EMA smoothing module is the stabilizing unit for the clipping parameters. It filters the target clipping range using an exponential moving average and then constrains the result to the interval [0.1, 0.3]. The purpose of introducing EMA is to eliminate noise interference in KL divergence estimation and avoid policy oscillations caused by frequent fluctuations in the clipping range, which can significantly improve training stability, especially in dynamic obstacle-dense scenarios.

[0066] The processing procedure is as follows:

[0067] ;

[0068] In the formula, The smoothing factor is the EMA. For the target cropping range, The clipping range over time t-1.

[0069] Subsequently, ensured by projection Stay within the predefined boundaries:

[0070] ;

[0071] In the formula, It is determined by the inner loop controller based on the current approximate KL divergence. The calculated target cropping range, β ∈(0,1) is the EMA coefficient, which controls the degree of smoothing. , [] indicates the allowed range of the cropping area.

[0072] The EMA mechanism acts as a low-pass filter, reducing the impact of noise or... Caused by instantaneous fluctuations Rapid changes. This stabilization is particularly beneficial in stochastic environments or near convergence, preventing sudden adjustments from disrupting training stability. By adjusting... β It is possible to Response speed and smoothness of change A balance must be struck between the degree of update: a smaller φ enhances stability but slows down adaptation, while a larger φ allows for faster responses but may increase volatility. This approach achieves this by maintaining a balance between the size of the trust region (i.e., the pruning range) and the degree of update: a smaller φ enhances stability but slows down adaptation, while a larger φ allows for faster responses but may increase volatility. Consistent and controlled adjustments ensure robust strategy optimization.

[0073] The improved PPO algorithm in this embodiment achieves end-to-end learning of the navigation strategy based on the above mechanisms. It includes an Actor network, a Critic network, and a joint loss function. Combined with the aforementioned feature extraction and dynamic pruning mechanisms, it completes parameter updates (strategy optimization). The algorithm's Markov decision process is as follows: Figure 4 As shown. The Actor network is the action decision sub-network. Its input is a 512-dimensional fused feature, and its output is continuous actions (linear velocity and angular velocity). It uses a Gaussian distribution to model action probabilities and optimizes parameters through a dynamically pruned PPO objective function. Its function is to generate actions that meet navigation requirements based on environmental features, with linear velocity ranging from [-1, 1] m / s and angular velocity ranging from [-1, 1] rad / s, adapting to robot motion constraints. The Critic network is the value assessment sub-network. Its input is the same as the Actor network, and its output is a state value estimate, optimized using mean squared error (MSE) loss. It calculates the advantage function through generalized advantage estimation (GAE), balancing bias and variance to provide a reliable gradient signal for policy updates. The joint loss function consists of shearing loss, value loss, and entropy loss. Introducing entropy loss can prevent the policy from converging prematurely to a local optimum, maintaining adaptability to new obstacles in dynamic environments and improving the robustness of the navigation strategy. The algorithm in this embodiment constructs a pruning objective function based on pruning parameters, combines it with the advantage function calculated by generalized advantage estimation (GAE), and uses the Adam optimizer to jointly optimize the policy network and the value network, balancing exploration efficiency and training robustness.

[0074] The algorithm achieves closed-loop iteration of the policy through a designed navigation execution and feedback module, which includes an action execution unit, a reward calculation unit, and a task termination judgment unit, supporting real-time interaction between the robot and the environment. An improvement to this module is the design of a multi-dimensional reward function to replace the traditional sparse reward (feedback only upon collision or reaching the target). Traditional reward mechanisms tend to lead to blind exploration by the agent and slow training convergence. This invention provides dense feedback for each action by combining target rewards (a first reward function based on whether the mobile robot reaches the navigation target area), distance rewards (a second reward function based on the distance between the mobile robot and the navigation target), forward rewards (a third reward function based on the change in distance of the mobile robot relative to the next waypoint within consecutive time steps), and obstacle rewards (a fourth reward function based on whether the mobile robot collides with an obstacle), thereby accelerating policy learning.

[0075] The motion execution unit is responsible for converting the motion commands output by the Actor network into robot control signals. It then uses ROS2 to publish linear and angular velocities to drive the four-wheeled differential robot. The differential drive model is chosen because of its simple structure, flexible control, and suitability for obstacle avoidance in confined spaces. The speed difference between the left and right wheels determines the steering angle, which aligns with the actual motion characteristics of mobile robots.

[0076] The reward calculation unit generates instant rewards based on the robot's state, including:

[0077] Reward function definition:

[0078] ;

[0079] In the formula, It is the target reward function (i.e., the first reward function). It is the distance reward function (i.e., the second reward function). It is the forward reward function (i.e., the third reward function). It is the obstacle reward function (i.e., the fourth reward function).

[0080] In the target reward function, It is usually set to a positive value to motivate the agent to approach the endpoint, and its calculation formula is:

[0081] ;

[0082] In the formula, This represents a positive reward value, which is awarded when the robot reaches the target area.

[0083] Distance reward function Using Euclidean distance as a heuristic function term can reduce the blind spots in the environment exploration process of the SAC algorithm and improve planning efficiency. Its calculation formula is:

[0084] ;

[0085] In the formula, It is a negative constant used to control the magnitude of the distance reward function. This represents the robot's position state at time k. This is the target location state. This means that if the robot gets closer to the target area, it will receive a smaller negative reward value. Therefore, This can help robots reach the target area quickly.

[0086] Forward reward function This is based on the change in distance of the robot relative to the next waypoint within consecutive time steps:

[0087] ;

[0088] In the formula, It is a positive integer used for control. Size, The distance between the robot's previous step and the next waypoint. This is the distance between the current step and the next waypoint. This reward motivates the agent to minimize the distance to the next waypoint at each step, ensuring the agent moves in the correct direction.

[0089] Obstacle penalty The value used to ensure the safety of the robot during operation depends on the collision risk:

[0090] ;

[0091] In the formula, A negative value is given when the robot collides with any obstacle in the environment, and there is no penalty when no collision occurs.

[0092] The task termination judgment unit is used to end the current training round. The triggering conditions include: the robot successfully reaches the target, collides with an obstacle, or the cumulative number of steps reaches a preset number (such as 1000 steps). This design can ensure the training efficiency of each round, and improve the generalization ability by frequently resetting the strategy to learn in various initial states.

[0093] Therefore, based on Figure 2 The algorithm framework of this invention involves the robot agent first constructing a state space based on LiDAR data and pose information. This state space matrix is ​​then input into a neural network feature extractor module based on a self-attention mechanism to capture key spatiotemporal features in the state space. The extracted features are fed in parallel to the Actor network and Critic network of the PPO algorithm. The Critic network calculates and outputs the temporal difference error. δ t This error is used both to update the parameters of the Critic network itself and to calculate the advantage function of the generalized advantage estimate (GAE). A t In GAE, γ This represents the discount factor, which determines the importance of future rewards; it is set to 0.97. λThe smoothing coefficient for GAE is 0.95, used to balance bias and variance. The Actor network outputs the policy probability ratio, which is the ratio of action probabilities under the new and old policies. This ratio is input to the dynamic pruning range adaptive mechanism module to calculate the dynamic pruning parameters. The algorithm combines generalized advantage estimation and pruning parameters to calculate the policy deviation of the new policy, and then optimizes the Actor network's objective. L clip ( θ The loss function is used to update the parameters of the Actor network. L clip ( θ )middle, θ These are the parameters of the Actor network; r t ( θ () represents the ratio of the probability of the new and old strategies on the same state action; A t It is the advantage function estimate calculated by GAE. yes clip Coefficients. Simultaneously, the Actor network outputs action instructions based on the current policy, namely linear velocity and angular velocity. After the agent executes this action, the environmental state transitions to the next time step, thus initiating the next processing cycle.

[0094] In summary, this invention specifically addresses the navigation problem in dynamic unknown environments through a multimodal self-attention feature extraction network and a dynamic pruning range adaptive strategy:

[0095] (1) The core function of the multimodal self-attention feature extraction network is to provide “high-quality state input” for the PPO algorithm: its output multidimensional fusion feature integrates the spatial relationship between the obstacle distribution and polar coordinate pose of the lidar point cloud, replacing the simple splicing of raw data or single-modal features in the traditional PPO, solving the problem of fuzzy environmental perception, enabling the policy network to accurately understand the robot’s pose information, surrounding obstacle information and target orientation information, providing a reliable environmental semantic basis for subsequent decision-making.

[0096] (2) The core function of the dynamic pruning range adaptive strategy is to perform precise optimization of the strategy update process based on the above high-quality state input: by calculating the KL divergence of the new and old strategies based on the fusion features, i.e. measuring the difference in decision, the pruning parameters of the PPO algorithm are dynamically adjusted to solve the problem of policy update imbalance, so that the strategy based on accurate environmental perception can both expand the exploration range to adapt quickly when dynamic obstacles appear, and tighten the constraints to converge robustly when the environment is stable.

[0097] This method is applicable to autonomous mobile robot navigation tasks in dynamic and unknown environments, such as warehousing and logistics, and disaster search and rescue, and has significant advantages, especially in complex environments containing both static and dynamic obstacles.

[0098] In one embodiment, the present invention uses four algorithms: PPO, PPO-SAM (PPO with Self-Attention Mechanism), PPO-DCRA (PPO with Dynamic Clip Range Adaptation), and PPO-DSAM. Figure 5 Experiments were conducted to verify this in the environmental map shown, specifically in... Figure 5 Train in (a), and then use the trained model in Figure 5 Generalization ability tests were conducted in (b) and (c).

[0099] The key hyperparameters involved in the experiment are shown in Table 1:

[0100] Table 1

[0101]

[0102] Figure 6 The images show the navigation path trajectories with the highest reward values ​​for four algorithms (PPO, PPO-SAM, PPO-DCRA, and PPO-DSAM) in a static environment for four multi-object navigation tasks (tasks I to IV). Specifically, (a) shows the trajectory of the PPO-DSAM algorithm in tasks I to IV (from left to right), (b) shows the trajectory of the PPO-DCRA algorithm in tasks I to IV (from left to right), (c) shows the trajectory of the PPO-SAM algorithm in tasks I to IV (from left to right), and (d) shows the trajectory of the PPO algorithm in tasks I to IV (from left to right).

[0103] The results show that in each multi-target navigation task, the PPO-DSAM algorithm outperformed the other three algorithms in terms of path smoothness, path efficiency, and navigation safety. PPO-DSAM, PPO-DCRA, and PPO-SAM algorithms did not experience collisions with obstacles or get stuck in any area in any task, and PPO-DSAM's path was smoother and more efficient. Conversely, while the PPO algorithm completed navigation tasks I and II, its trajectory exhibited significant fluctuations and unnecessary turns. Furthermore, collisions with obstacles occurred in navigation tasks III and IV, leading to task failure.

[0104] Table 2 shows the performance metrics of each algorithm in 1000 tests for each task. The results show that in multi-target navigation tasks in static environments, the PPO-DSAM algorithm exhibits significant generalization ability and performance advantages. Across all tasks, the average task success rate of PPO-DSAM is 97.1% (95.5%-99.2%), an improvement of 8.9%-14.4% compared to the second-best algorithm, PPO-DCRA (83.2%-93.1%). Notably, the traditional PPO algorithm completely fails in tasks III and IV, with its 100% collision rate directly leading to task failure, while PPO-DSAM achieves a collision rate of only 3.9% in task IV, where other algorithms generally show higher collision rates.

[0105] Table 2

[0106]

[0107] Figure 7 The image shows typical trajectories of four algorithms (PPO, PPO-SAM, PPO-DCRA, and PPO-DSAM) in multi-target navigation tasks in dynamic environments (Tasks I to IV). Specifically, (a) shows the trajectory of the PPO-DSAM algorithm in Tasks I to IV (from left to right), (b) shows the trajectory of the PPO-DCRA algorithm in Tasks I to IV (from left to right), (c) shows the trajectory of the PPO-SAM algorithm in Tasks I to IV (from left to right), and (d) shows the trajectory of the PPO algorithm in Tasks I to IV (from left to right).

[0108] It can be observed that the path generated by PPO-DSAM exhibits higher smoothness and predictability when avoiding dynamic obstacles. Its trajectory turning amplitude is significantly smaller than that of PPO-DCRA and PPO-SAM, and it does not exhibit the frequent collisions or local oscillations caused by decision errors that occur in traditional PPO algorithms.

[0109] Table 3's quantitative analysis further validates the robustness of PPO-DSAM under dynamic disturbances. In the four tasks, PPO-DSAM outperformed other algorithms with an average reward of 11.85%, representing a 38.7% improvement over PPO-DCRA (average 8.72%) and a 19.1% improvement over PPO-SAM (average 10.09%). Regarding path planning efficiency, PPO-DSAM achieved the best performance in Task I with an average path length of 41.58m, a 10.2% reduction compared to PPO-DCRA and a 9.4% reduction compared to PPO-SAM. Notably, as task complexity increased (e.g., Task III involved longer-distance dynamic obstacle avoidance), PPO-DSAM's path length (56.69m) remained significantly lower than PPO-DCRA (67.46m) and PPO-SAM (61.28m), indicating its stronger adaptability to dynamic environmental changes. In terms of safety, PPO-DSAM demonstrated excellent obstacle avoidance capabilities. Its collision rate is significantly lower than that of PPO-DCRA and PPO-SAM. Especially in the more challenging Task III, PPO-DSAM's collision rate was controlled at 12.4%, while PPO-DCRA's was as high as 21.6%. In terms of success rate, PPO-DSAM achieved an average of 90.2% across the four tasks, an improvement of 9.3% and 7.4% compared to PPO-DCRA and PPO-SAM, respectively; while the traditional PPO algorithm completely failed in dynamic environments.

[0110] Table 3

[0111]

[0112] As can be seen, comparative experiments between the PPO-DSAM model of this invention and existing PPO, PPO-SAM, and PPO-DCRA models show that the present invention has significant improvements in indicators such as average reward, path efficiency, completion time, and task success rate, especially in the dynamic obstacle scenario where the generalization ability is better.

[0113] The steps of the various methods described above are only for clarity. In practice, they can be combined into one step or some steps can be split into multiple steps. As long as they include the same logical relationship, they are all within the protection scope of this invention. Adding insignificant modifications or introducing insignificant designs to the algorithm or process, without changing the core design of the algorithm and process, are also within the protection scope of this invention.

[0114] Another embodiment of the present invention relates to an autonomous navigation system for a mobile scanning robot. The implementation details of this embodiment's autonomous navigation system are described below. The following details are provided for ease of understanding and are not essential for implementing this solution. The autonomous navigation system for the mobile scanning robot in this embodiment includes:

[0115] The data acquisition module is used to acquire the polar coordinate pose of the mobile robot in the target environment, as well as the LiDAR point cloud obtained by the mobile robot scanning the target environment.

[0116] The feature extraction module is used to extract features from the polar coordinate pose and the lidar point cloud respectively, so as to obtain pose features containing the spatial geometric relationship between the mobile robot and the navigation target in the target environment, and environmental features containing the distribution of obstacles in the target environment.

[0117] The feature fusion module is used to fuse pose features and environmental features to obtain fused features;

[0118] The autonomous navigation module is used to determine the navigation strategy of the mobile robot based on the Proximal Policy Optimization (PPO) algorithm by fusing features for autonomous navigation. The pruning range of the PPO algorithm is dynamically adjusted according to the KL divergence between the old and new navigation strategies to control the magnitude of the navigation strategy update of the PPO algorithm.

[0119] It is not difficult to see that this embodiment is a system embodiment corresponding to the above method embodiments, and this embodiment can be implemented in conjunction with the above method embodiments. The relevant technical details and technical effects mentioned in the above embodiments are still valid in this embodiment, and will not be repeated here to reduce repetition. Accordingly, the relevant technical details mentioned in this embodiment can also be applied to the above embodiments.

[0120] It is worth mentioning that all modules involved in this embodiment are logical modules. In practical applications, a logical unit can be a physical unit, a part of a physical unit, or a combination of multiple physical units. Furthermore, to highlight the innovative aspects of this invention, this embodiment does not introduce units that are not closely related to solving the technical problem proposed by this invention; however, this does not mean that other units are absent from this embodiment.

[0121] Another embodiment of the present invention relates to a computer device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the autonomous navigation method for the mobile scanning robot in the above embodiments.

[0122] The memory and processor are connected via a bus, which can include any number of interconnecting buses and bridges, connecting various circuits of one or more processors and memories. The bus can also connect various other circuits, such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and will not be described further herein. The bus interface provides an interface between the bus and the transceiver. The transceiver can be a single element or multiple elements, such as multiple receivers and transmitters, providing a unit for communicating with various other devices over a transmission medium. Data processed by the processor is transmitted over the wireless medium via an antenna, which further receives data and transmits it to the processor.

[0123] The processor manages the bus and general processing, and also provides various functions, including timing, peripheral interfaces, voltage regulation, power management, and other control functions. Memory is used to store data used by the processor during operation.

[0124] Another embodiment of the present invention relates to a computer-readable storage medium storing a computer program. When executed by a processor, the computer program implements the method embodiments described above.

[0125] That is, those skilled in the art will understand that all or part of the steps in the methods of the above embodiments can be implemented by a program instructing related hardware. This program is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0126] Those skilled in the art will understand that the above embodiments are specific embodiments for implementing the present invention, and in practical applications, various changes can be made to them in form and detail without departing from the spirit and scope of the present invention.

Claims

1. A method for autonomous navigation of a mobile scanning robot, characterized in that, Applied to mobile robots, the method includes: The polar coordinate pose of the mobile robot in the target environment and the LiDAR point cloud obtained by the mobile robot scanning the target environment are obtained. Feature extraction is performed on polar coordinate pose and lidar point cloud to obtain pose features containing the spatial geometric relationship between the mobile robot and the navigation target in the target environment, as well as environmental features containing the distribution of obstacles in the target environment. Using pose features as the query for multi-head self-attention and environmental features as the key / value pair, the feature correlation between pose features and environmental features is obtained to filter out environmental regions in the target environment that are related to the current navigation intent; the environmental features of the filtered environmental regions are fused and then concatenated with the environmental features of the target environment to obtain the fused features; Based on the Proximal Policy Optimization (PPO) algorithm, a navigation strategy for a mobile robot is determined by fusing features for autonomous navigation. The clipping range of the PPO algorithm is dynamically adjusted according to the KL divergence between the old and new navigation strategies and processed using the Exponential Moving Average (EMA) method to control the magnitude of navigation strategy updates. When the KL divergence exceeds 1.5 times the preset threshold, the minimum clipping range is used; when it is less than 1.5 times the preset threshold, the maximum clipping range is used. The KL divergence is estimated using the following formula: ； In the formula, For experience playback batch size, and These represent the navigation strategies before and after the update. and These represent the policy parameters before and after the navigation policy update. and Representing the first The actions and states of each round; The clipping range is generated based on the estimated KL divergence using the following formula: ； In the formula, and These represent the minimum and maximum values ​​of the clipping range, respectively. For the preset target threshold, and These represent the maximum and minimum values ​​of the KL divergence, respectively.

2. The autonomous navigation method for a mobile scanning robot according to claim 1, characterized in that, The reward function of the PPO algorithm includes: a first reward function that rewards based on whether the mobile robot reaches the area where the navigation target is located; a second reward function that rewards based on the distance between the mobile robot and the navigation target; a third reward function that rewards based on the change in distance of the mobile robot relative to the next waypoint within consecutive time steps; and a fourth reward function that rewards based on whether the mobile robot collides with an obstacle.

3. The autonomous navigation method for a mobile scanning robot according to claim 1, characterized in that, The process of extracting features from the polar coordinate pose and the lidar point cloud to obtain pose features containing the spatial geometric relationship between the mobile robot and the navigation target in the target environment, and environmental features containing the distribution of obstacles in the target environment, includes: The 2D polar coordinate pose is mapped to 128 dimensions through the first layer MLP in the two-layer multilayer perceptron MLP. After regularization, it is extended to 256 dimensions through the second layer MLP to obtain pose features that include the spatial geometric relationship between the mobile robot and the navigation target in the target environment. The 64-dimensional LiDAR point cloud is embedded into a 128-dimensional space through a projection layer. After modeling the long-range dependency of the LiDAR point cloud through a multi-head self-attention mechanism, the environmental features including the distribution of obstacles in the target environment are obtained by combining the feedforward network processing.

4. The autonomous navigation method for a mobile scanning robot according to claim 1, characterized in that, The Proximity Policy Optimization (PPO) algorithm determines the navigation strategy of a mobile robot through feature fusion for autonomous navigation, including: Based on the PPO algorithm, the angular velocity and linear velocity of the mobile robot are determined by fusing features, and autonomous navigation is performed based on the angular velocity and linear velocity of the mobile robot.

5. A system based on the autonomous navigation method for a mobile scanning robot as described in any one of claims 1 to 4, characterized in that, The system includes: The data acquisition module is used to acquire the polar coordinate pose of the mobile robot in the target environment, as well as the LiDAR point cloud obtained by the mobile robot scanning the target environment. The feature extraction module is used to extract features from the polar coordinate pose and the lidar point cloud respectively, so as to obtain pose features containing the spatial geometric relationship between the mobile robot and the navigation target in the target environment, and environmental features containing the distribution of obstacles in the target environment. The feature fusion module is used to fuse pose features and environmental features to obtain fused features; The autonomous navigation module is used to determine the navigation strategy of the mobile robot based on the Proximal Policy Optimization (PPO) algorithm by fusing features for autonomous navigation. The pruning range of the PPO algorithm is dynamically adjusted according to the KL divergence between the old and new navigation strategies to control the magnitude of the navigation strategy update of the PPO algorithm.

6. A computer device, characterized in that, include: At least one processor; And a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the autonomous navigation method for a mobile scanning robot as described in any one of claims 1 to 4.

7. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the autonomous navigation method for a mobile scanning robot as described in any one of claims 1 to 4.

Images