An end-to-end mobile robot navigation obstacle avoidance method

Abstract

This invention discloses an end-to-end navigation and obstacle avoidance method for mobile robots. This method innovatively constructs a semantically guided anisotropic constraint mechanism, transforming high-level visual semantics into low-level navigation constraint parameters through a differentiable parameter generation network. Based on this, it utilizes a Riemannian metric field to reshape the gradient optimization space of the deep unfolded neural network encoder, mapping traditional isotropic Euclidean geometric features into semantic latent distance features with direction sensitivity and differentiated constraint attributes. Finally, combined with a semantically guided regularized neural motion planner, it effectively solves problems such as narrow-path edge-hugging passage, flexible obstacle crossing, and negative obstacle avoidance in unstructured environments, achieving intelligent navigation with both high safety and high efficiency.

Description

Technical Field

[0001] This invention relates to the technical field of mobile robot navigation and autonomous driving, and in particular to an end-to-end mobile robot navigation and obstacle avoidance method. Background Technology

[0002] In navigation tasks for mobile robots and autonomous driving, achieving real-time and safe obstacle avoidance in complex, crowded, and unknown environments is a core challenge. Traditional navigation methods typically employ a modular architecture, i.e., a sequential pipeline of "perception-mapping-planning-control." However, this approach suffers from error accumulation and often simplifies the handling of non-convex obstacles by using bounding boxes, leading to reduced maneuverability.

[0003] In recent years, end-to-end model learning methods based on point clouds, represented by NeuPAN, have made significant progress. NeuPAN directly maps the original laser point cloud into latent distance features through a deep unfolded network and combines it with a neural regularized motion planner to achieve high-precision geometric obstacle avoidance. However, the existing NeuPAN framework is a purely "geometric" navigation framework, where the environment is abstracted as a series of attribute-less geometric points. This "semantic blind spot" leads to the following serious problems, limiting its application in real-world social environments: From a purely geometric perspective, roadside weeds, slopes, etc., are identified as obstacles, causing robots to take detours and fail to utilize accessible shortcuts; LiDAR relies on the echo of emitted beams for distance measurement. For downward staircases, deep pits, or completely transparent glass doors, the laser often shines into infinity or falls into blind spots, resulting in no echo data. Purely geometric methods may mistakenly assume that the path ahead is open and continue moving at full speed, easily leading to falls or collisions; It lacks directional sensitivity, and the obstacle avoidance strategy based on Euclidean distance generates isotropic repulsive fields, failing to distinguish between "collision with a wall" and "grazing a wall," causing robots to adopt overly conservative movement strategies in confined spaces such as narrow passages, reducing environmental traversal efficiency, etc.

[0004] Therefore, the industry urgently needs a navigation method that can deeply integrate high-level visual semantic information into low-level geometric motion planning, which can retain the high accuracy, real-time performance and interpretability of NeuPAN-like methods, while also giving robots the ability to understand environmental attributes, and achieve intelligent navigation that can "see obstacles and understand obstacles". Summary of the Invention

[0005] The purpose of this invention is to overcome the shortcomings and deficiencies of existing technologies and provide an end-to-end navigation and obstacle avoidance method for mobile robots. It innovatively constructs a semantically guided anisotropic constraint mechanism, transforming high-level visual semantics into low-level navigation constraint parameters through a differentiable parameter generation network. Based on this, it utilizes a Riemannian metric field to reshape the gradient optimization space of the deep unfolded neural network encoder, mapping traditional isotropic Euclidean geometric features into semantic latent distance features with direction sensitivity and differentiated constraint attributes. Finally, combined with a semantically guided regularized neural motion planner, it effectively solves problems such as narrow-path edge-hugging passage, flexible obstacle crossing, and negative obstacle avoidance in unstructured environments, achieving intelligent navigation with both high safety and high efficiency.

[0006] To achieve the above objectives, the technical solution provided by this invention is: an end-to-end mobile robot navigation and obstacle avoidance method, comprising the following steps:

[0007] 1) Multimodal feature extraction and alignment: Simultaneously collect LiDAR point cloud data and visual image data of the robot; use a visual semantic segmentation network to extract semantic category information of the environment, and spatially align the semantic category information with the LiDAR point cloud data; secondly, extract local geometric feature information of the LiDAR point cloud data based on the spatial neighborhood relationship of the LiDAR point cloud data; construct a multimodal composite feature set containing semantic category information and local geometric feature information.

[0008] 2) Navigation constraint parameter generation: Construct a differentiable parameter generation network, taking the multimodal composite feature set from step 1) as input, and mapping it to a set of continuous adaptive navigation constraint parameters. These parameters are used to quantify the normal repulsion strength, orientation sensitivity, and boundary relaxation of obstacles in the optimized model of the subsequently constructed deep unfolded neural network encoder. The differentiable parameter generation network is constructed through a multilayer sensing mechanism and has complete differentiability, allowing the use of the chain rule to backpropagate the physical execution deviations in the actual operation of the robot as gradients to calibrate the network weights online.

[0009] 3) Semantic-aware feature encoding: Construct a semantic-aware deep unfolded neural network encoder. The deep unfolded neural network encoder is based on the iterative steps of the convex optimization algorithm. An anisotropic metric matrix is ​​constructed using the adaptive navigation constraint parameters in step 2). A semantically guided Riemann metric field is established in the feature space using the anisotropic metric matrix. The gradient update in the optimization iteration process is preprocessed using the Riemann metric field, and the LiDAR point cloud data is mapped into semantic latent distance features that contain semantic constraint information.

[0010] 4) Motion planning and control: Based on the model predictive control framework, a semantically guided regularized neural motion planner is constructed. It directly receives semantic latent distance features as input. These semantic latent distance features have been pre-incorporated with direction sensitivity and soft and hard constraints through a Riemannian metric field. The objective function, which includes trajectory tracking error and obstacle avoidance cost of semantic latent distance features, is minimized through gradient descent. This enables the semantically guided regularized neural motion planner to generate trajectory strategies that conform to semantic logic during the mathematical optimization process, and finally outputs smooth control commands that satisfy dynamic constraints.

[0011] 5) Closed-loop iteration and adaptive calibration: The predicted trajectory generated by the semantically guided regularized neural motion planner is fed back to the deep unfolded neural network encoder in step 3), the semantic latent distance features are recalculated, and alternating iterative optimization is performed until the final control command is output. At the same time, the deviation between the robot's actual motion execution state and the planned expected state is monitored in real time. Based on the deviation, the network weights of the differentiable parameter generation network in step 2) are updated online, so that the robot can automatically adjust the adaptive navigation constraint parameters of obstacles in actual operation.

[0012] Furthermore, in step 1), the specific implementation of obtaining and aligning the multimodal composite feature set includes: using a lightweight visual model to perform reasoning on the image to obtain a semantic mask. The lightweight visual model uses the YOLO series or SegFormer; for the i-th point cloud acquired by the LiDAR... ,in , , Represents the geometric position coordinates of the i-th point cloud, using the extrinsic parameter matrix. and intrinsic parameter matrix Calculate Projected coordinates on the image ,in and These represent the horizontal and vertical pixel coordinates of the i-th point cloud in the image pixel coordinate system, respectively; index semantic mask. Obtain the semantic category of the i-th point cloud. and classify the semantics Encoded as semantic vectors The augmented input vector is formed by concatenating it with the original geometric position coordinates. .

[0013] Furthermore, in step 1), the specific implementation method for extracting local geometric feature information using the spatial neighborhood relationship of LiDAR point cloud data is as follows: Utilizing the scan line arrangement structure of the LiDAR point cloud data, a sliding window algorithm with linear time complexity is used for feature extraction. First, a topological neighborhood index is constructed for the v-th point cloud on the u-th scan line. Where u is the harness index, v is the horizontal scan index, and the index is in Left and right neighbor point sets on the same scan line ,in , Let and represent the ya and y+a points on the u-th scan line, respectively, where 'a' is the preset window radius. Next, a geometric continuity check is performed. To eliminate interference from abrupt depth changes at object edges, the left and right neighbor point sets are calculated. The Euclidean distance between adjacent points in the middle, if the distance between adjacent points is greater than 1. ,in If a preset breakage threshold is set, a geometric discontinuity is determined, and all points outside the breakage point are removed. If the distance between adjacent points is less than or equal to... If the target point is found to be geometrically continuous, the adjacent point is retained, and a final structure containing the target point is constructed. Maximum contiguous subset Then, based on the tangent vector of local regression, the maximum continuous subset is calculated. Perform local linear regression or principal component analysis, by minimizing Given the sum of squared orthogonal distances from all points to the fitted line, calculate the unit direction vector of that line, defined as the target point. tangent vector This characterizes the local extension direction of the obstacle surface; then, the local geometric feature vector is calculated through vector cross product operation. ,in As the vertical axis vector, the final result is the local geometric feature information representing the repulsion direction.

[0014] Furthermore, in step 1), a geometric completion mechanism for negative obstacle hazard areas is also included: a joint determination method of visual edge detection and laser ranging status is used to construct a virtual wall to close the hazard area; when the visual semantic segmentation network identifies areas such as "ground edge", "staircase entrance" or "sunken area" in front, if the laser radar shows signal loss or abrupt change in depth data in the projection direction of these areas, it is determined that there is a negative obstacle in the area and the area is defined as a negative obstacle hazard area; using camera intrinsic parameters and pose information, the edge pixels in the image are back-projected to three-dimensional space to fit a three-dimensional edge baseline located on the plane; starting from the three-dimensional edge baseline, a virtual point cloud array is generated by interpolation according to a preset point cloud density to form a virtual wall that physically closes the negative obstacle hazard area, and the virtual point cloud array is assigned a high-risk semantic label; the generated virtual point cloud array is merged with the original laser radar point cloud data as input for subsequent steps.

[0015] Furthermore, in steps 2) and 5), the specific topology and online adaptive update mechanism of the differentiable parameter generation network are as follows:

[0016] The differentiable parameter generation network adopts a lightweight fully connected neural network structure, including an input layer, several hidden layers, and an output layer; the input layer receives multimodal feature vectors constructed based on a multimodal composite feature set. ,in Let E be the semantic category probability distribution vector, and E be the visual prediction information entropy. This represents the statistics of the local geometric feature vectors. In the output layer, to ensure the mathematical validity of the physical parameters, a Softplus activation function is used to impose a non-negativity constraint on the output, generating a set of positive definite physical impedance parameters. The adaptive navigation constraint parameters mentioned above have the following meanings: Normal repulsion coefficient Used to define the intensity of the repulsive force along the normal direction of the obstacle; tangential sensitivity. Used to define constraint weights along the tangent direction of the obstacle surface; relaxation factor This is used to define the virtual indentation or allowable penetration depth of obstacle geometry boundaries to handle passable soft obstacles. Additionally, this differentiable parameter generation network uses virtual data generated from a pre-set expert rule table for cold-start pre-training, enabling the network to possess basic navigational common sense. Next, the physical consistency loss function is constructed. During robot operation, a physical consistency loss function that measures the "planning-execution" deviation is built in real time. :

[0017] ;

[0018] In the formula, The desired velocity vector generated by the regularized neural motion planner. The actual velocity vector fed back by the wheel speed gauge. The vertical vibration amplitude detected by the inertial measurement unit. , The weights are defined as follows: Then, based on the chain rule, end-to-end gradient backpropagation is performed, utilizing the fully differentiable properties of the deep unfolded neural network encoder and the regularized neural motion planner to construct an end-to-end gradient path from the loss function to the network weights W. :

[0019] ;

[0020] In the formula, The adaptive navigation constraint parameter set is used, and LDF represents the semantic latent distance feature. By calculating the gradient based on the chain rule, the motion hindrance or terrain bumps encountered by the robot in the physical world are transformed into effects on the network weights. The correction signal is then used; finally, a single-step online calibration is performed, and the network weights are updated online using a single-step stochastic gradient descent strategy. This enables the network to automatically adjust the output impedance parameters based on real-time physical feedback when encountering unknown terrain textures, thereby achieving online system identification and adaptation of environmental physical properties.

[0021] Furthermore, in step 3), the specific construction and operation process of the semantic-aware deep unrolled neural network encoder is as follows:

[0022] The deep unfolded neural network encoder is a deep neural network unfolded based on the iterative steps of the classical optimization algorithm PIBCD to solve the distance minimization problem. It contains K cascaded semantic optimization layers, each corresponding to one gradient descent iteration on the semantic Riemannian manifold space. First, an anisotropy metric matrix is ​​constructed using the normal repulsion coefficient output in step 2). and tangential sensitivity Combined with the local geometric feature vector extracted in step 1), Through the spectral decomposition formula Construct an anisotropy metric matrix M, where The anisotropy, represented by the identity matrix, indicates a spatial metric scale that differentiates between the normal and tangent directions of the obstacle surface. Mathematically, this defines a local Riemannian metric field, causing the shape of the equipotential surfaces around the obstacle to change semantically. Secondly, Riemannian preprocessing gradient updates are performed: in the forward propagation of each layer of the deep unfolded neural network encoder, the original geometric gradient based on Euclidean distance is first calculated. Then, the inverse of the anisotropy metric matrix is ​​used. For the original geometric gradient Left multiplication is performed, followed by Riemannian preprocessing to suppress gradient components in highly repulsive directions and enhance gradient components in less sensitive directions. Finally, a relaxation bias is introduced for state iteration: a relaxation factor is introduced into the gradient update formula. The defined bias term, bias, is used to update the state of the k-th layer as follows:

[0023] ;

[0024] In the formula, , These are the semantic latent distance features output from layers k and (k+1), respectively. The network step size parameter, after K layers of iteration, outputs semantic latent distance features that implicitly contain semantic physical attributes. Unlike obstacle avoidance strategies based on Euclidean distance, this anisotropic metric matrix M makes the semantic latent distance features generated by the semantic perception deep unfolded neural network encoder exhibit an ellipsoidal equipotential surface in spatial distribution that varies with semantic category. Through this anisotropic constraint mechanism, differentiated spatial constraints and avoidance strategies for obstacles with different attributes are realized.

[0025] Furthermore, in step 4), the specific implementation of the semantically guided regularized neural motion planner includes the following:

[0026] The first step is to construct a differentiable model predictive control optimization problem, defining the prediction time domain as H. The goal of the regularized neural motion planner is to find a set of optimal control sequences at time t. This allows the robot to predict its state sequence at time t. Satisfy kinematic constraints and minimize the total cost function ,in , , These represent the control commands at times t, t+1, and t+H-1, respectively. , , Let represent the predicted states at times t, t+1, and t+H-1, respectively; next, we construct a composite objective function including Riemannian manifold constraints, i.e., the total cost function. The total cost function It consists of three parts, specifically expressed as follows:

[0027] ;

[0028] In the formula, To determine the trajectory tracking error, the predicted state at time j is calculated. With global reference path point The Euclidean distance error between them is used to guide the robot to the target point; To control the smoothing term, the control command at time j is calculated. The amplitude and norm of its rate of change are used to limit the robot's maximum speed and acceleration to ensure smooth motion; the third term This is a penalty term for semantic latent distance features. This is a collection of lidar point cloud data. The output of step 3) The semantic latent distance feature value of a point at time j. As a penalty function, a one-sided activation function can be used to map semantic latent distance features to obstacle avoidance costs: ,in, As a safety threshold, since the input semantic latent distance feature value has been distorted by the Riemann metric field, minimizing the third term is equivalent to forcing the robot to automatically find the region with the lowest "semantic cost value", thereby spontaneously generating intelligent behaviors such as "traversing along the edge", "staying away from high-risk areas" or "crossing flexible obstacles". Then, the optimal control sequence is updated by gradient descent or Newton's method. After several iterations and convergence, the first term of the output optimal control sequence can be used as the control command at the current moment and sent to the robot's underlying controller.

[0029] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0030] 1. Real-time performance: Feature extraction with linear complexity is achieved by utilizing the topology of the LiDAR scanning line, avoiding time-consuming nearest neighbor search and meeting the real-time requirements of embedded platforms.

[0031] 2. Anisotropy and Flexible Passage: By introducing Riemannian metric fields and boundary relaxation quantities through a semantically perceptive deep unfolded neural network encoder, the robot can distinguish between "untouchable hard constraints" and "passable soft constraints" and has the ability to pass through "tangentially along the edge", which significantly improves the passability in narrow and crowded environments.

[0032] 3. Model-based end-to-end learnability and interpretability: Inheriting the mathematical rigor of the NeuPAN architecture, the network layers correspond to the iterative steps of the optimization algorithm, which has good interpretability.

[0033] 4. No need for large-scale training and adaptive calibration: It abandons the dependence of end-to-end large models on massive amounts of labeled data and adopts a mechanism of "adaptive calibration based on physical execution deviations". The robot can automatically calibrate its perception of the environment through deviation feedback in actual operation and has the ability to continuously evolve.

[0034] 5. Comprehensive safety protection: A geometric completion mechanism based on visual edges is designed for negative obstacle danger areas to generate a virtual point cloud array with high repulsion intensity, filling the perception blind spot of LiDAR.

[0035] 6. Higher passage efficiency: For soft obstacles or slopes, semantic weighting is used to prevent the robot from getting stuck by "false obstacles". Attached Figure Description

[0036] Figure 1 This is a flowchart of the method of the present invention, which shows the closed-loop data flow from perception, differentiable parameter generation network, to semantic perception deep unfolded neural network encoder, and then to semantic guidance regularized neural motion planner.

[0037] Figure 2 This is a schematic diagram of extracting geometric features and anisotropic measurement fields using scan line topology.

[0038] Figure 3 This is a diagram of the joint structure of a differentiable parameter generation network and a semantically perceptive deep unfolded neural network encoder. Detailed Implementation

[0039] The present invention will be further described in detail below with reference to the embodiments and accompanying drawings, but the embodiments of the present invention are not limited thereto.

[0040] like Figures 1 to 3 As shown in the figure, this embodiment discloses an end-to-end mobile robot navigation and obstacle avoidance method, the implementation steps of which are as follows:

[0041] 1) Multimodal feature extraction and alignment

[0042] like Figure 1 As shown, the first step is semantic segmentation: camera image data is input into a lightweight visual semantic segmentation network such as YOLOv11-Seg to extract semantic category information of the environment; the second step is geometric extraction based on the scan line arrangement structure: utilizing the scan line arrangement structure of the LiDAR point cloud data, such as... Figure 2 As shown, for any point in the lidar point cloud data It directly indexes its local neighborhood points on the same scan line, tangent vector The extension direction and normal vector of the obstacle surface are characterized by calculations using local difference or linear fitting. By combining the ground plane normal vector and using the cross product operation, the local geometric feature vector is calculated. ,in The vertical axis vector ultimately yields local geometric features representing the repulsion direction. There's also geometric completion for negative obstacle hazard areas: when the visual semantic segmentation network detects areas like "ground edges," "stairwells," or "sunken areas," if the lidar experiences signal loss or abrupt changes in depth data in the projection direction of these areas, it's determined that a negative obstacle exists in that area, defining it as a negative obstacle hazard area. Using camera intrinsic parameters and pose information, edge pixels in the image are back-projected into 3D space, fitting a 3D edge baseline on the plane. Starting from this 3D edge baseline, a virtual point cloud array is generated by interpolation according to a preset point cloud density, forming a virtual wall that physically encloses the negative obstacle hazard area. The virtual point cloud array is then assigned a high-risk semantic label. The generated virtual point cloud array is merged with the original lidar point cloud data as input for subsequent steps. Finally, there's fusion alignment: through sensor joint calibration parameters, visual semantic information is projected onto the 3D lidar point cloud, constructing a multimodal composite feature set containing location information, semantic information, and local geometric feature information.

[0043] 2) Generation of navigation constraint parameters

[0044] This step constructs a differentiable parameter generation network, whose function is to convert semantic information into adaptive navigation constraint parameters; for example... Figure 3 As shown, the differentiable parameter generation network adopts a lightweight fully connected neural network structure, including an input layer, several hidden layers composed of fully connected (FC) layers and ReLU activation functions, and an output layer. The input layer receives multimodal feature vectors constructed based on a multimodal composite feature set. ,in Let E be the semantic category probability distribution vector, and E be the visual prediction information entropy. As a statistical measure of the local geometric feature vector, in the output layer, to ensure the mathematical validity of the physical parameters, a Softplus activation function is used to impose a non-negativity constraint on the output, generating a set of positive definite physical impedance parameters. That is, the above adaptive navigation constraint parameters, where the normal repulsion coefficient is... Define the avoidance priority along the normal direction. For virtual points containing walls or negative obstacles, output a very high value to create a hard constraint; for grass, output a lower value. Tangential sensitivity. Define the cost weights along the tangent direction, such as... Figure 2 As shown in case A, for the wall, Output a lower value, much smaller than The algorithm allows tangential close-to-the-edge and edge-to-edge passage, but prohibits head-on collisions; relaxation factor Define the virtual indentation degree of the obstacle's geometric boundary. In case B, for grass, output a positive value, allowing the planned trajectory to enter the geometric boundary and virtually penetrate it. In addition, the network uses virtual data generated by a pre-set expert rule table for cold start pre-training, enabling the network to have basic navigation common sense.

[0045] The next step is to construct the physical consistency loss function. During the robot's operation, the physical consistency loss function is constructed in real time to measure the deviation between planning and execution. :

[0046] ;

[0047] In the formula, The desired velocity vector generated by the regularized neural motion planner. The actual velocity vector fed back by the wheel speed gauge. The vertical vibration amplitude detected by the inertial measurement unit. , The weights are defined as follows: Then, based on the chain rule, end-to-end gradient backpropagation is performed, utilizing the fully differentiable properties of the deep unfolded neural network encoder and the regularized neural motion planner to construct an end-to-end gradient path from the loss function to the network weights W. :

[0048] ;

[0049] In the formula, The adaptive navigation constraint parameter set is used, and LDF represents the semantic latent distance feature. By calculating the gradient based on the chain rule, the motion hindrance or terrain bumps encountered by the robot in the physical world are transformed into effects on the network weights. The correction signal is then used; finally, a single-step online calibration is performed, and the network weights are updated online using a single-step stochastic gradient descent strategy. This enables the network to automatically adjust the output impedance parameters based on real-time physical feedback when encountering unknown terrain textures, thereby achieving online system identification and adaptation of environmental physical properties.

[0050] 3) Construct a semantic-aware deep unfolded neural network encoder

[0051] The encoder is a deep neural network based on the iterative steps of the classic optimization algorithm PIBCD to solve the distance minimization problem. It contains K cascaded semantic optimization layers, each corresponding to one gradient descent iteration on the semantic Riemannian manifold space. The first step is the construction of the anisotropy metric matrix, such as... Figure 3 As shown, the normal repulsion coefficient output in step 2) is used. and tangential sensitivity Combined with the local geometric feature vector extracted in step 1), Through the spectral decomposition formula Construct an anisotropy metric matrix M, where The anisotropy, represented by the identity matrix, indicates a spatial metric scale that differentiates between the normal and tangent directions of the obstacle surface. Mathematically, this defines a local Riemannian metric field, causing the shape of the equipotential surfaces around the obstacle to change semantically, such as a flattened ellipsoid around a wall. Secondly, Riemann preprocessing gradient updates are performed: in each layer of the encoder's forward propagation, the original geometric gradient based on Euclidean distance is first calculated. Then, the inverse of the anisotropy metric matrix is ​​used. For the original geometric gradient Left multiplication is performed, followed by Riemannian preprocessing to suppress gradient components in highly repulsive directions and enhance gradient components in less sensitive directions. Finally, a relaxation bias is introduced for state iteration: a relaxation factor is introduced into the gradient update formula. The defined bias term, bias, is updated in the following formula for each layer:

[0052] ;

[0053] In the formula, x represents the semantic latent distance feature. The network step size parameter, after being processed by layers 1, 2, ..., K, results in a semantic latent distance feature that implicitly contains semantic physical attributes. Unlike obstacle avoidance strategies based on Euclidean distance, this anisotropic metric matrix causes the semantic latent distance feature generated by the semantic perception deep unfolded neural network encoder to exhibit an ellipsoidal equipotential surface in spatial distribution that varies with semantic category, thereby enabling differentiated spatial constraints and avoidance strategies for obstacles with different attributes.

[0054] 4) Construct a semantically guided regularized neural motion planner

[0055] The first step is to construct a differentiable model predictive control optimization problem, defining the prediction time domain as H. The goal of the regularized neural motion planner is to find a set of optimal control sequences at time t. This allows the robot to predict its state sequence at time t. Satisfy kinematic constraints and minimize the total cost function ,in , , These represent the control commands at times t, t+1, and t+H-1, respectively. , , Let represent the predicted states at times t, t+1, and t+H-1, respectively; next, we construct a composite objective function including Riemannian manifold constraints, i.e., the total cost function. The total cost function It consists of three parts, specifically expressed as follows:

[0056] ;

[0057] In the formula, To determine the trajectory tracking error, the predicted state at time j is calculated. With global reference path point The Euclidean distance error between them is used to guide the robot to the target point; To control the smoothing term, the control command at time j is calculated. The amplitude and norm of its rate of change are used to limit the robot's maximum speed and acceleration to ensure smooth motion; the third term This is a penalty term for semantic latent distance features. This is a collection of lidar point cloud data. The output of step 3) The semantic latent distance feature value of a point at time j. As a penalty function, a one-sided activation function can be used to map semantic latent distance features to obstacle avoidance costs: ,in, As a safety threshold, since the input semantic latent distance feature values ​​have already been distorted by the Riemann metric field, minimizing the third term is equivalent to forcing the robot to automatically find the region with the lowest "semantic cost," thereby spontaneously generating intelligent behaviors such as "traversing along the edge," "avoiding high-risk areas," or "crossing flexible obstacles." Then, the optimal control sequence is updated using gradient descent or Newton's method. After several iterations and convergence, the first term of the output optimal control sequence can be used as the control command for the current moment and sent to the robot's underlying controller. For example, when facing grass, since... The existence of semantic potential distance features leads to a very large calculated value. The planner considers this "collision-free" and generates crossing instructions, etc. For walls in a narrow corridor: the normal vector direction of the wall, perpendicular to the wall surface, represents the high-risk direction for the robot to directly collide with the wall, while the tangential vector direction, parallel to the wall surface, represents the safe direction for the robot to travel along the wall. In the traditional Euclidean distance field, the cost of these two directions at the same distance is the same, causing the robot to be unable to distinguish between "colliding with the wall" and "sticking to the wall". This invention constructs an anisotropic metric matrix, assigning a large repulsion coefficient to the normal direction, significantly amplifying the potential collision cost in this direction and forcing the robot to move away. At the same time, it assigns a small sensitivity coefficient to the tangential direction, suppressing the cost growth in this direction. This is equivalent to constructing a "flat" safety potential field around the wall, allowing the robot to perceive at the algorithm level that: vertically colliding with the wall is "extremely dangerous", while moving parallel to the wall is "relatively safe", thus enabling it to smoothly pass through narrow passages along the edge.

[0058] 5) Closed-loop iteration and adaptive calibration

[0059] like Figure 1 As shown, the predicted trajectory generated by the regularized neural motion planner is fed back to the deep unfolded neural network encoder in step 3 to recalculate the semantic latent distance features. Alternating iterative optimization is performed until convergence and the final control command is output. Additionally, the deviation between the robot's planned desired state and its actual operating state (e.g., wheel speed feedback speed, chassis vibration) is monitored in real time, and online gradient updates are performed to construct an execution consistency loss function, such as the square of the speed error plus a vibration penalty term. Utilizing the differentiability of the deep unfolded neural network encoder and the regularized neural motion planner, the weight gradient of this loss function relative to the differentiable parameter generation network is calculated, and a single-step update is performed. For example, when the robot attempts to traverse an area visually perceived as "passable" but actually causing severe wheel slippage or obstruction, the generated speed error gradient will propagate back, forcing the regularized neural motion planner to increase the repulsion strength parameter corresponding to this type of texture feature. The next time a similar texture is encountered, the network will automatically output a high repulsion strength, and the planner will choose to detour, achieving self-supervised learning based on operational feedback.

[0060] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above embodiments. Any changes, modifications, substitutions, combinations, or simplifications made without departing from the spirit and principle of the present invention shall be considered equivalent substitutions and shall be included within the protection scope of the present invention.

Claims

1. An end-to-end navigation and obstacle avoidance method for a mobile robot, characterized in that, Includes the following steps: 1) Multimodal feature extraction and alignment: Simultaneously collect LiDAR point cloud data and visual image data of the robot; use a visual semantic segmentation network to extract semantic category information of the environment, and spatially align the semantic category information with the LiDAR point cloud data; secondly, extract local geometric feature information of the LiDAR point cloud data based on the spatial neighborhood relationship of the LiDAR point cloud data. Construct a multimodal composite feature set that includes semantic category information and local geometric feature information; 2) Navigation constraint parameter generation: Construct a differentiable parameter generation network, taking the multimodal composite feature set from step 1) as input, and mapping it to a set of continuous adaptive navigation constraint parameters. These parameters are used to quantify the normal repulsion strength, orientation sensitivity, and boundary relaxation of obstacles in the optimized model of the subsequently constructed deep unfolded neural network encoder. The differentiable parameter generation network is constructed through a multilayer sensing mechanism and has complete differentiability, allowing the use of the chain rule to backpropagate the physical execution deviations in the actual operation of the robot as gradients to calibrate the network weights online. 3) Semantic-aware feature encoding: Construct a semantic-aware deep unfolded neural network encoder. The deep unfolded neural network encoder is based on the iterative steps of the convex optimization algorithm. An anisotropic metric matrix is ​​constructed using the adaptive navigation constraint parameters in step 2). A semantically guided Riemann metric field is established in the feature space using the anisotropic metric matrix. The gradient update in the optimization iteration process is preprocessed using the Riemann metric field, and the LiDAR point cloud data is mapped into semantic latent distance features that contain semantic constraint information. 4) Motion planning and control: Based on the model predictive control framework, a semantically guided regularized neural motion planner is constructed. It directly receives semantic latent distance features as input. These semantic latent distance features have been pre-incorporated with direction sensitivity and soft and hard constraints through a Riemannian metric field. The objective function, which includes trajectory tracking error and obstacle avoidance cost of semantic latent distance features, is minimized through gradient descent. This enables the semantically guided regularized neural motion planner to generate trajectory strategies that conform to semantic logic during the mathematical optimization process, and finally outputs smooth control commands that satisfy dynamic constraints. 5) Closed-loop iteration and adaptive calibration: The predicted trajectory generated by the semantically guided regularized neural motion planner is fed back to the deep unfolded neural network encoder in step 3), the semantic latent distance features are recalculated, and alternating iterative optimization is performed until the final control command is output. At the same time, the deviation between the robot's actual motion execution state and the planned expected state is monitored in real time. Based on the deviation, the network weights of the differentiable parameter generation network in step 2) are updated online, so that the robot can automatically adjust the adaptive navigation constraint parameters of obstacles in actual operation.

2. The end-to-end mobile robot navigation and obstacle avoidance method according to claim 1, characterized in that, In step 1), the specific implementation of obtaining and aligning the multimodal composite feature set includes: using a lightweight visual model to perform inference on the image to obtain a semantic mask. The lightweight visual model uses the YOLO series or SegFormer; for the i-th point cloud acquired by the LiDAR... ,in , , Represents the geometric position coordinates of the i-th point cloud, using the extrinsic parameter matrix. and intrinsic parameter matrix Calculate Projected coordinates on the image ,in and These represent the horizontal and vertical pixel coordinates of the i-th point cloud in the image pixel coordinate system, respectively; index semantic mask. Obtain the semantic category of the i-th point cloud. and classify the semantics Encoded as semantic vectors The augmented input vector is formed by concatenating the original geometric position coordinates with the original input vector. .

3. The end-to-end mobile robot navigation and obstacle avoidance method according to claim 2, characterized in that, In step 1), the specific implementation method for extracting local geometric feature information using the spatial neighborhood relationship of LiDAR point cloud data is as follows: Utilizing the scan line arrangement structure of the LiDAR point cloud data, a sliding window algorithm with linear time complexity is used for feature extraction. First, a topological neighborhood index is constructed for the v-th point cloud on the u-th scan line. Where u is the harness index, v is the horizontal scan index, and the index is in Left and right neighbor point sets on the same scan line ,in , Let and represent the ya and y+a points on the u-th scan line, respectively, where 'a' is the preset window radius. Next, a geometric continuity check is performed. To eliminate interference from abrupt depth changes at object edges, the left and right neighbor point sets are calculated. The Euclidean distance between adjacent points in the middle, if the distance between adjacent points is greater than 1. ,in If a preset breakage threshold is set, a geometric discontinuity is determined, and all points outside the breakage point are removed. If the distance between adjacent points is less than or equal to... If the target point is found to be geometrically continuous, the adjacent point is retained, and a final structure containing the target point is constructed. Maximum contiguous subset Then, based on the tangent vector of local regression, the maximum continuous subset is calculated. Perform local linear regression or principal component analysis, by minimizing Given the sum of squared orthogonal distances from all points to the fitted line, calculate the unit direction vector of that line, defined as the target point. tangent vector This characterizes the local extension direction of the obstacle surface; then, the local geometric feature vector is calculated through vector cross product operation. ,in As the vertical axis vector, the final result is the local geometric feature information representing the repulsion direction.

4. The end-to-end mobile robot navigation and obstacle avoidance method according to claim 3, characterized in that, Step 1) also includes a geometric completion mechanism for negative obstacle hazard areas: a joint determination method of visual edge detection and laser ranging status is used to construct a virtual wall to close the hazard area; when the visual semantic segmentation network identifies areas such as "ground edge", "staircase entrance" or "sunken area" in front, if the laser radar shows signal loss or abrupt change in depth data in the projection direction of these areas, it is determined that there is a negative obstacle in the area and the area is defined as a negative obstacle hazard area; using camera intrinsic parameters and pose information, the edge pixels in the image are back-projected to three-dimensional space to fit a three-dimensional edge baseline located on the plane; starting from the three-dimensional edge baseline, a virtual point cloud array is generated by interpolation according to a preset point cloud density to form a virtual wall that physically closes the negative obstacle hazard area, and the virtual point cloud array is assigned a high-risk semantic label; the generated virtual point cloud array is merged with the original laser radar point cloud data as input for subsequent steps.

5. The end-to-end mobile robot navigation and obstacle avoidance method according to claim 4, characterized in that, In steps 2) and 5), the specific topology and online adaptive update mechanism of the differentiable parameter generation network are as follows: The differentiable parameter generation network adopts a lightweight fully connected neural network structure, including an input layer, several hidden layers, and an output layer; the input layer receives multimodal feature vectors constructed based on a multimodal composite feature set. ,in Let E be the semantic category probability distribution vector, and E be the visual prediction information entropy. This represents the statistics of the local geometric feature vectors. In the output layer, to ensure the mathematical validity of the physical parameters, a Softplus activation function is used to impose a non-negativity constraint on the output, generating a set of positive definite physical impedance parameters. The adaptive navigation constraint parameters mentioned above have the following meanings: Normal repulsion coefficient Used to define the intensity of the repulsive force along the normal direction of the obstacle; tangential sensitivity. , used to define constraint weights along the tangent direction of the obstacle surface; relaxation factor This is used to define the virtual indentation or allowable penetration depth of obstacle geometric boundaries to handle passable soft obstacles. Additionally, this differentiable parameter generation network uses virtual data generated from a pre-set expert rule table for cold-start pre-training, enabling the network to possess basic navigational common sense. Next, the physical consistency loss function is constructed. During robot operation, a physical consistency loss function that measures the "planning-execution" deviation is built in real time. : ； In the formula, The desired velocity vector generated by the regularized neural motion planner. The actual velocity vector fed back by the wheel speed gauge. The vertical vibration amplitude detected by the inertial measurement unit. , The weights are defined as follows: Then, based on the chain rule, end-to-end gradient backpropagation is performed, utilizing the fully differentiable properties of the deep unfolded neural network encoder and the regularized neural motion planner to construct an end-to-end gradient path from the loss function to the network weights W. : ； In the formula, The adaptive navigation constraint parameter set is used, and LDF represents the semantic latent distance feature. By calculating the gradient based on the chain rule, the motion hindrance or terrain bumps encountered by the robot in the physical world are transformed into effects on the network weights. The correction signal is then used; finally, a single-step online calibration is performed, and the network weights are updated online using a single-step stochastic gradient descent strategy. This enables the network to automatically adjust the output impedance parameters based on real-time physical feedback when encountering unknown terrain textures, thereby achieving online system identification and adaptation of environmental physical properties.

6. The end-to-end mobile robot navigation and obstacle avoidance method according to claim 5, characterized in that, In step 3), the specific construction and operation process of the semantic-aware deep unrolled neural network encoder is as follows: The deep unfolded neural network encoder is a deep neural network unfolded based on the iterative steps of the classical optimization algorithm PIBCD to solve the distance minimization problem. It contains K cascaded semantic optimization layers, each corresponding to one gradient descent iteration on the semantic Riemannian manifold space. First, an anisotropy metric matrix is ​​constructed using the normal repulsion coefficient output in step 2). and tangential sensitivity Combined with the local geometric feature vector extracted in step 1), Through the spectral decomposition formula Construct an anisotropy metric matrix M, where The anisotropy, represented by the identity matrix, indicates a spatial metric scale that differentiates between the normal and tangent directions of the obstacle surface. Mathematically, this defines a local Riemannian metric field, causing the shape of the equipotential surfaces around the obstacle to change semantically. Secondly, Riemannian preprocessing gradient updates are performed: in the forward propagation of each layer of the deep unfolded neural network encoder, the original geometric gradient based on Euclidean distance is first calculated. Then, the inverse of the anisotropy metric matrix is ​​used. For the original geometric gradient Left multiplication is performed, followed by Riemannian preprocessing to suppress gradient components in highly repulsive directions and enhance gradient components in less sensitive directions. Finally, a relaxation bias is introduced for state iteration: a relaxation factor is introduced into the gradient update formula. The defined bias term, bias, is used to update the state of the k-th layer as follows: ； In the formula, , These are the semantic latent distance features output from layers k and (k+1), respectively. The network step size parameter, after K layers of iteration, outputs semantic latent distance features that imply semantic physical attributes; Unlike obstacle avoidance strategies based on Euclidean distance, this anisotropic metric matrix M makes the semantic latent distance features generated by the semantic perception deep unfolding neural network encoder exhibit an ellipsoidal equipotential surface in spatial distribution that varies with semantic category. Through this anisotropic constraint mechanism, differentiated spatial constraints and avoidance strategies for obstacles with different attributes are realized.

7. The end-to-end mobile robot navigation and obstacle avoidance method according to claim 6, characterized in that, In step 4), the specific implementation of the semantically guided regularized neural motion planner includes the following: The first step is to construct a differentiable model predictive control optimization problem, defining the prediction time domain as H. The goal of the regularized neural motion planner is to find a set of optimal control sequences at time t. This allows the robot to predict its state sequence at time t. Satisfy kinematic constraints and minimize the total cost function ,in , , These represent the control commands at times t, t+1, and t+H-1, respectively. , , Let represent the predicted states at times t, t+1, and t+H-1, respectively; next, we construct a composite objective function including Riemannian manifold constraints, i.e., the total cost function. The total cost function It consists of three parts, specifically expressed as follows: ； In the formula, To determine the trajectory tracking error, the predicted state at time j is calculated. With global reference path point The Euclidean distance error between them is used to guide the robot to the target point; To control the smoothing term, the control command at time j is calculated. The amplitude and norm of its rate of change are used to limit the robot's maximum speed and acceleration to ensure smooth motion; the third term This is a penalty term for semantic latent distance features. This is a collection of lidar point cloud data. The output of step 3) The semantic latent distance feature value of a point at time j. As a penalty function, a one-sided activation function can be used to map semantic latent distance features to obstacle avoidance costs: ,in, As a safety threshold, since the input semantic latent distance feature value has been distorted by the Riemann metric field, minimizing the third term is equivalent to forcing the robot to automatically find the region with the lowest "semantic cost value", thereby spontaneously generating intelligent behaviors such as "traversing along the edge", "staying away from high-risk areas" or "crossing flexible obstacles". Then, the optimal control sequence is updated by gradient descent or Newton's method. After several iterations and convergence, the first term of the output optimal control sequence can be used as the control command at the current moment and sent to the robot's underlying controller.

Images