A Humanoid Robot Navigation Method Based on Visual Semantic Segmentation and Radar Obstacle Detection

Abstract

This invention belongs to the field of robot navigation technology, specifically relating to a humanoid robot navigation method based on visual semantic segmentation and radar obstacle detection. It utilizes a visual sensor to simultaneously acquire RGB and depth maps of the humanoid robot's current environment, and uses a LiDAR to collect point cloud data. Semantic segmentation is performed on the preprocessed RGB map; radar point cloud obstacle detection is performed; the semantic segmentation map and the LiDAR point cloud map are fused, and Bayesian decision-making is introduced to determine whether passage is possible. The fused cost value is then calculated to obtain a fused cost map; the optimal path is output using the RRT* / TEB algorithm; and a nonlinear optimization solver is used to generate a sequence of landing points accurate to each step. This invention constructs a "semantic-geometric" two-dimensional navigation cost model; it deeply binds Bayesian inference with the navigation scene, improving the adaptability of navigation in unstructured environments; the navigation method is suitable for humanoid robots, with low energy consumption, low navigation deviation, and high safety.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of robot navigation, and particularly relates to a navigation method and system for visual semantic segmentation and radar obstacle detection of a humanoid robot. BACKGROUND

[0002] In current robot navigation technology, the multi-sensor fusion scheme for wheeled robots is relatively mature, but humanoid robots face special challenges in complex terrain environments (such as outdoor unstructured road surfaces and indoor multi-obstacle scenes) due to their complex movement patterns (relying on biped gait adjustment) and higher environmental interaction requirements (such as ascending and descending stairs and avoiding dynamic obstacles). In existing technologies, the navigation scheme for humanoid robots still has problems such as dependence on a single sensor or insufficient targeting of the fusion strategy.

[0003] Typical technical routes include pure visual navigation and pure radar navigation. Among them, the pure visual scheme collects image data through a camera and uses computer vision algorithms to achieve environmental understanding; the pure radar scheme obtains environmental point cloud data through laser radar or millimeter wave radar to complete obstacle detection and three-dimensional modeling. Single-sensor navigation has limitations, pure visual navigation often misjudges "passable areas" due to the lack of height information (such as step drop), leading to robot collisions or falls; pure radar navigation cannot adaptively adjust the gait due to the lack of material identification (such as grass friction coefficient), resulting in navigation energy waste or the risk of slipping. There have been attempts in existing technologies to combine machine vision and laser radar for navigation, but most of them are aimed at the map construction or positioning needs of wheeled robots, and do not fully adapt to the gait control characteristics of humanoid robots. For example: Chinese patent 202311043148.9 discloses a navigation and positioning method based on machine vision and laser radar and a robot, which combines the advantages of machine vision and laser radar, uses a depth camera to obtain environmental depth map and RGB data, then converts the data into laser type data for fusion with the data collected by the laser radar, and finally constructs an environmental two-dimensional grid map; compared with the use of a single sensor, this method can significantly improve the accuracy and completeness of the global map. Chinese patent 202211523195.9 discloses a local semantic passable probability grid map construction method for autonomous mobile robots, which includes: S1: constructing an occupancy grid map according to the measurement value of laser data; S2: performing semantic segmentation using a real-time semantic segmentation model, extracting semantic labels, and performing 2D semantic segmentation; S3: after semantic segmentation, projecting semantic information according to the coordinate relationship using a depth map, i.e., constructing a semantic grid map according to semantic information; S4: after probability conversion of the semantic grid map, the semantic grid map is fused with a multi-layer cost map: the semantic information is converted into a probability through distance mapping to form a semantic layer cost map, which is then fused with an obstacle layer cost map of the laser to obtain the final local semantic passable probability grid map.

[0004] But the prior art does not directly integrate "semantic information (material)" and "geometric information (obstacle height)" into the navigation decision chain, resulting in the inability to determine "whether it is worth detouring" in semi-passable areas (such as light waterlogging), or missing key obstacle points in complex terrain (such as grassland steps) due to insufficient data complementarity, directly affecting navigation safety and path efficiency. SUMMARY

[0005] The purpose of the present application is to overcome the defects in the prior art that the multi-sensor fusion strategy is not well adapted to the characteristics of biped motion when navigating a humanoid robot, and to provide a humanoid robot navigation method based on visual semantic segmentation and radar obstacle detection to achieve high-precision and high-adaptability autonomous navigation in complex environments.

[0006] To achieve the above purpose, the present application provides a humanoid robot navigation method based on visual semantic segmentation and radar obstacle detection, the specific steps comprising:

[0007] Step 1: Environmental perception data acquisition

[0008] Synchronously acquire the RGB image and depth image of the current environment of the humanoid robot using a visual sensor, and collect point cloud data using a laser radar.

[0009] Step 2: Data preprocessing and feature extraction

[0010] 1) Visual data preprocessing and feature extraction

[0011] Correct the distortion and denoise the RGB image.

[0012] Fill in the holes and convert the scale of the depth image, converting the pixel distance to actual physical distance; extract the nearest obstacle point in each column of the depth image, retaining the height feature of the obstacle.

[0013] 2) Laser point cloud preprocessing

[0014] Reduce redundant points and retain key geometric features by voxel filtering the collected point cloud data; then perform ground segmentation and obstacle clustering on the key geometric features, and extract the bounding rectangle parameters of the obstacle.

[0015] Step 3: Visual semantic segmentation and radar obstacle detection

[0016] 1) Visual semantic segmentation: use the SegFormer-B2 network model based on Transformer and known material properties to perform semantic segmentation on the preprocessed RGB image, obtaining a semantic segmentation image with material properties.

[0017] 2) Radar point cloud obstacle detection: According to the parameters of the bounding rectangle of the obstacle obtained in step 2 and the IMU data, the laser-inertial tightly coupled is realized through the LIO-SAM algorithm, the point cloud pose accuracy is optimized, the laser point cloud map is generated, and then the obstacle judgment is carried out to mark "crossable obstacle" or "uncrossable obstacle".

[0018] Step 4: Multi-source information fusion and cost map construction

[0019] Firstly, the RGB image obtained by the visual sensor and the point cloud data collected by the radar are divided into 0.1m×0.1m grid units; then the visual semantic segmentation image and the laser point cloud image of the radar are fused, and the Bayesian decision is introduced to judge whether each grid is passable, and the qualitative labels of "safe path", "cautious path" or "prohibited path" are marked; then the fusion cost value of each grid marked as "safe path" and "cautious path" is calculated to obtain the fusion cost map.

[0020] Step 5: Path planning

[0021] The RRT* algorithm is used for global path planning, and the TEB algorithm is used for local path planning. Through the differentiated obstacle judgment logic (distinguishing between passable obstacles and impassable obstacles), the "qualitative label" and "fusion cost" are combined to output the optimal path.

[0022] Step 6: Footprint sequence generation

[0023] Using a nonlinear optimization solver, the ground material characteristics and obstacle geometry information are combined to generate a precise foot sequence for each step based on the local path, so that the robot can maintain stability and efficiently move in complex environments; Specifically: first, according to the step length corresponding to the current ground material, the local path is equally divided into several segments, and the endpoint of each segment is used as a candidate landing point; then check the obstacle information corresponding to each candidate point, if there is a crossable obstacle in front, adjust the landing point position of this step and set the corresponding lifting height; if an impassable obstacle is encountered, it is marked as needing to detour, and the upper path planning recalculates the local path; after determining the landing point position, the toe direction is calculated according to the turning curvature of the local path, which is generally between 0° and 30° to ensure stability when turning or crossing; the output result is a set of landing point information arranged in time sequence, including the three-dimensional coordinates and toe direction angle of each step, which is directly transmitted to the gait control module of the humanoid robot to realize closed-loop control from path planning to actual walking.

[0024] The specific step of the semantic segmentation of step 3 is: extracting global features through a Transformer encoder, outputting pixel-level semantic labels by a decoder, and dividing the labels into 3 categories according to ground material properties: dry cement ground and hard flat ground are passable areas, marked as "label 0"; grassland, light water accumulation and slopes with a slope of <15° are semi-passable areas, marked as "label 1"; deep pits, stairs and water surfaces are impassable areas, marked as "label 2"; and a semantic segmentation map with labels and material properties is output.

[0025] The criterion for obstacle judgment of step 3 is that when the obstacle height is < the maximum leg lifting height of the robot and the width is < the foot spacing, it is marked as "crossable obstacle"; otherwise, it is marked as "uncrossable obstacle".

[0026] The judgment criterion of the Bayesian decision in step 4 is: if the visual semantic segmentation map is marked as "label 0" and the radar laser point cloud map is marked as "crossable obstacle", it is determined as "safe path"; if the visual semantic segmentation map is marked as "label 1" or the radar laser point cloud map is marked as "crossable obstacle", it is determined as "cautious path"; if the visual semantic segmentation map is marked as "label 2" or the radar laser point cloud map is marked as "uncrossable obstacle", it is determined as "prohibited path".

[0027] The calculation method of the fusion cost value in step 4 is: for the "safe path" and "cautious path" filtered by the Bayesian decision, the semantic cost value and the geometric cost value of each grid unit are calculated, and then the fusion cost value is calculated, specifically: first, the semantic cost value is calculated according to the semantic segmentation map: the semantic cost value of the passable area is 0, the semantic cost value of the semi-passable area is 50, and the semantic cost value of the impassable area is 100; then the geometric cost value is calculated, based on the straight line distance distance between the current position of the robot and the obstacle on the laser point cloud map, the closer the distance, the higher the cost value, and the calculation formula of the geometric cost value cost is: .

[0028] The calculation formula of the fusion cost value total_cost is: , wherein, is the semantic cost weight coefficient.

[0029] The input of the nonlinear optimization solver in step 6 of the application includes three parts: a local path, a ground material, and an obstacle attribute, the local path is obtained from the TEB algorithm local path optimization result of the previous step, the path is smoothed and meets the obstacle avoidance constraint, but has not been refined to a specific landing point; the ground material is obtained from the visual semantic segmentation of step 3, the material categories include cement ground, grassland, accumulated water and the like, different materials will affect the step length, step frequency and gait of the robot; the obstacle attribute comes from the laser point cloud map, the height, width and position of the obstacle can be accurately obtained, and the depth map is used to assist the measurement of close-range obstacles when necessary, the obstacle can be directly crossed when the height is less than the maximum leg lifting height of the robot and the width does not exceed the foot spacing, otherwise it needs to be avoided at the path level.

[0030] The application directly integrates semantic information (material) and geometric information (obstacle height) into the navigation decision chain, realizes real-time planning of the path according to the map, and generates a footprint for the robot through the path, realizing closed-loop control of the path and the footprint.

[0031] Compared with the prior art, the application has the following beneficial effects:

[0032] (1) In the information dimension, the existing technology mainly uses wheeled robots, and the map is only single-dimensional, while the cost map (special for humanoid robots) of the application is a two-dimensional deep fusion: it contains "semantic information (ground material, friction coefficient, passing level label 0 / 1 / 2)" and "geometric information (obstacle distance, height, crossability)", which are bound through the cost formula and Bayesian decision. From "single environment description" to "decision basis suitable for biped motion", the core needs of humanoid robots considering "material safety (slip, energy consumption)" and "obstacle crossing (step length, leg lifting height)" are solved.

[0033] (2) In the functional positioning, the existing technology serves "map construction / positioning", and the target is to improve the map accuracy (such as patent 202311043148.9 emphasizing "global map integrity"), which is not directly related to navigation decision and motion control; while the application serves "navigation decision + gait control": through the grid cost (semantic cost + geometric cost), it directly outputs "safe / cautious / forbidden" path label, and provides parameters for subsequent footprint generation and gait adjustment (such as material -> step frequency mapping). Skip the intermediate conversion link of "map -> decision", realize the end-to-end connection of "perception - planning - execution", and avoid the problem of "high map accuracy but disconnected from humanoid motion" in the prior art (such as navigation interruption in complex terrain and energy waste).

[0034] (3) In terms of environmental adaptability, the prior art map-adaptive wheeled robot "moves on a plane" and does not consider the influence of unstructured environments (grass, water accumulation, steps) on biped gait, can only determine "whether it is passable", and cannot determine "whether it can cross / how to adjust the gait", while the adaptive humanoid robot of the present application "moves on two feet": for semi-passable areas (such as light water accumulation) and crossable obstacles (such as a 10cm curb), the "small step exploration" "leg lifting crossing" strategy is given through Bayesian decision, and the safety and efficiency are balanced by combining the cost model. The defects of the prior art in unstructured environments, such as "missing key obstacle points" and "unable to determine whether it is worth detouring", are solved, and the navigation adaptability of the humanoid robot in complex scenes such as outdoor grass and indoor steps is improved.

[0035] In summary, the present application constructs a "semantic-geometric" two-dimensional navigation cost model, which directly converts sensor data into navigation decision factors of "whether it is passable" and "whether it can cross"; Bayesian inference is bound with the depth of the navigation scene (such as semi-passable areas prefer to choose the "small step exploration" strategy), which improves the navigation adaptability in unstructured environments; end-to-end optimization from path planning to gait control: through the "material-gait" mapping table (such as concrete ground -> fast step frequency), the navigation energy consumption is reduced, and at the same time the navigation deviation is reduced when crossing obstacles; nonlinear QP optimization directly serves the navigation safety, ensuring that the footprint point is always located in the passable area, avoiding the risk of stepping out during navigation. The navigation method of the present application is suitable for humanoid robots, has high environmental adaptability, low energy consumption, low navigation deviation and high safety. BRIEF DESCRIPTION OF DRAWINGS

[0036] Figure 1 The present application relates to a humanoid robot navigation method based on visual semantic segmentation and radar obstacle detection.

[0037] Figure 2 The present application relates to a humanoid robot navigation method based on visual semantic segmentation and radar obstacle detection. DETAILED DESCRIPTION

[0038] The present application will be further described in detail below through specific embodiments and in conjunction with the drawings.

[0039] Example 1:

[0040] The present embodiment relates to a humanoid robot navigation method based on visual semantic segmentation and radar obstacle detection, and the specific steps include:

[0041] Step 1: Environmental perception data acquisition

[0042] The RGB image and depth image of the current environment of the humanoid robot are synchronously acquired by using a visual sensor, and point cloud data is collected by using a laser radar, and the specific steps are as follows:

[0043] 1) Visual sensor data acquisition

[0044] The RGB image and depth image of the environment are synchronously acquired by using a visual sensor (such as a depth camera Intel RealSense), the RGB image is used for subsequent semantic segmentation, the resolution is set to 1280×720, the frame rate is 30fps, and the ground material (such as grass, cement ground, and accumulated water) and obstacle appearance features are collected; the depth image is used to extract three-dimensional spatial information, and pixel-level distance data are acquired through the built-in infrared sensor, the range is 0.5-10m, and the height, contour and other geometric features of the obstacle are reserved.

[0045] 2) Laser radar data acquisition

[0046] Point cloud data is collected by using a 16-line laser radar (such as Velodyne VLP-16), the scanning frequency is 10Hz, the horizontal field of view is 360°, the vertical field of view is ±15°, and the point cloud density is ≥200 points / ㎡; three-dimensional coordinates (X, Y, Z) and reflection intensity of the detected target are obtained, which are used for obstacle positioning and terrain modeling.

[0047] Step 2: Data preprocessing and feature extraction

[0048] 1) Visual data preprocessing and feature extraction

[0049] The RGB image is corrected for distortion (based on the camera intrinsic matrix) and denoised (Gaussian filter, σ=1.0).

[0050] The depth image is filled with holes (using the neighborhood interpolation method) and scaled, and the pixel distance is converted into actual physical distance (unit: meter).

[0051] The nearest obstacle point in each column of the depth image is extracted (the point closest to the imaging plane in each column is taken as the nearest obstacle detected by the visual sensor), and the height feature of the obstacle is reserved (the extraction formula: (d_i) = min(r_{i1}, r_{i2},..., r_{ij}) j=1, 2, 3…N; where (d_i) is the distance of the nearest obstacle in the i-th column from the imaging plane, and r_{ij} is the height information of the j-th pixel in the column).

[0052] 2) Laser point cloud preprocessing

[0053] The collected point cloud data is filtered by voxel filtering (voxel size 0.1m x 0.1m x 0.1m) to reduce redundant points and retain key geometric features; then the key geometric features are ground segmented and obstacle clustered.

[0054] Ground segmentation: RANSAC algorithm is used to fit the ground plane (plane equation: Ax + By + Cz + D = 0) to separate ground points from obstacle points.

[0055] Obstacle clustering: DBSCAN algorithm (neighborhood radius 0.3m, minimum point number 5) is used to extract the bounding rectangle parameters (length, width, height) of the obstacle.

[0056] Step 3: Visual semantic segmentation and radar obstacle detection

[0057] 1) Visual semantic segmentation: SegFormer-B2 (SOTA) network model based on Transformer is used to perform semantic segmentation on the preprocessed RGB image with known material properties, and a semantic segmentation image with material properties is obtained, which is: global features are extracted by the Transformer encoder, and pixel-level semantic labels are output by the decoder, and the labels are divided into 3 categories according to the ground material properties: dry cement ground and hard flat ground (confidence ≥0.9) are marked as "label 0"; grassland, light water and slope <15° (confidence 0.5-0.9) are marked as "label 1"; deep pit, stairs and water surface (confidence ≥0.8) are marked as "label 2"; the output semantic segmentation image (with the same resolution as the RGB image) with labels and material properties (such as friction coefficient: cement ground 0.8, grassland 0.6, water 0.3).

[0058] 2) Radar point cloud obstacle detection: according to the bounding rectangle parameters of the obstacle obtained in step 2 and the IMU data (acceleration, angular velocity), laser-inertial tightly coupled is realized by LIO-SAM algorithm, the point cloud pose accuracy is optimized (error ≤2cm), laser point cloud map is generated, and then obstacle judgment is performed, marked as "crossable obstacle" or "non-crossable obstacle"; the IMU is an inertial measurement unit carried by the robot, used to obtain the motion state of the robot in real time.

[0059] Obstacle judgment: when the obstacle height < robot maximum leg lifting height (such as 15cm) and width < foot spacing (such as 20cm), it is marked as "crossable obstacle"; otherwise, it is marked as "non-crossable obstacle", and its three-dimensional coordinates and contour information are output.

[0060] Step 4: Multi-source information fusion and cost map construction

[0061] First, the RGB image obtained by the current visual sensor and the point cloud data collected by the radar are divided into 0.1m×0.1m grid units; then the visual semantic segmentation image and the laser point cloud image of the radar are fused, and the Bayesian decision is introduced to judge whether each grid is passable, and the fusion cost of each grid is calculated to obtain the fusion cost map, which is:

[0062] 1) Semantic - point cloud map feature fusion:

[0063] The visual semantic segmentation image and the laser point cloud map obtained in step 3 are converted to the same robot coordinate system through the external parameter matrix (rotation matrix R and translation vector T of the camera and the radar), and the "semantic-geometric fusion map" is obtained after fusion. Then the Bayesian decision is introduced to judge the "passability" of each grid / path segment, and the qualitative labels of "safe / cautious / forbidden" are output.

[0064] The "semantic-geometric fusion map" is based on the robot coordinate system, and the "semantic attribute information" (such as ground material, pass level label) output by the visual sensor and the "geometric space information" (such as obstacle three-dimensional coordinates, terrain height) output by the laser radar are aligned at pixel level / grid level, forming an integrated environment model that "each spatial location contains semantic label and geometric parameter". For example: a certain grid unit (0.1m×0.1m) not only marks "label 1 (semi-passable area, grass material, friction coefficient 0.6)", but also marks "the nearest obstacle distance is 0.8m, the obstacle height is less than 15cm, and the obstacle can be crossed", providing complete data support for subsequent Bayesian decision and cost calculation.

[0065] Bayesian decision: if the visual semantic segmentation image is marked with "label 0" and the radar laser cloud point map is marked with "obstacle can be crossed", it is determined as "safe path"; if the visual semantic segmentation image is marked with "label 1" or the radar laser cloud point map is marked with "obstacle can be crossed", it is determined as "cautious path"; if the visual semantic segmentation image is marked with "label 2" or the radar laser cloud point map is marked with "obstacle cannot be crossed", it is determined as "forbidden path".

[0066] 2) Fusion cost map construction:

[0067] The "safe path" and "cautious path" selected by the Bayesian decision are calculated for the semantic cost and geometric cost of each grid unit, and the fusion cost is calculated to obtain the fusion cost map.

[0068] Semantic cost calculation according to semantic segmentation map: the semantic cost of passable area is 0, the semantic cost of semi-passable area is 50, and the semantic cost of impassable area is 100.

[0069] Geometric cost is used to quantify the obstacle avoidance risk of the area, which is based on the straight-line distance distance between the current position of the robot and the obstacle on the laser point cloud map. The closer the distance, the higher the cost. The calculation formula of geometric cost cost is: .

[0070] Total cost is used to quantify the environmental risk. According to the size of the total cost, the risk degree is quantified and sorted. The larger the total cost, the greater the risk degree. The calculation formula is: .

[0071] wherein, a is the semantic cost weight coefficient, which is an adjustable parameter that dynamically adapts to the scene, and is an adjustable parameter that comprehensively considers semantic labels and obstacle markers, with a value range of 0-1. For example: when the semantic label is 0 (passable area), if there is no obstacle, i.e. the obstacle avoidance demand is low, a can be taken as 0.3-0.4, and if the obstacle avoidance demand is high, a can be taken as 0.1-0.2; when the semantic label is 1 (semi-passable area), if the semantic risk is dominant (such as grassland), a is taken as 0.6-0.7, and if the obstacle avoidance is dominant, a is taken as 0.4-0.5; when the semantic label is 2 (impassable area), a is taken as 1, and the total cost is 100, which is the maximum value. The total cost between 0 and 50 can be defined as low risk, the total cost between 50 and 70 can be defined as medium risk, and the total cost between 70 and 100 can be defined as high risk.

[0072] Step 5: path planning

[0073] The RRT* algorithm is used for global path planning, and the TEB algorithm is used for local path planning. The optimal path is output by combining the differentiated obstacle judgment logic (distinguishing between passable obstacles and impassable obstacles) with the "semantic label" and "total cost". Specifically, it includes:

[0074] 1. Global path planning (based on RRT* algorithm)

[0075] Input: total cost map (get semantic label and total cost information, etc.), start point and target point coordinates.

[0076] Constraints: path distance to impassable area ≥ 0.5m, cumulative turn angle ≤ 180°; passable obstacles (e.g. low curb ≤ 5cm, grass depression) are allowed to be crossed, but marked as "gait adjustment area"; cumulative turn angle ≤ 180°; target: minimum "fusion cost sum".

[0077] Output: global optimal path (a series of waypoints, 1m interval, with passable attribute labels).

[0078] 2. Local path optimization (based on TEB algorithm)

[0079] The core of local path planning is to deal with real-time dynamic obstacles (such as sudden pedestrians). Based on the global path and real-time radar scanning data, the fusion cost increment of the grid around the dynamic obstacle is calculated, the local path is finely adjusted, and the corresponding gait is triggered according to the "risk level" of the fusion cost; the gait is a small step trial, which is triggered in the global path with medium risk.

[0080] The input of TEB algorithm is global path (with passable attribute) and robot kinematics model (e.g. maximum speed 1m / s, maximum acceleration 0.5m / s²); the robot kinematics model refers to the hardware parameters of the robot itself.

[0081] Optimization goal: path smoothness (curvature ≤ 0.5rad / m), obstacle avoidance safety (distance to obstacle ≥ 0.3m), gait adaptability (preferably select passable areas matching the robot's step length).

[0082] The output of TEB algorithm is: local fine path (timestamp + position coordinates + gait adjustment instruction), the timestamp is the time when the robot is expected to arrive at the path point, and the position coordinates are the spatial position of the fine path.

[0083] For example: in the area of grass (semantic label 1, semantic cost 50) and distance from obstacle 0.5m (geometric cost = 100 / (0.5+1)≈66.7), if a=0.6, fusion cost = 0.6x50 + 0.4x66.7≈56.7, it will be determined as "medium risk section in cautious path"; when global path planning, it will prefer to avoid this area and choose a safe path around it (area with fusion cost less than 50), if there is a path after avoiding that has a greater fusion cost, it will be included in the options. When local path planning, assuming that the global planning has chosen a "safe path", but when the robot is running, the laser radar suddenly detects a dynamic obstacle (such as a pedestrian walking into the original safe path), then the geometric cost increment around the dynamic obstacle needs to be recalculated, combined with the original semantic cost to recalculate the new fusion cost, and generate a local adjustment path; if the global planning has chosen the above-mentioned grass, then generate "small step exploration" gait instructions, such as adjusting the step length, reducing the risk of slipping at each step, etc.

[0084] Step 6: Footprint sequence generation (based on nonlinear QP optimization)

[0085] Using a nonlinear optimization solver, the ground material characteristics and obstacle geometry information are combined on the basis of the local path to generate a precise footprint sequence for each step, so that the robot can maintain stability and efficiently move forward in complex environments; first, according to the step length corresponding to the current ground material, the local path is equally divided into several segments, and the endpoint of each segment is used as a candidate landing point; then check the obstacle information corresponding to each candidate point, if there is a crossable obstacle in front, adjust the landing point position of this step and set the corresponding lifting height; if an un-crossable obstacle is encountered, it is marked as needing to be bypassed, and the local path is recalculated by the upper path planning; after determining the landing point position, the toe direction is calculated according to the turning curvature of the local path, generally within 0° to 30°, to ensure stability when turning or crossing.

[0086] The input of the nonlinear optimization solver includes three parts: the local path, the ground material, and the obstacle attribute. The local path is obtained from the local path optimization result of the TEB algorithm in the previous step, the path is smoothed and meets the obstacle avoidance constraint, but has not been refined to specific landing points. The ground material is obtained from the visual semantic segmentation in step 3, the material categories include cement ground, grassland, accumulated water, etc. Different materials will affect the robot's step length, step frequency and gait, for example, the cement ground is flat and has large friction, so it can use 0.5m step length and increase the step frequency; the grassland has small friction and is soft, so the step length is shortened to 0.3m and the step frequency is appropriately reduced; the accumulated water area has low friction coefficient, so the step length can be further shortened to 0.25m and the center of gravity is reduced to prevent slipping; the obstacle attribute comes from the laser point cloud map, which can accurately obtain the height, width and position of the obstacle. If necessary, the depth map is used to assist in measuring close-range obstacles. When the obstacle height is less than the maximum leg lifting height of the robot and the width does not exceed the foot spacing, it can be directly crossed, otherwise it needs to be avoided at the path level. The input of the nonlinear optimization solver also includes the kinematic parameters of the robot's legs, including the inherent parameters of the robot's hardware (such as leg link length, joint range of motion, maximum step length and maximum leg lifting height) and real-time sensor parameters (including joint angle, etc.). The sensors include joint encoders, six-axis force sensors, IMU inertial measurement units, etc.

[0087] The optimization objective is , and the constructed minimization objective function is:

[0088]

[0089]

[0090] wherein: footprint sequence, each is the three-dimensional coordinates of the landing point ; : the reference path point coordinates on the local path corresponding to the kth landing point; : the expected step length corresponding to the current ground material (determined by the built-in material-gait mapping table, for example, cement ground 0.5m, grassland 0.3m); : the obstacle height corresponding to the landing point position (from laser point cloud or depth map); : the maximum crossable height of the robot; : the toe pointing angle of the kth step; : weight coefficient, used to balance the importance of different objectives (for example, path following is more important than energy consumption, so ).

[0091] The optimization solution uses a nonlinear QP solver. Under the constraints of gait stability, obstacle avoidance safety, and energy minimization, the final footprint sequence is output. The output result is a set of foot placement information arranged in chronological order, including the three-dimensional coordinates (X,Y,Z) of each step and the toe angle (0° to 30°). This result will be directly transmitted to the gait control module of the humanoid robot to realize closed-loop control from path planning to actual walking.

[0092] Example 2:

[0093] This embodiment is an experiment demonstrating the effectiveness of the method described in Embodiment 1. Figure 2 As shown in Figure A, the result of conventional path planning is presented by a blue dashed line. Starting from the marked "starting point", the path needs to take a long detour to avoid the marked "obstacles" and "puddles". The overall route is relatively tortuous, and in complex areas, it is necessary to take a roundabout approach to reach the "target point".

[0094] Figure B shows the path planned by the navigation method of this invention, marked with footsteps. The humanoid robot can use its ability to cross obstacles to plan a path more directly by stepping when encountering gravel roads, "obstacles", and "puddles". Compared with the conventional path on the left, the route is more efficient, simpler and more in line with the movement characteristics of the humanoid robot.

Claims

1. A humanoid robot navigation method based on visual semantic segmentation and radar obstacle detection, characterized in that, The specific steps include: Step 1: Environmental perception data acquisition Synchronously acquire the RGB image and depth image of the current environment of the humanoid robot using a visual sensor, and acquire point cloud data using a laser radar; Step 2: Data preprocessing and feature extraction 1) Visual data preprocessing and feature extraction Correct distortion and denoise the RGB image; Fill in the holes of the depth image and convert the pixel distance to the actual physical distance; extract the nearest obstacle point in each column of the depth image and retain the height feature of the obstacle; 2) Laser point cloud preprocessing and denoising Reduce redundant points and retain key geometric features by voxel filtering on the collected point cloud data; then perform ground segmentation and obstacle clustering on the key geometric features, and extract the bounding rectangle parameters of the obstacle; Step 3: Visual semantic segmentation and radar obstacle detection 1) Visual semantic segmentation: use the SegFormer-B2 network model based on transformer and known material properties to perform semantic segmentation on the preprocessed RGB image, obtaining a semantic segmentation image with material properties; 2) Radar point cloud obstacle detection: according to the bounding rectangle parameters of the obstacle obtained in step 2 and the IMU data, realize laser-inertial tightly coupled through LIO-SAM algorithm, optimize the point cloud pose accuracy, generate laser point cloud map, and then perform obstacle judgment, marking "crossable obstacle" or "uncrossable obstacle"; Step 4: Multi-source information fusion and cost map construction Divide the RGB image obtained by the visual sensor and the point cloud data collected by the radar into 0.1m×0.1m grid units; then fuse the visual semantic segmentation image and the laser point cloud image, introduce Bayesian decision to judge whether each grid is passable, and mark "safe path", "cautious path" or "prohibited path" qualitative labels; calculate the fusion cost value of each grid marked as "safe path" and "cautious path", and obtain the fusion cost map; Step 5: Path planning Use RRT* algorithm for global path planning and TEB algorithm for local path planning, and output the optimal path through differentiated obstacle judgment logic combined with "semantic label" and "fusion cost"; Step 6: Footprint sequence generation The nonlinear optimization solver is used to generate a landing point sequence accurate to each step on the basis of the local path, combined with the ground material properties and obstacle geometry information, so that the robot can maintain stability and efficiently advance in complex environments; Specifically, first, the local path is equally divided into several segments according to the step length corresponding to the current ground material, and the endpoint of each segment is taken as a candidate landing point; Then check the obstacle information corresponding to each candidate point, if there is a crossable obstacle in front, adjust the landing point position of this step and set the corresponding leg lifting height; If it encounters an un-crossable obstacle, it is marked as needing to detour, and the local path is recalculated by the upper path planning; After determining the landing point position, the toe direction is calculated according to the turning curvature of the local path, generally within 0° to 30°, to ensure stability when turning or crossing; The output result is a set of landing point information arranged in time sequence, including the three-dimensional coordinates and toe direction angle of each step, which is directly transmitted to the gait control module of the humanoid robot to realize closed-loop control from path planning to actual walking.

2. The visual semantic segmentation and radar obstacle detection based humanoid robot navigation method according to claim 1, characterized in that, The specific steps of semantic segmentation in step 3 are: extracting global features through an encoder, and outputting pixel-level semantic labels through a decoder, and according to the ground material properties, the labels are divided into 3 categories: dry cement ground and hard flat ground are passable areas, marked as "label 0"; grassland, light water accumulation and slopes with a slope of <15° are semi-passable areas, marked as "label 1"; deep pits, stairs and water surfaces are impassable areas, marked as "label 2"; and a semantic segmentation map with labels and material properties is output. 3.The visual semantic segmentation and radar obstacle detection based humanoid robot navigation method of claim 1, wherein, The standard for obstacle judgment in step 3 is that when the obstacle height < the maximum leg lifting height of the robot and the width < the foot spacing, it is marked as "crossable obstacle"; otherwise, it is marked as "uncrossable obstacle".

4. The visual semantic segmentation and radar obstacle detection based humanoid robot navigation method of claim 1, wherein, The judgment criteria for Bayesian decision in step 4 are: If the visual semantic segmentation map is marked as "label 0" and the radar laser cloud point map is marked as "crossable obstacle", it is determined as "safe path"; If the visual semantic segmentation map is marked as "label 1" or the radar laser cloud point map is marked as "crossable obstacle", it is determined as "cautious path"; If the visual semantic segmentation map is marked as "label 2" or the radar laser cloud point map is marked as "uncrossable obstacle", it is determined as "prohibited path".

5. The visual semantic segmentation and radar obstacle detection based humanoid robot navigation method of claim 1, wherein, The calculation method of the fusion cost value in step 4 is: for the "safe path" and "cautious path" filtered by Bayesian decision, the semantic cost value and geometric cost value of each grid unit are calculated, and then the fusion cost value is calculated, specifically: first, the semantic cost value is calculated according to the semantic segmentation map: the semantic cost value of the passable area is 0, the semantic cost value of the semi-passable area is 50, and the semantic cost value of the impassable area is 100; Then calculate the geometric cost value, based on the straight line distance distance between the current position of the robot and the obstacle on the laser point cloud map, the closer the distance, the higher the cost value, the calculation formula of the geometric cost value cost is: ； The calculation formula of the fusion cost value total_cost is: ； wherein, is a semantic cost weight coefficient. 6.The visual semantic segmentation and radar obstacle detection based humanoid robot navigation method of claim 1, wherein, The input of the nonlinear optimization solver in step 6 includes three parts: the local path, the ground material and the obstacle attribute. The local path is obtained by the local path optimization result of the TEB algorithm in the last step, the path is smoothed and meets the obstacle avoidance constraint, but has not been refined to the specific landing point. The ground material is obtained by the visual semantic segmentation in step 3, the material categories include cement ground, grassland and accumulated water, and different materials will affect the step length, step frequency and gait of the robot. The obstacle attribute comes from the laser point cloud map, which can accurately obtain the height, width and position of the obstacle, and the depth map is used to assist the measurement of close-range obstacles. When the height of the obstacle is less than the maximum leg-lifting height of the robot and the width does not exceed the foot spacing, it can be directly crossed, otherwise it needs to be avoided at the path level.

Images