Robot automatic mapping method and system based on RTK and visual perception

Abstract

The invention relates to the technical field of robot autonomous navigation, in particular to a robot automatic mapping method and system based on RTK and visual perception, and the method comprises the steps: obtaining RTK data, a laser radar point cloud and a visual perception image; segmenting the visual perception image by using a semantic segmentation model to determine contour points in the road surface region; contour points in the road surface area are converted into visual detection road surface point clouds under a laser radar coordinate system; and fusing the visual detection road surface point cloud with the laser radar point cloud and RTK data to complete map construction. According to the scheme of the invention, full-automatic, high-precision and road-oriented mapping of an outdoor scene is realized, the labor cost is remarkably reduced, and the efficiency is improved.

Description

Technical Field

[0001] The present invention generally relates to the field of robot autonomous navigation technology. More specifically, the present invention relates to a robot automatic mapping method and system based on RTK and visual perception. Background Art

[0002] Terms like humanoid robots, robot dogs, and embodied intelligence are frequently mentioned. Robotics research and development is experiencing an unprecedented surge in growth, accelerating the transition from the laboratory to diverse applications in industry, healthcare, and the home. Autonomous navigation robots integrate advanced technology and intelligence, enabling them to autonomously perform tasks like cleaning and navigation.

[0003] However, there are still certain gaps in the domestic robotics field: researchers in the fields of SLAM, motion planning, control, robotic arms, etc. have relatively complex academic backgrounds, most of them have not systematically studied the basic knowledge of robotics, and most of them are "groping forward."

[0004] Currently, autonomous navigation robots require high-precision mapping of their target areas during operation. Current methods for map construction primarily include manual remote control mapping, laser SLAM mapping, and RTK+laser integrated mapping. However, each of these methods has significant drawbacks.

[0005] (1) Manual remote control mapping: The operator manually controls the robot to move and collect data. This method is inefficient and relies on experience, has high labor costs, and cannot be operated remotely. In addition, complex areas are not fully covered.

[0006] (2) Pure laser SLAM mapping: This relies on autonomous exploration using LiDAR, but lacks global coordinate alignment capabilities, and dynamic obstacles can easily cause map fragmentation. Accumulated errors lead to map distortion, requiring manual correction later. Furthermore, the lack of road semantic information results in low path planning efficiency.

[0007] (3) RTK+Laser Mapping: This method combines real-time dynamic positioning and laser SLAM mapping to achieve high-precision map construction. However, this method has difficulty determining road boundaries and cannot automatically scan along the road. In addition, RTK is not reliable enough under multipath interference, making it impossible to achieve effective road-oriented mapping.

[0008] Based on this, how to solve the current problem of poor accuracy and low efficiency in mapping in outdoor scenes is one of the key directions of current autonomous navigation research. Summary of the Invention

[0009] To address the technical issues of poor accuracy and low efficiency in mapping in outdoor scenes, the present invention provides solutions in the following aspects.

[0010] In a first aspect, the present invention provides a robot automatic mapping method based on RTK and visual perception, comprising: acquiring RTK data, a lidar point cloud, and a visual perception image; segmenting the visual perception image using a semantic segmentation model to determine contour points in a road surface area; converting the contour points in the road surface area into a visually detected road surface point cloud in a lidar coordinate system; and fusing the visually detected road surface point cloud with the lidar point cloud and RTK data to complete map construction.

[0011] Preferably, the contour points in the road surface area are converted into a visually detected road surface point cloud in a lidar coordinate system, including: converting the pixel coordinates of the contour points in the road surface area into 3D points in a lidar coordinate system through a calibration matrix; and generating a visually detected road surface point cloud in a lidar coordinate system.

[0012] Preferably, before fusing the visually detected road surface point cloud with the lidar point cloud data, the method further includes: performing point cloud verification and correction on the ground point cloud in the lidar point cloud and the visually detected road surface point cloud.

[0013] Preferably, the verification process includes: determining the degree of overlap between the laser radar point cloud and the road surface area detected in the visual detection road surface point cloud; in response to the overlap being greater than a first set value, determining it to be reliable; in response to the overlap being less than the first set value and greater than a second set value, using the laser radar point cloud as the main component and the visual detection road surface point cloud to supplement details; in response to the overlap being less than the second set value, triggering an alarm.

[0014] Preferably, the correction process includes: in response to the visual perception image being recognized as a road surface and the lidar point cloud being recognized as an obstacle, the lidar data recognition result shall prevail; in response to the lidar point cloud being missing and the visual perception image being recognized as a road surface, the visual perception image recognition result shall prevail.

[0015] Preferably, the visually detected road surface point cloud is integrated with the lidar point cloud and RTK data to complete the map construction, including: generating a 3D point cloud map using the laser point cloud, and superimposing the visually detected road surface point cloud onto the grid layer; performing global coordinate detection based on the RTK data to correct the map offset.

[0016] Preferably, it also includes: using the laser radar point cloud to determine the reliability of RTK to reduce multipath interference.

[0017] Preferably, the RTK reliability is determined using a laser radar point cloud, including: obtaining laser point cloud and RTK positioning data; determining the occlusion area in the laser point cloud using a multi-path error detection algorithm to calculate the RTK reliability; and fusing the laser point cloud and RTK positioning data using a multi-layer sensor fusion architecture with dynamic weight allocation to achieve robot navigation, wherein the fusion weight is determined using a multi-level weight decision model, and the basic weight in the multi-level weight decision model is determined according to the level corresponding to the RTK reliability.

[0018] Preferably, a multi-level weight decision model is used to determine the dynamic weight, including: high-level decision-making determines the basic weight through RTK reliability, and the RTK reliability is determined according to the size of the occluded area in the laser point cloud; the middle-level correction obtains the RTK weight through dynamic correction of the point cloud density, wherein the RTK weight is negatively correlated with the point cloud density of the laser point cloud; the bottom-level compensation compensates the Kalman filter through high-frequency injection of IMU / odometer.

[0019] In a second aspect, the present invention also provides a robot automatic mapping system based on RTK and visual perception, comprising: a processor; a memory, wherein the memory stores computer program instructions, and when the computer program instructions are executed by the processor, a robot navigation method based on RTK and lidar as described in one or more of the above embodiments is implemented.

[0020] The beneficial effects of the present invention are as follows: according to the solution of the present invention, by converting the contour points in the road surface area into a visually detected road surface point cloud in the lidar coordinate system; fusing the visually detected road surface point cloud with the lidar point cloud and RTK data, and coordinating the visual perception of road semantics with RTK global positioning, fully automatic, high-precision, road-oriented mapping of outdoor scenes is achieved, significantly reducing labor costs and improving efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The above and other objects, features and advantages of the exemplary embodiments of the present invention will become readily understood by reading the following detailed description with reference to the accompanying drawings. In the accompanying drawings, several embodiments of the present invention are shown in an illustrative and non-limiting manner, and the same or corresponding reference numerals represent the same or corresponding parts, wherein:

[0022] Figure 1 is a flowchart illustrating a method for automatic robot mapping based on RTK and visual perception according to an embodiment of the present invention;

[0023] Figure 2 3 is a block diagram showing a structure of a robot automatic mapping system based on RTK and visual perception according to an embodiment of the present invention. DETAILED DESCRIPTION

[0024] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work shall fall within the scope of protection of the present invention.

[0025] The specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0026] Figure 1 1 is a flow chart illustrating a method 100 for automatic robot mapping based on RTK and visual perception according to an embodiment of the present invention.

[0027] like Figure 1 As shown, in step S101, RTK data, lidar point cloud and visual perception image are obtained.

[0028] In step S102 , the visual perception image is segmented using a semantic segmentation model to determine contour points in the road surface area.

[0029] At step S103, the contour points in the road surface area are converted into a visually detected road surface point cloud in a LiDAR coordinate system. In some embodiments, the pixel coordinates of the contour points in the road surface area are converted into 3D points in a LiDAR coordinate system using a calibration matrix, thereby generating a visually detected road surface point cloud in the LiDAR coordinate system.

[0030] Furthermore, before fusing the visually detected road surface point cloud with the lidar point cloud data, point cloud verification and correction can be performed on the ground point cloud in the lidar point cloud and the visually detected road surface point cloud.

[0031] The verification process includes:

[0032] The system determines the degree of overlap between the LiDAR point cloud and the road surface area detected by the visual inspection point cloud. If the overlap is greater than a first set value, the system determines the area is reliable. If the overlap is less than the first set value and greater than a second set value, the LiDAR point cloud is used as the primary focus, and the visual inspection road surface point cloud is used to supplement the details. If the overlap is less than the second set value, an alarm is triggered.

[0033] The correction process includes: in response to the visual perception image being recognized as a road surface and the lidar point cloud being recognized as an obstacle, the lidar data recognition result shall prevail; in response to the lack of the lidar point cloud and the visual perception image being recognized as a road surface, the visual perception image recognition result shall prevail.

[0034] At step S104, the visually detected road surface point cloud is fused with the LiDAR point cloud and RTK data to complete the map construction. In some embodiments, the LiDAR point cloud is used to generate a 3D point cloud map, and the visually detected road surface point cloud is superimposed on a grid layer; global coordinate detection is performed based on the RTK data to correct for map offsets.

[0035] Furthermore, it also includes using lidar point clouds to determine RTK reliability to reduce multipath interference.

[0036] Specifically, the RTK reliability is determined using the lidar point cloud, including obtaining the laser point cloud and RTK positioning data; the occlusion area in the laser point cloud is determined using a multi-path error detection algorithm to calculate the RTK reliability; and the laser point cloud and RTK positioning data are fused using a multi-layer sensor fusion architecture with dynamic weight allocation to achieve robot navigation. The fusion weight is determined using a multi-level weight decision model, and the basic weight in the multi-level weight decision model is determined according to the level corresponding to the RTK reliability.

[0037] In some embodiments, when a multi-level weight decision model is used to determine dynamic weights, the high-level decision determines the basic weight through RTK reliability, and the RTK reliability is determined according to the size of the occluded area in the laser point cloud; the middle-level correction obtains the RTK weight through dynamic correction of the point cloud density, where the RTK weight is negatively correlated with the point cloud density of the laser point cloud; the bottom-level compensation compensates the Kalman filter through high-frequency injection of the IMU / odometer.

[0038] Next, the solution of the present invention will be described in detail in conjunction with the specific implementation process.

[0039] 1. Robot navigation system initialization and coordinate alignment.

[0040] Hardware configuration: The robot is equipped with 3D lidar, visual camera, IMU, wheel odometer, and RTK equipment.

[0041] Remote start process: The user remotely controls the robot to an open area via the app, ensuring the RTK solution is fixed. The robot automatically aligns the LiDAR coordinate system with the WGS84 coordinate system.

[0042] 2. User interaction and area planning.

[0043] GIS Map Selection: The user app displays a real-time GIS map and the robot's location. Users can draw a closed polygon on the app to define a target area. Vector road data (including road type, width, direction, and other attributes) for the target area is downloaded from the open map platform. The app displays "GIS Road Data Loading Completed" and highlights the accessible road network. The system extracts all road nodes within the circle and generates a road topology network diagram.

[0044] Path planning algorithm: In the robot mapping scenario, it can be compared to "letting the robot cover all roads within the user-defined area with the shortest path." The actual roads are abstracted into "nodes" and "edges":

[0045] Node: represents a road intersection or endpoint (such as points A, B, and C in the figure);

[0046] Edge: represents a road segment between two nodes (such as AB, BC), and each edge is marked with its length (or travel time).

[0047] Traverse all paths and solve the optimal path, which covers all roads and has the shortest total distance; dynamically adjust the path order to avoid obstacles detected in real time.

[0048] 3. Fully automatic mapping execution.

[0049] Visual perception of the road: In this embodiment, a camera is used to capture road surface images in real time. A deep learning model (such as ResNet-50+UNet) is used to segment the road area and identify elements such as lane lines, curbs, and sidewalks. The road boundary is fitted with the lidar point cloud to generate a drivable area.

[0050] Specifically, step 1: sensor calibration.

[0051] Objective: To establish the coordinate transformation relationship between the visual camera and the lidar.

[0052] Operation: Fix the camera and lidar on the robot to ensure that their fields of view overlap; use a calibration plate (such as a chessboard) to synchronously collect data from both; calculate the conversion matrix from the camera image pixel coordinates to the lidar 3D coordinates.

[0053] Result: The position of any image point is known and can be converted into 3D coordinates in the lidar coordinate system.

[0054] Step 2: The visual model detects the road surface (camera part).

[0055] Model selection: Use a lightweight semantic segmentation model (such as MobileNet+UNet) to segment road surface, lane lines, curbs, and other features in the image in real time.

[0056] Output: The road surface area in the image is marked in green; the curb / obstacle is marked in red.

[0057] Step 3: Project the visual results to the lidar coordinate system.

[0058] Operation: Extract contour points (pixel coordinates) from the road surface area segmented in the image; convert pixel coordinates into 3D points in the LiDAR coordinate system through the calibration matrix; generate a "visual detection road surface point cloud" in the LiDAR coordinate system.

[0059] As an example, a road point in an image (x_img, y_img) is converted to lidar coordinates (x_lidar, y_lidar, z_lidar).

[0060] Step 4: LiDAR point cloud verification and correction.

[0061] Verification logic: extract ground points from lidar point cloud (by height filtering or plane fitting);

[0062] Comparison of "visual inspection road point cloud" and "lidar ground point cloud":

[0063] If consistent: the overlap between the visual and laser detected road areas is greater than 90%, it is determined to be reliable;

[0064] If the overlap is 50% to 90%, the laser data is used as the main data, and the visual results are used to assist in completing the details.

[0065] If conflict: coincidence < 50%, trigger alarm and recalibrate the sensor.

[0066] Corrective action: If the visual system detects the road surface but the laser displays it as an obstacle (such as misjudgment of reflection from accumulated water), the laser data shall prevail. If the laser point cloud is missing but the visual system detects a clear road surface (such as strong light interfering with the laser), the visual data shall prevail.

[0067] In this embodiment, a laser radar is added to achieve reliable RTK judgment.

[0068] Specifically, the RTK multipath error detection method performs horizontal sector segmentation and vertical layer segmentation on the laser point cloud to divide the laser point cloud into multiple sectors. In one application scenario, horizontal sector segmentation divides the horizontal 360° point cloud into 24 sectors (S1 to S24) at 15° intervals, improving directional resolution (from 30° to 15°).

[0069] Vertical layering: The point cloud is divided into three layers (0-2m, 2-5m, and above 5m) in the vertical direction. The occlusion of each layer is analyzed separately to prevent low obstacles (such as shrubs) from misidentifying high-level occlusion. For example, the lower layer (0-2m) is mainly used to detect low obstacles, the middle layer (2-5m) is mainly used to navigate the obstacle layer, and the upper layer (>5m) is used to analyze building occlusion.

[0070] Refined judgment of occluded areas.

[0071] Perform Euclidean clustering on the point cloud in each sector and remove noise points (clusters with points < 10); calculate the effective point cloud density ρ (number of points / sector volume); if ρ> threshold ρ_th (e.g. 100 points / m 3 ) and the farthest point cloud distance in the fan-shaped area is D_max<50m, it is marked as a strong occlusion area; if ρ≤ρ_th and D_max<50m, it is marked as a weak occlusion area (possibly a sparse obstacle or a dynamic object).

[0072] Furthermore, dynamic obstacles can be filtered: by comparing 5 consecutive frames of point cloud, dynamic obstacles that appear briefly (such as pedestrians and vehicles) are eliminated; only occlusion areas that persist for ≥ 3 frames are retained for RTK reliability assessment.

[0073] RTK reliability rule optimization.

[0074] The present invention uses a multi-dimensional evaluation model to register and classify RTK reliability. For example, in the following Table 1:

[0075] Table 1 RTK reliability evaluation matrix

[0076]

[0077] The number of strong and weak obstruction sectors and the distribution pattern can be used to categorize RTK reliability levels. For example, in the distribution pattern, the sectors are arranged in the order 12345678. A continuous distribution indicates that 123456 are all obstruction areas.

[0078] Furthermore, the present invention can also detect temporary interference in the environment through annular distribution detection. Specifically, if the obstruction sectors are annularly symmetrically distributed (for example, the sectors include multiple sector-shaped areas, which are divided one by one in a counterclockwise direction, and the sectors are numbered in ascending order, if not continuously distributed, then intermittently distributed, such as sectors S1, S7, S13, and S19), it is determined to be temporary interference in the open environment, and the RTK weight is not reduced.

[0079] Multi-frame sliding window detection.

[0080] The present invention also uses a multi-frame sliding window detection process to eliminate the impact of transient fluctuations. In one application scenario, a 10-frame sliding window was used to count the number of occluded sectors to eliminate transient fluctuations. Finally, N_strong / N_weak was averaged within the window to improve detection stability.

[0081] Multi-sensor fusion positioning:

[0082] Laser positioning: SLAM provides local poses through point cloud feature matching. Laser is an odometry model, and the error between each frame is assumed to be fixed. For example, in practice, 0.5% means a cumulative error of 0.5 over 100 meters. Based on this, the weight of the laser point cloud is fixed during fusion.

[0083] RTK global correction: aligns laser positioning results with RTK coordinates at a frequency of 1 Hz to eliminate accumulated errors.

[0084] Dynamic weight adjustment:

[0085] Open area (RTK high reliability): visual weighting 30%, RTK weighting 70%;

[0086] Occluded area (RTK low reliability): visual weight 70%, RTK weight 30%.

[0087] Obstacle avoidance and path tracking: When moving along a planned path, the vehicle uses LiDAR to detect obstacles (such as pedestrians and vehicles) within 5 meters ahead, triggering the Dynamic Window Algorithm (DWA) to adjust speed and direction in real time. If the road is blocked by a temporary obstacle, the coordinates of that point are automatically recorded and a detour route is replanned.

[0088] 4. Map fusion and closed-loop verification.

[0089] Real-time map stitching: Laser SLAM generates a 3D point cloud map, overlaying visual semantic information (road type, obstacle labels) onto the grid layer. After every 200m of path scanning, a global loop closure check is performed using RTK coordinates and GIS data to correct map offsets. Visual semantic information is detected in real time, allowing for loop closure searches.

[0090] Automatic map completion detection: calculate the map coverage (actual scanned road length / total planned road length). If it is less than 95%, mark the uncovered area and automatically re-scan it.

[0091] This invention achieves fully automatic, high-precision, road-guided mapping of outdoor scenes by synergizing visual perception of road semantics with RTK global positioning. No human intervention is required from user selection to map completion, saving over 90% of operation time. An automatic rescan mechanism ensures full map coverage. Scan paths are optimized, reducing repeated movement distances by >30%. Dynamic obstacle avoidance prevents mission interruptions. Visual semantics enhance map readability (e.g., distinguishing lanes from sidewalks). RTK and visual SLAM complement each other to suppress cumulative errors, achieving a positioning accuracy of ≤0.2m.

[0092] Figure 2 3 is a block diagram showing a structure of a robot automatic mapping system based on RTK and visual perception according to an embodiment of the present invention.

[0093] The present invention also provides a robot automatic mapping system based on RTK and visual perception. Figure 2 As shown, the system includes a processor and a memory, and the memory stores computer program instructions. When the computer program instructions are executed by the processor, a robot automatic mapping method based on RTK and visual perception according to the first aspect of the present invention is implemented.

[0094] The system further includes other components well known to those skilled in the art, such as a communication bus and a communication interface. The configuration and functions of these components are known in the art and thus will not be described in detail here.

[0095] In the present invention, the aforementioned memory can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. For example, the computer-readable storage medium can be any suitable magnetic storage medium or magneto-optical storage medium, such as resistive random access memory (RRAM), dynamic random access memory (DRAM), static random access memory (SRAM), enhanced dynamic random access memory (EDRAM), high-bandwidth memory (HBM), hybrid memory cube (HMC), etc., or any other medium that can be used to store the required information and can be accessed by an application, module, or both. Any such computer storage medium can be part of a device or accessible or connectable to a device. Any application or module described in the present invention can be implemented using computer-readable / executable instructions that can be stored or otherwise retained by such a computer-readable medium.

[0096] In the description of this specification, "multiple" and "several" mean at least two, such as two, three or more, unless otherwise clearly defined.

[0097] While several embodiments of the present invention have been shown and described herein, it will be apparent to those skilled in the art that such embodiments are provided by way of example only. Numerous modifications, variations, and alternatives will occur to those skilled in the art without departing from the concept and spirit of the present invention. It should be understood that various alternatives to the embodiments of the present invention described herein may be employed in practicing the present invention.

Claims

1. A robot automatic mapping method based on RTK and visual perception, characterized in that: include: Acquire RTK data, lidar point clouds, and visual perception images; Using semantic segmentation models to segment visual perception images to determine contour points in road areas; Convert the contour points in the road surface area into a visual detection road surface point cloud in the LiDAR coordinate system; The visually detected road point cloud is integrated with the lidar point cloud and RTK data to complete the map construction.

2. The robot automatic mapping method based on RTK and visual perception according to claim 1 is characterized in that: The contour points in the road area are converted into a visual detection road point cloud in the LiDAR coordinate system, including: The pixel coordinates of the contour points in the road area are converted into 3D points in the LiDAR coordinate system through the calibration matrix; Generate visually detected road point cloud in the lidar coordinate system.

3. The method for automatic robot mapping based on RTK and visual perception according to claim 1, characterized in that: Before fusing the visual inspection road point cloud with the lidar point cloud data, the following steps are also required: Verify and correct the ground point cloud in the lidar point cloud and the visual detection road point cloud.

4. The method for automatic robot mapping based on RTK and visual perception according to claim 3, characterized in that: The verification process includes: Determine the degree of overlap between the LiDAR point cloud and the road surface area detected in the visual inspection road surface point cloud; In response to the overlap being greater than a first set value, determining that the overlap is reliable; In response to the overlap being less than a first set value and greater than a second set value, the road surface point cloud is visually detected with the lidar point cloud as the primary detection point cloud to supplement details; In response to the overlap being less than a second set value, an alarm is triggered.

5. The method for automatic robot mapping based on RTK and visual perception according to claim 3, characterized in that: The correction process includes: In response to the visual perception image being identified as a road surface and the lidar point cloud being identified as an obstacle, the lidar data recognition result shall prevail; In response to the lack of a lidar point cloud and the visual perception image being recognized as a road surface, the visual perception image recognition result shall prevail.

6. The method for automatic robot mapping based on RTK and visual perception according to claim 1, characterized in that: Fusion of visually detected road point clouds with LiDAR point clouds and RTK data to complete map construction, including: Generate 3D point cloud map using laser point cloud, and superimpose visual inspection road point cloud onto the grid layer; Perform global coordinate detection based on RTK data to correct map offset.

7. The method for automatic robot mapping based on RTK and visual perception according to claim 1, characterized in that: Also includes: LiDAR point cloud is used to determine RTK reliability to reduce multipath interference.

8. The method for automatic robot mapping based on RTK and visual perception according to claim 7, characterized in that: Use LiDAR point clouds to determine RTK reliability, including: Obtain laser point cloud and RTK positioning data; Use multipath error detection algorithms to determine the occluded areas in the laser point cloud to calculate RTK reliability; A multi-layer sensor fusion architecture with dynamic weight allocation is used to fuse laser point cloud and RTK positioning data to achieve robot navigation. The fusion weight is determined by a multi-level weight decision model, and the basic weight in the multi-level weight decision model is determined according to the level of RTK reliability.

9. The method for automatic robot mapping based on RTK and visual perception according to claim 8, characterized in that: A multi-level weight decision model is used to determine the dynamic weight, including: High-level decisions are weighted by RTK reliability, which is determined by the size of the occluded area in the laser point cloud. The mid-layer correction obtains the RTK weight through dynamic correction of the point cloud density, where the RTK weight is negatively correlated with the point cloud density of the laser point cloud; The underlying compensation compensates the Kalman filter by injecting high frequency into the IMU / odometer.

10. A robot automatic mapping system based on RTK and visual perception, characterized in that: include: processor; A memory storing computer program instructions, wherein when the computer program instructions are executed by the processor, a robot navigation method based on RTK and laser radar according to any one of claims 1 to 9 is implemented.

Images